Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6618

FSNamesystem#delete drops the FSN lock between removing INodes from the tree and deleting them from the inode map

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.5.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      After HDFS-6527, we have not seen the edit log corruption for weeks on multiple clusters until yesterday. Previously, we would see it within 30 minutes on a cluster.

      But the same condition was reproduced even with HDFS-6527. The only explanation is that the RPC handler thread serving addBlock() was accessing stale parent value. Although nulling out parent is done inside the FSNamesystem and FSDirectory write lock, there is no memory barrier because there is no "synchronized" block involved in the process.

      I suggest making parent volatile.

      1. HDFS-6618.simpler.patch
        3 kB
        Kihwal Lee
      2. HDFS-6618.patch
        2 kB
        Kihwal Lee
      3. HDFS-6618.inodeRemover.v2.patch
        52 kB
        Kihwal Lee
      4. HDFS-6618.inodeRemover.patch
        52 kB
        Kihwal Lee
      5. HDFS-6618.AbstractList.patch
        4 kB
        Kihwal Lee

        Issue Links

          Activity

          Kihwal Lee created issue -
          Kihwal Lee made changes -
          Field Original Value New Value
          Attachment HDFS-6618.patch [ 12653420 ]
          Kihwal Lee made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Kihwal Lee made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Kihwal Lee made changes -
          Attachment HDFS-6681.AbstractList.patch [ 12653485 ]
          Kihwal Lee made changes -
          Attachment HDFS-6681.AbstractList.patch [ 12653485 ]
          Kihwal Lee made changes -
          Attachment HDFS-6618.AbstractList.patch [ 12653487 ]
          Kihwal Lee made changes -
          Attachment HDFS-6618.inodeRemover.patch [ 12653531 ]
          Kihwal Lee made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Kihwal Lee made changes -
          Attachment HDFS-6618.inodeRemover.v2.patch [ 12653582 ]
          Kihwal Lee made changes -
          Summary Edit log corruption may still happen even after HDFS-6527 Remove deleted INodes from INodeMap right away
          Kihwal Lee made changes -
          Assignee Kihwal Lee [ kihwal ]
          Kihwal Lee made changes -
          Attachment HDFS-6618.simpler.patch [ 12654834 ]
          Kihwal Lee made changes -
          Link This issue is depended upon by HDFS-6647 [ HDFS-6647 ]
          Colin Patrick McCabe made changes -
          Summary Remove deleted INodes from INodeMap right away FSNamesystem#delete drops the FSN lock between removing INodes from the tree and deleting them from the inode map
          Colin Patrick McCabe made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.5.0 [ 12326264 ]
          Resolution Fixed [ 1 ]
          Karthik Kambatla (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Kihwal Lee
              Reporter:
              Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development