Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7065

Pipeline close recovery race can cause block corruption

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.6.0
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If multiple pipeline close recoveries are performed against the same block, the replica may go corrupt. Here is one case I have observed:

      The client tried to close a block, but the ACK timed out. It excluded the first DN and tried pipeline recovery (recoverClose). It too failed and another recovery was attempted with only one DN. This took more than usual but the client eventually got an ACK and the file was closed successfully. Later on the one and only replica was found to be corrupt.

      It turned out the DN was having transient slow disk I/O issue at that time. The first recovery was stuck until the second recovery was attempted 30 seconds later. After few seconds, they both threads started running. The second recovery finished first and then the first recovery with an older gen stamp finished, turning gen stamp backward.

      There is a sanity check in recoverCheck(), but since check and modify are not synchronized, recoverClose() is not multi-thread safe.

        Attachments

        1. HDFS-7065.patch
          0.9 kB
          Kihwal Lee

          Activity

            People

            • Assignee:
              kihwal Kihwal Lee
              Reporter:
              kihwal Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: