Hadoop Common
  1. Hadoop Common
  2. HADOOP-3339

DFS Write pipeline does not detect defective datanode correctly if it times out.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Improved failure handling of last Data Node in write pipeline.
      Show
      Improved failure handling of last Data Node in write pipeline.

      Description

      When DFSClient is writing to DFS, it does not correctly detect the culprit datanode (rather datanodes do not inform) properly if the bad node times out. Say, the last datanode in in 3 node pipeline is is too slow or defective. In this case, pipeline removes the first two datanodes in first two attempts. The third attempt has only the 3rd datanode in the pipeline and it will fail too. If the pipeline detects the bad 3rd node when the first failure occurs, the write will succeed in the second attempt.

      I will attach example logs of such cases. I think this should be fixed in 0.17.x.

      1. HADOOP-3339.patch
        3 kB
        Raghu Angadi
      2. tmp-3339-dn.patch
        1 kB
        Raghu Angadi

        Issue Links

          Activity

          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Robert Chansler made changes -
          Release Note Some of the failures on 3rd datanode in DFS write pipelie are not detected properly. This could lead to hard failure of client's write operation.
          Improved failure handling of last Data Node in write pipeline.
          Raghu Angadi made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Release Note Some of the failures on 3rd datanode in DFS write pipelie are not detected properly. This could lead to hard failure of client's write operation.
          Resolution Fixed [ 1 ]
          Raghu Angadi made changes -
          Link This issue is related to HADOOP-3416 [ HADOOP-3416 ]
          Raghu Angadi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Raghu Angadi made changes -
          Attachment HADOOP-3339.patch [ 12382218 ]
          Raghu Angadi made changes -
          Attachment tmp-3339-dn.patch [ 12381704 ]
          Nigel Daley made changes -
          Fix Version/s 0.18.0 [ 12312972 ]
          Raghu Angadi made changes -
          Field Original Value New Value
          Description
          When DFSClient is writing to DFS, it does not correctly detect the culprit datanode (rather datanodes do not inform) properly. Say, the last datanode in in 3 node pipeline is is too slow or defective. In this case, pipeline removes the first two datanodes in first two attempts. The third attempt has only the 3rd datanode in the pipeline and it will fail too. If the pipeline detects the bad 3rd node when the first failure occurs, the write will succeed in the second attempt.

          I will attach example logs of such cases. I think this should be fixed in 0.17.x.
          When DFSClient is writing to DFS, it does not correctly detect the culprit datanode (rather datanodes do not inform) properly if the bad node times out. Say, the last datanode in in 3 node pipeline is is too slow or defective. In this case, pipeline removes the first two datanodes in first two attempts. The third attempt has only the 3rd datanode in the pipeline and it will fail too. If the pipeline detects the bad 3rd node when the first failure occurs, the write will succeed in the second attempt.

          I will attach example logs of such cases. I think this should be fixed in 0.17.x.
          Raghu Angadi created issue -

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development