Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9178

Slow datanode I/O can cause a wrong node to be marked bad

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1
    • None
    • None
    • Reviewed

    Description

      When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the downstream node can timeout on reading packet since even the heartbeat packets will not be relayed down.

      The packet read timeout is set in DataXceiver#run():

        peer.setReadTimeout(dnConf.socketTimeout);
      

      When the downstream node times out and closes the connection to the upstream, the upstream node's PacketResponder gets EOFException and it sends an ack upstream with the downstream node status set to ERROR. This caused the client to exclude the downstream node, even though the upstream node was the one got stuck.

      The connection to downstream has longer timeout, so the downstream will always timeout first. The downstream timeout is set in writeBlock()

                int timeoutValue = dnConf.socketTimeout +
                    (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length);
                int writeTimeout = dnConf.socketWriteTimeout +
                    (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length);
                NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue);
                OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock,
                    writeTimeout);
      

      Attachments

        1. 002-HDFS-9178.branch-2.6.patch
          9 kB
          Junping Du
        2. HDFS-9178.branch-2.6.patch
          9 kB
          Kihwal Lee
        3. HDFS-9178.patch
          9 kB
          Kihwal Lee

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kihwal Kihwal Lee
            kihwal Kihwal Lee
            Votes:
            0 Vote for this issue
            Watchers:
            19 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment