Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8311

DataStreamer.transfer() should timeout the socket InputStream.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      While validating some HA failure modes we found that HDFS clients can take a long time to recover or sometimes don't recover at all since we don't setup the socket timeout in the InputStream:

      private void transfer () { ...
      ...
       OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
       InputStream unbufIn = NetUtils.getInputStream(sock);
      ...
      }
      

      The InputStream should have its own timeout in the same way as the OutputStream.

        Issue Links

          Activity

          Hide
          esteban Esteban Gutierrez added a comment -

          Kiran Kumar M R is ok for you if I reassign the JIRA to me? I have an initial patch that re-uses the existing write timeout in the OuputStream and seems to be fine.

          Show
          esteban Esteban Gutierrez added a comment - Kiran Kumar M R is ok for you if I reassign the JIRA to me? I have an initial patch that re-uses the existing write timeout in the OuputStream and seems to be fine.
          Hide
          esteban Esteban Gutierrez added a comment -

          Added correctly timeout for both DS.createBlockOutputStream() and DS.transfer() and now it relies on DFSClient.getDatanodeReadTimeout() (both usually should be the same, but now is semantically correct)

          Show
          esteban Esteban Gutierrez added a comment - Added correctly timeout for both DS.createBlockOutputStream() and DS.transfer() and now it relies on DFSClient.getDatanodeReadTimeout() (both usually should be the same, but now is semantically correct)
          Hide
          kiranmr Kiran Kumar M R added a comment -

          Thats fine, reassigned

          Show
          kiranmr Kiran Kumar M R added a comment - Thats fine, reassigned
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Esteban Gutierrez. +1 pending jenkins

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Esteban Gutierrez . +1 pending jenkins
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 41s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 27s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 15s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 0s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 13s Pre-build of native portion
          +1 hdfs tests 167m 45s Tests passed in hadoop-hdfs.
              210m 31s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730229/HDFS-8311.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ffce9a3
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10818/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10818/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10818/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 27s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 15s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 0s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 13s Pre-build of native portion +1 hdfs tests 167m 45s Tests passed in hadoop-hdfs.     210m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730229/HDFS-8311.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ffce9a3 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10818/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10818/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10818/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Esteban Gutierrez,

          Thanks for addressing my verbal comments.

          I found that there are other places that might need similar handling, like
          DataXceiver#replaceBlock

                  OutputStream unbufProxyOut = NetUtils.getOutputStream(proxySock,
                      dnConf.socketWriteTimeout);
                  InputStream unbufProxyIn = NetUtils.getInputStream(proxySock);
          

          DataXceiver#writeBlock

                    OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock,
                        writeTimeout);
                    InputStream unbufMirrorIn = NetUtils.getInputStream(mirrorSock);
          

          DataNode#run

                  OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
                  InputStream unbufIn = NetUtils.getInputStream(sock);
          

          And some other places.

          I know that Esteban Gutierrez has verified the uploaded patch solve one real issue, so maybe it's ok for us to address the remaining places in separate jiras.

          What do you think Colin P. McCabe?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Esteban Gutierrez , Thanks for addressing my verbal comments. I found that there are other places that might need similar handling, like DataXceiver#replaceBlock OutputStream unbufProxyOut = NetUtils.getOutputStream(proxySock, dnConf.socketWriteTimeout); InputStream unbufProxyIn = NetUtils.getInputStream(proxySock); DataXceiver#writeBlock OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, writeTimeout); InputStream unbufMirrorIn = NetUtils.getInputStream(mirrorSock); DataNode#run OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout); InputStream unbufIn = NetUtils.getInputStream(sock); And some other places. I know that Esteban Gutierrez has verified the uploaded patch solve one real issue, so maybe it's ok for us to address the remaining places in separate jiras. What do you think Colin P. McCabe ? Thanks.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Good catch, Yongjun Zhang. We should fix those other cases as well. I think we should do those in separate JIRAs, if that's more convenient for you. Also, it would be nice to have unit tests for these timeouts at some point, to ensure that they don't get removed.

          +1 again for the patch. Thanks, guys.

          Show
          cmccabe Colin P. McCabe added a comment - Good catch, Yongjun Zhang . We should fix those other cases as well. I think we should do those in separate JIRAs, if that's more convenient for you. Also, it would be nice to have unit tests for these timeouts at some point, to ensure that they don't get removed. +1 again for the patch. Thanks, guys.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Colin P. McCabe. +1 and will commit soon.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Colin P. McCabe . +1 and will commit soon.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7778 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7778/)
          HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7778 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7778/ ) HDFS-8311 . DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Esteban Gutierrez for reporting the issue and the fix, thanks Colin for the review, I committed to trunk and branch-2.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Esteban Gutierrez for reporting the issue and the fix, thanks Colin for the review, I committed to trunk and branch-2.
          Hide
          esteban Esteban Gutierrez added a comment -

          Thanks Yongjun Zhang!

          Show
          esteban Esteban Gutierrez added a comment - Thanks Yongjun Zhang !
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Yongjun Zhang

          Separate is logged for following..? If not, can I file jira and fix..?
          DataXceiver#replaceBlock

           OutputStream unbufProxyOut = NetUtils.getOutputStream(proxySock,
                      dnConf.socketWriteTimeout);
                  InputStream unbufProxyIn = NetUtils.getInputStream(proxySock);

          DataXceiver#writeBlock

           OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock,
                        writeTimeout);
                    InputStream unbufMirrorIn = NetUtils.getInputStream(mirrorSock);

          DataNode#run

          OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
                  InputStream unbufIn = NetUtils.getInputStream(sock);
          Show
          brahmareddy Brahma Reddy Battula added a comment - Yongjun Zhang Separate is logged for following..? If not, can I file jira and fix..? DataXceiver#replaceBlock OutputStream unbufProxyOut = NetUtils.getOutputStream(proxySock, dnConf.socketWriteTimeout); InputStream unbufProxyIn = NetUtils.getInputStream(proxySock); DataXceiver#writeBlock OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, writeTimeout); InputStream unbufMirrorIn = NetUtils.getInputStream(mirrorSock); DataNode#run OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout); InputStream unbufIn = NetUtils.getInputStream(sock);
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Brahma Reddy Battula,

          Thanks for your interest, I was about to check with Esteban Gutierrez, so I will let you guys decide who will fix it. Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Brahma Reddy Battula , Thanks for your interest, I was about to check with Esteban Gutierrez , so I will let you guys decide who will fix it. Thanks.
          Hide
          esteban Esteban Gutierrez added a comment -

          Brahma Reddy Battula, Yongjun Zhang I've created HDFS-8359. Depending what we might find down the road we can move that jira as an umbrella jura and this jira as a subtask of HDFS-8359. Thanks again!

          Show
          esteban Esteban Gutierrez added a comment - Brahma Reddy Battula , Yongjun Zhang I've created HDFS-8359 . Depending what we might find down the road we can move that jira as an umbrella jura and this jira as a subtask of HDFS-8359 . Thanks again!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/)
          HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/ ) HDFS-8311 . DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/)
          HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/ ) HDFS-8311 . DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/)
          HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/ ) HDFS-8311 . DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/)
          HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/ ) HDFS-8311 . DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/)
          HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/ ) HDFS-8311 . DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/)
          HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/ ) HDFS-8311 . DataStreamer.transfer() should timeout the socket InputStream. (Esteban Gutierrez via Yongjun Zhang) (yzhang: rev 730f9930a48259f34e48404aee51e8d641cc3d36) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java

            People

            • Assignee:
              esteban Esteban Gutierrez
              Reporter:
              esteban Esteban Gutierrez
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development