Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11363

Need more diagnosis info when seeing Slow waitForAckedSeqno

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When client writes a file, it may get the following message when ACK doesn't get back in a timely manner:

      WARN hdfs.DFSClient: Slow waitForAckedSeqno took 39264ms (threshold=30000ms)

      It would be nice to tell what file it's writing, and what DataNodes are in the pipeline, together with this message, to facilitate investigating related performance issue.

      1. HDFS-11363.01.patch
        1.0 kB
        Xiao Chen
      2. HDFS-11363.02.patch
        1.0 kB
        Xiao Chen
      3. HDFS-11363.03.patch
        1.0 kB
        Xiao Chen

        Activity

        Hide
        xiaochen Xiao Chen added a comment -

        Attaching patch 1 for the logs.
        Added filename, block, and datanode list to the message. A sample output looks like:

        2017-01-24 18:15:02,402 WARN hdfs.DataStreamer: Slow waitForAckedSeqno took 144ms (threshold=30000ms). 
        File: /dir/f._COPYING_, block: BP-1326647282-172.16.3.121-1485309974936:blk_1073741825_1001, 
        datanodes: [DatanodeInfoWithStorage[127.0.0.1:8765,DS-f00ca0c5-5cfe-41d1-910f-c0494a50334d,DISK]]
        

        Yongjun Zhang, is this what you had in mind? Thanks.

        Show
        xiaochen Xiao Chen added a comment - Attaching patch 1 for the logs. Added filename, block, and datanode list to the message. A sample output looks like: 2017-01-24 18:15:02,402 WARN hdfs.DataStreamer: Slow waitForAckedSeqno took 144ms (threshold=30000ms). File: /dir/f._COPYING_, block: BP-1326647282-172.16.3.121-1485309974936:blk_1073741825_1001, datanodes: [DatanodeInfoWithStorage[127.0.0.1:8765,DS-f00ca0c5-5cfe-41d1-910f-c0494a50334d,DISK]] Yongjun Zhang , is this what you had in mind? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 16m 16s trunk passed
        +1 compile 0m 41s trunk passed
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 41s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        +1 findbugs 1m 50s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 39s the patch passed
        +1 compile 0m 34s the patch passed
        +1 javac 0m 34s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 43s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 48s the patch passed
        +1 javadoc 0m 23s the patch passed
        +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        28m 29s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11363
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849209/HDFS-11363.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 93c9b40dfd4a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b57368b
        Default Java 1.8.0_121
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18254/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18254/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 16m 16s trunk passed +1 compile 0m 41s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 41s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 43s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 23s the patch passed +1 unit 1m 9s hadoop-hdfs-client in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 28m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11363 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849209/HDFS-11363.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 93c9b40dfd4a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b57368b Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18254/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18254/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Xiao Chen for the patch.

        The patch is largely what I want, but some comments/suggestions:
        1. print out the array in the current patch way won't work. suggest to use Arrays.toString(array)
        2. Replace File: with File being written:
        3. Replace datanodes: with Write pipeline datanodes:

        Does that make sense to you?

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Xiao Chen for the patch. The patch is largely what I want, but some comments/suggestions: 1. print out the array in the current patch way won't work. suggest to use Arrays.toString(array) 2. Replace File: with File being written: 3. Replace datanodes: with Write pipeline datanodes: Does that make sense to you? Thanks.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Yongjun Zhang for the review.

        The placeholder in slf4j supports array parameters, so it works. See https://www.slf4j.org/faq.html#paramException

        Texts in #2 and #3 are updated in patch 2.

        Show
        xiaochen Xiao Chen added a comment - Thanks Yongjun Zhang for the review. The placeholder in slf4j supports array parameters, so it works. See https://www.slf4j.org/faq.html#paramException Texts in #2 and #3 are updated in patch 2.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Xiao Chen for the new rev, except for an extra ":", I'm +1.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Xiao Chen for the new rev, except for an extra ":", I'm +1.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 14m 25s trunk passed
        +1 compile 0m 34s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 37s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 36s trunk passed
        +1 javadoc 0m 23s trunk passed
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 33s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 43s the patch passed
        +1 javadoc 0m 20s the patch passed
        +1 unit 1m 0s hadoop-hdfs-client in the patch passed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        25m 4s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11363
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850524/HDFS-11363.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c9ea178ebf2c 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b6f290d
        Default Java 1.8.0_121
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18313/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18313/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 25s trunk passed +1 compile 0m 34s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 36s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 34s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 43s the patch passed +1 javadoc 0m 20s the patch passed +1 unit 1m 0s hadoop-hdfs-client in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 25m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11363 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850524/HDFS-11363.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c9ea178ebf2c 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b6f290d Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18313/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18313/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Oops, thanks Yongjun Zhang, good catch!

        I'd like to fix it at commit time if you agree.

        Show
        xiaochen Xiao Chen added a comment - Oops, thanks Yongjun Zhang , good catch! I'd like to fix it at commit time if you agree.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 15m 0s trunk passed
        +1 compile 0m 40s trunk passed
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 41s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 41s trunk passed
        +1 javadoc 0m 26s trunk passed
        +1 mvninstall 0m 36s the patch passed
        +1 compile 0m 27s the patch passed
        +1 javac 0m 27s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 33s the patch passed
        +1 javadoc 0m 20s the patch passed
        +1 unit 0m 58s hadoop-hdfs-client in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        25m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11363
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850658/HDFS-11363.03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux dfcebbb27ee6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3433f57
        Default Java 1.8.0_121
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18317/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18317/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 0s trunk passed +1 compile 0m 40s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 41s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 26s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 20s the patch passed +1 unit 0m 58s hadoop-hdfs-client in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 25m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11363 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850658/HDFS-11363.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dfcebbb27ee6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3433f57 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18317/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18317/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Patch 3 fixed the ':' that Yongjun +1 pending'ed.

        No test needed due to message change only.

        Committed to trunk and branch-2, thanks Yongjun Zhang for reporting the issue and the review.

        Show
        xiaochen Xiao Chen added a comment - Patch 3 fixed the ':' that Yongjun +1 pending'ed. No test needed due to message change only. Committed to trunk and branch-2, thanks Yongjun Zhang for reporting the issue and the review.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11207 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11207/)
        HDFS-11363. Need more diagnosis info when seeing Slow waitForAckedSeqno. (xiao: rev 0914fcca312b5e9d20bcf1b6633bc13c9034ba46)

        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11207 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11207/ ) HDFS-11363 . Need more diagnosis info when seeing Slow waitForAckedSeqno. (xiao: rev 0914fcca312b5e9d20bcf1b6633bc13c9034ba46) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        any plan to backport to branch-2.8 and branch-2.7..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - any plan to backport to branch-2.8 and branch-2.7..?

          People

          • Assignee:
            xiaochen Xiao Chen
            Reporter:
            yzhangal Yongjun Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development