Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10303

DataStreamer#ResponseProcessor calculates packet ack latency incorrectly.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Packets acknowledge duration should be calculated based on the packet send time.

      1. HDFS-10303-002.patch
        7 kB
        Surendra Singh Lilhore
      2. HDFS-10303-001.patch
        7 kB
        Surendra Singh Lilhore

        Activity

        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Kihwal Lee for review and commit.

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Kihwal Lee for review and commit.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9792 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9792/)
        HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency (kihwal: rev 4a5819dae2b0ca8f8b6d94ef464882d079d86593)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataStream.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9792 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9792/ ) HDFS-10303 . DataStreamer#ResponseProcessor calculates packet ack latency (kihwal: rev 4a5819dae2b0ca8f8b6d94ef464882d079d86593) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataStream.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
        Hide
        kihwal Kihwal Lee added a comment -

        I've committed this to trunk, branch-2 and branch-2.8. Thanks for fixing the issue, Surendra Singh Lilhore.

        Show
        kihwal Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-2.8. Thanks for fixing the issue, Surendra Singh Lilhore .
        Hide
        kihwal Kihwal Lee added a comment -

        +1 The latest patch looks good. The failed test cases are unrelated to this change.

        Show
        kihwal Kihwal Lee added a comment - +1 The latest patch looks good. The failed test cases are unrelated to this change.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 6m 50s trunk passed
        +1 compile 1m 20s trunk passed with JDK v1.8.0_91
        +1 compile 1m 23s trunk passed with JDK v1.7.0_101
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 1m 28s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 3m 43s trunk passed
        +1 javadoc 1m 29s trunk passed with JDK v1.8.0_91
        +1 javadoc 2m 24s trunk passed with JDK v1.7.0_101
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 27s the patch passed
        +1 compile 1m 27s the patch passed with JDK v1.8.0_91
        +1 javac 1m 27s the patch passed
        +1 compile 1m 30s the patch passed with JDK v1.7.0_101
        +1 javac 1m 30s the patch passed
        -1 checkstyle 0m 34s hadoop-hdfs-project: patch generated 5 new + 77 unchanged - 1 fixed = 82 total (was 78)
        +1 mvnsite 1m 29s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 4m 28s the patch passed
        +1 javadoc 1m 43s the patch passed with JDK v1.8.0_91
        +1 javadoc 2m 20s the patch passed with JDK v1.7.0_101
        +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_91.
        -1 unit 63m 31s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101.
        -1 unit 54m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        158m 34s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.TestAsyncDFSRename
        JDK v1.7.0_101 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
          hadoop.hdfs.shortcircuit.TestShortCircuitCache



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804160/HDFS-10303-002.patch
        JIRA Issue HDFS-10303
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a25fa5a10944 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ad94411
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15445/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15445/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 50s trunk passed +1 compile 1m 20s trunk passed with JDK v1.8.0_91 +1 compile 1m 23s trunk passed with JDK v1.7.0_101 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 43s trunk passed +1 javadoc 1m 29s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 24s trunk passed with JDK v1.7.0_101 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 1m 27s the patch passed with JDK v1.8.0_91 +1 javac 1m 27s the patch passed +1 compile 1m 30s the patch passed with JDK v1.7.0_101 +1 javac 1m 30s the patch passed -1 checkstyle 0m 34s hadoop-hdfs-project: patch generated 5 new + 77 unchanged - 1 fixed = 82 total (was 78) +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 25s the patch passed -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 4m 28s the patch passed +1 javadoc 1m 43s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 20s the patch passed with JDK v1.7.0_101 +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_91. -1 unit 63m 31s hadoop-hdfs in the patch failed with JDK v1.8.0_91. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_101. -1 unit 54m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 158m 34s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestAsyncDFSRename JDK v1.7.0_101 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804160/HDFS-10303-002.patch JIRA Issue HDFS-10303 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a25fa5a10944 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ad94411 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15445/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15445/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15445/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Kihwal Lee for review comments.
        Attached patch v.2., addressed above comments.
        Please review.

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Kihwal Lee for review comments. Attached patch v.2., addressed above comments. Please review.
        Hide
        kihwal Kihwal Lee added a comment -
        • There is a duplicate check for the heartbeat sequence number. The inner one is no longer necessary.
        • The existing log level may not be ideal. This is directly visible to users and unless something actually failed, it is better to log at INFO.
        • I am concerned about the new test case becoming one of the flaky ones that fail when running on a busy build machine. We need to make sure it is reliable.
        Show
        kihwal Kihwal Lee added a comment - There is a duplicate check for the heartbeat sequence number. The inner one is no longer necessary. The existing log level may not be ideal. This is directly visible to users and unless something actually failed, it is better to log at INFO . I am concerned about the new test case becoming one of the flaky ones that fail when running on a busy build machine. We need to make sure it is reliable.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Checkstyle warning and Failed test case is unrelated..

        Please review..

        org.apache.hadoop.hdfs.TestHFlush.testHFlushInterrupted

        This test case fixed in HDFS-2043.

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Checkstyle warning and Failed test case is unrelated.. Please review.. org.apache.hadoop.hdfs.TestHFlush.testHFlushInterrupted This test case fixed in HDFS-2043 .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 43s Maven dependency ordering for branch
        +1 mvninstall 6m 48s trunk passed
        +1 compile 1m 20s trunk passed with JDK v1.8.0_91
        +1 compile 1m 20s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 1m 23s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 36s trunk passed
        +1 javadoc 1m 24s trunk passed with JDK v1.8.0_91
        +1 javadoc 2m 12s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 15s the patch passed
        +1 compile 1m 9s the patch passed with JDK v1.8.0_91
        +1 javac 1m 9s the patch passed
        +1 compile 1m 16s the patch passed with JDK v1.7.0_95
        +1 javac 1m 16s the patch passed
        -1 checkstyle 0m 25s hadoop-hdfs-project: patch generated 1 new + 77 unchanged - 1 fixed = 78 total (was 78)
        +1 mvnsite 1m 19s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 57s the patch passed
        +1 javadoc 1m 20s the patch passed with JDK v1.8.0_91
        +1 javadoc 2m 6s the patch passed with JDK v1.7.0_95
        +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_91.
        -1 unit 56m 42s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
        -1 unit 53m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        147m 57s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestHFlush
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802364/HDFS-10303-001.patch
        JIRA Issue HDFS-10303
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5e9d2fc6ac74 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 72b0477
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15370/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15370/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 6m 48s trunk passed +1 compile 1m 20s trunk passed with JDK v1.8.0_91 +1 compile 1m 20s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 36s trunk passed +1 javadoc 1m 24s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 12s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 9s the patch passed with JDK v1.8.0_91 +1 javac 1m 9s the patch passed +1 compile 1m 16s the patch passed with JDK v1.7.0_95 +1 javac 1m 16s the patch passed -1 checkstyle 0m 25s hadoop-hdfs-project: patch generated 1 new + 77 unchanged - 1 fixed = 78 total (was 78) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 57s the patch passed +1 javadoc 1m 20s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 6s the patch passed with JDK v1.7.0_95 +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_91. -1 unit 56m 42s hadoop-hdfs in the patch failed with JDK v1.8.0_91. +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 53m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 147m 57s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestHFlush JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802364/HDFS-10303-001.patch JIRA Issue HDFS-10303 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5e9d2fc6ac74 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 72b0477 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15370/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15370/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Attached initial patch..

        Maintaining one map packetSendTime with packet sequence number and packet send time.

        After receiving acknowledge for packet this time is used for calculating the packet acknowledge duration.

        Please review..

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Attached initial patch.. Maintaining one map packetSendTime with packet sequence number and packet send time. After receiving acknowledge for packet this time is used for calculating the packet acknowledge duration. Please review..
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        I am getting the "Slow ReadProcessor read" log in my cluster when I increase the socket timeout for client.

        16/04/14 17:57:59 WARN DataStreamer: Slow ReadProcessor read fields for block BP-873267638-192.168.100.12-1460002479721:blk_1073752739_11917 took 47858ms (threshold=30000ms); ack: seqno: 3 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803180 flag: 0 flag: 0 flag: 0, targets: [DatanodeInfoWithStorage[192.168.100.9:25009,DS-d552bfd7-1c38-430d-8703-c3b539caf351,DISK], DatanodeInfoWithStorage[192.168.100.11:25009,DS-02897c9b-bceb-4790-b08a-f711d8e3fd81,DISK], DatanodeInfoWithStorage[192.168.100.10:25009,DS-fae7b497-a269-4614-afe5-7006660eafcf,DISK]]
        

        But when I checked the packet send time, it is same as packet acknowledge time

        16/04/14 17:57:59 DEBUG DataStreamer: DataStreamer block BP-873267638-192.168.100.12-1460002479721:blk_1073752739_11917 sending packet packet seqno: 3 offsetInBlock: 8704 lastPacketInBlock: false lastByteOffsetInBlock: 12316
        

        This is coming because ResponseProcessor set the current time as begin time and wait for the packet ack, after getting the ack it will calculate the duration and compare with the dfs.client.slow.io.warning.threshold.ms.

                  // read an ack from the pipeline
                  long begin = Time.monotonicNow();
                  ack.readFields(blockReplyStream);
                  long duration = Time.monotonicNow() - begin;
        

        Suppose client sent two packets and now he doesn't have data to write, after some time he got more data and sent third packet.

        Client waited for some time after sending second packet. Time between second packet and third packet should not be considered by ResponseProcessor in packet acknowledge duration.

        Show
        surendrasingh Surendra Singh Lilhore added a comment - I am getting the "Slow ReadProcessor read" log in my cluster when I increase the socket timeout for client. 16/04/14 17:57:59 WARN DataStreamer: Slow ReadProcessor read fields for block BP-873267638-192.168.100.12-1460002479721:blk_1073752739_11917 took 47858ms (threshold=30000ms); ack: seqno: 3 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803180 flag: 0 flag: 0 flag: 0, targets: [DatanodeInfoWithStorage[192.168.100.9:25009,DS-d552bfd7-1c38-430d-8703-c3b539caf351,DISK], DatanodeInfoWithStorage[192.168.100.11:25009,DS-02897c9b-bceb-4790-b08a-f711d8e3fd81,DISK], DatanodeInfoWithStorage[192.168.100.10:25009,DS-fae7b497-a269-4614-afe5-7006660eafcf,DISK]] But when I checked the packet send time, it is same as packet acknowledge time 16/04/14 17:57:59 DEBUG DataStreamer: DataStreamer block BP-873267638-192.168.100.12-1460002479721:blk_1073752739_11917 sending packet packet seqno: 3 offsetInBlock: 8704 lastPacketInBlock: false lastByteOffsetInBlock: 12316 This is coming because ResponseProcessor set the current time as begin time and wait for the packet ack, after getting the ack it will calculate the duration and compare with the dfs.client.slow.io.warning.threshold.ms . // read an ack from the pipeline long begin = Time.monotonicNow(); ack.readFields(blockReplyStream); long duration = Time.monotonicNow() - begin; Suppose client sent two packets and now he doesn't have data to write, after some time he got more data and sent third packet. Client waited for some time after sending second packet. Time between second packet and third packet should not be considered by ResponseProcessor in packet acknowledge duration.

          People

          • Assignee:
            surendrasingh Surendra Singh Lilhore
            Reporter:
            surendrasingh Surendra Singh Lilhore
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development