Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9576

HTrace: collect position/length information on read operations

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: hdfs-client, tracing
    • Labels:
      None
    • Target Version/s:

      Description

      HTrace currently collects the path of each read operation (both stateful and position reads). To better understand applications' I/O behavior, it is also useful to track the position and length of read operations.

      1. HDFS-9576.00.patch
        5 kB
        Zhe Zhang
      2. HDFS-9576.01.patch
        5 kB
        Zhe Zhang
      3. HDFS-9576.02.patch
        4 kB
        Zhe Zhang
      4. HDFS-9576.03.patch
        4 kB
        Zhe Zhang
      5. HDFS-9576.04.patch
        4 kB
        Zhe Zhang
      6. HDFS-9576.05.patch
        4 kB
        Zhe Zhang
      7. HDFS-9576.06.patch
        4 kB
        Zhe Zhang
      8. HDFS-9576.07.patch
        4 kB
        Zhe Zhang

        Activity

        Hide
        zhz Zhe Zhang added a comment -

        Thanks Masatake, Xiao, and Colin for the reviews!

        Show
        zhz Zhe Zhang added a comment - Thanks Masatake, Xiao, and Colin for the reviews!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9141 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9141/)
        HDFS-9576: HTrace: collect position/length information on read (cmccabe: rev 7905788db94d560e6668af0d4bed22b326961aaf)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9141 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9141/ ) HDFS-9576 : HTrace: collect position/length information on read (cmccabe: rev 7905788db94d560e6668af0d4bed22b326961aaf) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Hide
        cmccabe Colin P. McCabe added a comment -

        committed to 2.9, thanks!

        Show
        cmccabe Colin P. McCabe added a comment - committed to 2.9, thanks!
        Hide
        cmccabe Colin P. McCabe added a comment -

        +1. Thanks, Zhe Zhang.

        Show
        cmccabe Colin P. McCabe added a comment - +1. Thanks, Zhe Zhang .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 44s trunk passed
        +1 compile 0m 30s trunk passed with JDK v1.8.0_66
        +1 compile 0m 34s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 5s trunk passed
        +1 javadoc 0m 23s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 35s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.8.0_66
        +1 javac 0m 28s the patch passed
        +1 compile 0m 29s the patch passed with JDK v1.7.0_91
        +1 javac 0m 29s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 19s the patch passed
        +1 javadoc 0m 23s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 25s the patch passed with JDK v1.7.0_91
        +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        24m 17s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783200/HDFS-9576.07.patch
        JIRA Issue HDFS-9576
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0e65e5ba82da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1acc509
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14163/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14163/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 44s trunk passed +1 compile 0m 30s trunk passed with JDK v1.8.0_66 +1 compile 0m 34s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 5s trunk passed +1 javadoc 0m 23s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 35s the patch passed +1 compile 0m 28s the patch passed with JDK v1.8.0_66 +1 javac 0m 28s the patch passed +1 compile 0m 29s the patch passed with JDK v1.7.0_91 +1 javac 0m 29s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 0m 23s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_91 +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 24m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783200/HDFS-9576.07.patch JIRA Issue HDFS-9576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0e65e5ba82da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1acc509 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14163/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14163/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 56s trunk passed
        +1 compile 0m 30s trunk passed with JDK v1.8.0_66
        +1 compile 0m 29s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 11s trunk passed
        +1 findbugs 1m 55s trunk passed
        +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 24s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 27s the patch passed with JDK v1.8.0_66
        +1 javac 0m 26s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.7.0_91
        +1 javac 0m 26s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 8s the patch passed
        +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 21s the patch passed with JDK v1.7.0_91
        +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        22m 4s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783173/HDFS-9576.06.patch
        JIRA Issue HDFS-9576
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 48d55dbe18e8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 89d1fd5
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14160/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14160/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 56s trunk passed +1 compile 0m 30s trunk passed with JDK v1.8.0_66 +1 compile 0m 29s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 24s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 30s the patch passed +1 compile 0m 27s the patch passed with JDK v1.8.0_66 +1 javac 0m 26s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_91 +1 javac 0m 26s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_91 +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 22m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783173/HDFS-9576.06.patch JIRA Issue HDFS-9576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 48d55dbe18e8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89d1fd5 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14160/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14160/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Good catch Colin! Uploading v07 patch to address.

        Show
        zhz Zhe Zhang added a comment - Good catch Colin! Uploading v07 patch to address.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Thanks, Zhe Zhang.

        993	      int retLen = readWithStrategy(byteBufferReader, 0, buf.remaining());
        994	      if (retLen < buf.remaining()) {
        995	        dfsClient.addRetLenToReaderScope(scope, retLen);
        996	      }
        

        Hmm. Isn't buf#remaining going to change between lines 993 and 994? It seems like this should be loaded into a variable.

        Show
        cmccabe Colin P. McCabe added a comment - Thanks, Zhe Zhang . 993 int retLen = readWithStrategy(byteBufferReader, 0, buf.remaining()); 994 if (retLen < buf.remaining()) { 995 dfsClient.addRetLenToReaderScope(scope, retLen); 996 } Hmm. Isn't buf#remaining going to change between lines 993 and 994? It seems like this should be loaded into a variable.
        Hide
        zhz Zhe Zhang added a comment -

        Updating patch to address comments from Colin P. McCabe.

        Show
        zhz Zhe Zhang added a comment - Updating patch to address comments from Colin P. McCabe .
        Hide
        cmccabe Colin P. McCabe added a comment -

        So how about only reporting "returned length" if it's different from requested?

        Sure.

        Show
        cmccabe Colin P. McCabe added a comment - So how about only reporting "returned length" if it's different from requested? Sure.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks for the review Colin.

        Can we use "pos" instead?

        Good point. Will update in next rev.

        Tracing the "requested length" is mostly for profiling application behavior. I agree that the "returned length" is what ultimately matters in the particular scenario being traced. But "requested length" gives insights on what-if analysis – e.g. a different system could return more data and fully satisfy the request.

        So how about only reporting "returned length" if it's different from requested?

        Show
        zhz Zhe Zhang added a comment - Thanks for the review Colin. Can we use "pos" instead? Good point. Will update in next rev. Tracing the "requested length" is mostly for profiling application behavior. I agree that the "returned length" is what ultimately matters in the particular scenario being traced. But "requested length" gives insights on what-if analysis – e.g. a different system could return more data and fully satisfy the request. So how about only reporting "returned length" if it's different from requested?
        Hide
        cmccabe Colin P. McCabe added a comment -

        Cool. This sounds useful for profiling applications.

            scope.addKVAnnotation("position", Long.toString(pos));
        

        Can we use "pos" instead? It would help keep span sizes down.

        Hmm. I'm curious whether we need to separate request length and returned length, or if we can just have one length, which is the length actually read. It seems like if you see a lot of really short lengths, you can do some deeper digging to figure out why... but I'm not sure if it's worth carrying this info on every read trace span.

        Show
        cmccabe Colin P. McCabe added a comment - Cool. This sounds useful for profiling applications. scope.addKVAnnotation( "position" , Long .toString(pos)); Can we use "pos" instead? It would help keep span sizes down. Hmm. I'm curious whether we need to separate request length and returned length, or if we can just have one length, which is the length actually read. It seems like if you see a lot of really short lengths, you can do some deeper digging to figure out why... but I'm not sure if it's worth carrying this info on every read trace span.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 6s trunk passed
        +1 compile 0m 33s trunk passed with JDK v1.8.0_66
        +1 compile 0m 30s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 56s trunk passed
        +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 24s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 31s the patch passed with JDK v1.8.0_66
        +1 javac 0m 31s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.7.0_91
        +1 javac 0m 28s the patch passed
        -1 checkstyle 0m 12s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs-client (total was 135, now 135).
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 1s the patch passed
        +1 javadoc 0m 22s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 20s the patch passed with JDK v1.7.0_91
        +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        22m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781310/HDFS-9576.05.patch
        JIRA Issue HDFS-9576
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7e7d85f5a625 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0af2022
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14076/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14076/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14076/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 6s trunk passed +1 compile 0m 33s trunk passed with JDK v1.8.0_66 +1 compile 0m 30s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 24s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 30s the patch passed +1 compile 0m 31s the patch passed with JDK v1.8.0_66 +1 javac 0m 31s the patch passed +1 compile 0m 28s the patch passed with JDK v1.7.0_91 +1 javac 0m 28s the patch passed -1 checkstyle 0m 12s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs-client (total was 135, now 135). +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 1s the patch passed +1 javadoc 0m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 20s the patch passed with JDK v1.7.0_91 +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 22m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781310/HDFS-9576.05.patch JIRA Issue HDFS-9576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7e7d85f5a625 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0af2022 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14076/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14076/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14076/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Xiao! Updating patch to fix the checkstyle issue.

        Show
        zhz Zhe Zhang added a comment - Thanks Xiao! Updating patch to fix the checkstyle issue.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Zhe for the new rev. +1 (non-binding) from me once the checkstyle is fixed.

        Show
        xiaochen Xiao Chen added a comment - Thanks Zhe for the new rev. +1 (non-binding) from me once the checkstyle is fixed.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 33s trunk passed
        +1 compile 0m 26s trunk passed with JDK v1.8.0_66
        +1 compile 0m 29s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 48s trunk passed
        +1 javadoc 0m 19s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 24s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 29s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.8.0_66
        +1 javac 0m 23s the patch passed
        +1 compile 0m 27s the patch passed with JDK v1.7.0_91
        +1 javac 0m 27s the patch passed
        -1 checkstyle 0m 13s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs-client (total was 136, now 137).
        +1 mvnsite 0m 30s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 59s the patch passed
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91
        +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        21m 13s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780808/HDFS-9576.04.patch
        JIRA Issue HDFS-9576
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6c850e486918 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 67c9780
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14047/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14047/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14047/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 33s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_66 +1 compile 0m 29s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 0m 19s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 24s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 29s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_66 +1 javac 0m 23s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_91 +1 javac 0m 27s the patch passed -1 checkstyle 0m 13s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs-client (total was 136, now 137). +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91 +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 21m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780808/HDFS-9576.04.patch JIRA Issue HDFS-9576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6c850e486918 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67c9780 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14047/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14047/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14047/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 30s trunk passed
        +1 compile 0m 27s trunk passed with JDK v1.8.0_66
        +1 compile 0m 29s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 11s trunk passed
        +1 findbugs 1m 57s trunk passed
        +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 25s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.8.0_66
        +1 javac 0m 25s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.7.0_91
        +1 javac 0m 26s the patch passed
        -1 checkstyle 0m 13s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs-client (total was 136, now 137).
        +1 mvnsite 0m 33s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 0s the patch passed
        +1 javadoc 0m 25s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 23s the patch passed with JDK v1.7.0_91
        +1 unit 1m 3s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        22m 12s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780803/HDFS-9576.03.patch
        JIRA Issue HDFS-9576
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 68830c147581 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b993668
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14044/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14044/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14044/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 30s trunk passed +1 compile 0m 27s trunk passed with JDK v1.8.0_66 +1 compile 0m 29s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 31s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_66 +1 javac 0m 25s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_91 +1 javac 0m 26s the patch passed -1 checkstyle 0m 13s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs-client (total was 136, now 137). +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 0m 25s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_91 +1 unit 1m 3s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 22m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780803/HDFS-9576.03.patch JIRA Issue HDFS-9576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 68830c147581 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b993668 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14044/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14044/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14044/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Xiao for the good catch! Updating the patch to address. Also renaming readScope to scope to be consistent with other places using temporary scope variables.

        Show
        zhz Zhe Zhang added a comment - Thanks Xiao for the good catch! Updating the patch to address. Also renaming readScope to scope to be consistent with other places using temporary scope variables.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks for the work, and sorry for jumping in.

        scope.addKVAnnotation("requiredLength", Integer.toString(reqLen));
        

        Should the key be "requestedLength"?

        Show
        xiaochen Xiao Chen added a comment - Thanks for the work, and sorry for jumping in. scope.addKVAnnotation( "requiredLength" , Integer .toString(reqLen)); Should the key be "requestedLength"?
        Hide
        zhz Zhe Zhang added a comment -

        Good catch! Updating the patch to address.

        Show
        zhz Zhe Zhang added a comment - Good catch! Updating the patch to address.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        I agree to fix tracing of write in follow-up.

        The 02 patch looks good but 1 nit. The name of variable should not be ignored because it is used now.

        Show
        iwasakims Masatake Iwasaki added a comment - I agree to fix tracing of write in follow-up. The 02 patch looks good but 1 nit. The name of variable should not be ignored because it is used now.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 27s trunk passed
        +1 compile 0m 26s trunk passed with JDK v1.8.0_66
        +1 compile 0m 28s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 49s trunk passed
        +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 23s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.8.0_66
        +1 javac 0m 25s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.7.0_91
        +1 javac 0m 26s the patch passed
        -1 checkstyle 0m 12s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs-client (total was 136, now 137).
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 58s the patch passed
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 21s the patch passed with JDK v1.7.0_91
        +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        21m 15s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780610/HDFS-9576.02.patch
        JIRA Issue HDFS-9576
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 646b5b8cfacd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 28bd138
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14031/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14031/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14031/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 27s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_66 +1 compile 0m 28s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 23s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 30s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_66 +1 javac 0m 25s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_91 +1 javac 0m 26s the patch passed -1 checkstyle 0m 12s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs-client (total was 136, now 137). +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 58s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_91 +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 21m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780610/HDFS-9576.02.patch JIRA Issue HDFS-9576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 646b5b8cfacd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28bd138 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14031/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14031/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14031/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Attaching new patch to include returned length in read trace scope.

        It's a little tricky to update the write scope. We can either create a new scope on the public write level, or even move the current write scope there. I removed the write change in the new patch, will repurpose this JIRA to focus on read only. I'll open a separate JIRA for the write scope change.

        Show
        zhz Zhe Zhang added a comment - Attaching new patch to include returned length in read trace scope. It's a little tricky to update the write scope. We can either create a new scope on the public write level, or even move the current write scope there. I removed the write change in the new patch, will repurpose this JIRA to focus on read only. I'll open a separate JIRA for the write scope change.
        Hide
        zhz Zhe Zhang added a comment -

        Good point Masatake Iwasaki! I should have traced the write length one level up – the len variable in the public write method. This is to give insights on applications' I/O behavior. E.g. if in a cluster most applications send large write requests, certain optimizations can be made.

        In the next rev I'll also try to trace the return value of read.

        Show
        zhz Zhe Zhang added a comment - Good point Masatake Iwasaki ! I should have traced the write length one level up – the len variable in the public write method. This is to give insights on applications' I/O behavior. E.g. if in a cluster most applications send large write requests, certain optimizations can be made. In the next rev I'll also try to trace the return value of read.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Thanks for working on this, Zhe Zhang. I tried the 01 patch and checked that the annotations are properly set in spans.

        @@ -209,7 +209,7 @@ protected TraceScope createWriteTraceScope() {
           private void writeChecksumChunks(byte b[], int off, int len)
           throws IOException {
             sum.calculateChunkedSums(b, off, len, checksum, 0);
        -    TraceScope scope = createWriteTraceScope();
        +    TraceScope scope = createWriteTraceScope(len);
        

        The len of "DFSOutputStream#write" span is basically size of internal buffer of FSOutputSummer and constant (4608) most of the time. Do you have specific usecase for this?

        For read, if the length is useful, recording the number of bytes actually read (which is return value of read) as annotation might be helpful too.

        Show
        iwasakims Masatake Iwasaki added a comment - Thanks for working on this, Zhe Zhang . I tried the 01 patch and checked that the annotations are properly set in spans. @@ -209,7 +209,7 @@ protected TraceScope createWriteTraceScope() { private void writeChecksumChunks( byte b[], int off, int len) throws IOException { sum.calculateChunkedSums(b, off, len, checksum, 0); - TraceScope scope = createWriteTraceScope(); + TraceScope scope = createWriteTraceScope(len); The len of "DFSOutputStream#write" span is basically size of internal buffer of FSOutputSummer and constant (4608) most of the time. Do you have specific usecase for this? For read, if the length is useful, recording the number of bytes actually read (which is return value of read) as annotation might be helpful too.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 26s trunk passed
        +1 compile 7m 57s trunk passed with JDK v1.8.0_66
        +1 compile 8m 46s trunk passed with JDK v1.7.0_91
        +1 checkstyle 1m 1s trunk passed
        +1 mvnsite 1m 37s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 3m 41s trunk passed
        +1 javadoc 1m 17s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 29s trunk passed with JDK v1.7.0_91
        +1 mvninstall 2m 9s the patch passed
        +1 compile 8m 22s the patch passed with JDK v1.8.0_66
        +1 javac 8m 22s the patch passed
        +1 compile 9m 50s the patch passed with JDK v1.7.0_91
        +1 javac 9m 50s the patch passed
        -1 checkstyle 1m 10s Patch generated 1 new checkstyle issues in root (total was 173, now 173).
        +1 mvnsite 1m 50s the patch passed
        +1 mvneclipse 0m 29s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 4m 11s the patch passed
        +1 javadoc 1m 19s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 33s the patch passed with JDK v1.7.0_91
        -1 unit 19m 34s hadoop-common in the patch failed with JDK v1.8.0_66.
        +1 unit 0m 53s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        +1 unit 8m 52s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 unit 1m 6s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 28s Patch does not generate ASF License warnings.
        112m 36s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager
          hadoop.fs.TestLocalFsFCStatistics
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780370/HDFS-9576.01.patch
        JIRA Issue HDFS-9576
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4ecbe74b3b78 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1af2917
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14020/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14020/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14020/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14020/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: .
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14020/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 26s trunk passed +1 compile 7m 57s trunk passed with JDK v1.8.0_66 +1 compile 8m 46s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 1s trunk passed +1 mvnsite 1m 37s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 41s trunk passed +1 javadoc 1m 17s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 29s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 9s the patch passed +1 compile 8m 22s the patch passed with JDK v1.8.0_66 +1 javac 8m 22s the patch passed +1 compile 9m 50s the patch passed with JDK v1.7.0_91 +1 javac 9m 50s the patch passed -1 checkstyle 1m 10s Patch generated 1 new checkstyle issues in root (total was 173, now 173). +1 mvnsite 1m 50s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 11s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 33s the patch passed with JDK v1.7.0_91 -1 unit 19m 34s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 53s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 8m 52s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 1m 6s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 112m 36s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.fs.TestLocalFsFCStatistics JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780370/HDFS-9576.01.patch JIRA Issue HDFS-9576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4ecbe74b3b78 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1af2917 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14020/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14020/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14020/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14020/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14020/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Updating patch to fix the checkstyle issue.

        Show
        zhz Zhe Zhang added a comment - Updating patch to fix the checkstyle issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 6s trunk passed
        +1 compile 7m 45s trunk passed with JDK v1.8.0_66
        +1 compile 8m 38s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 56s trunk passed
        +1 mvnsite 1m 35s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 36s trunk passed
        +1 javadoc 1m 14s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 27s trunk passed with JDK v1.7.0_91
        +1 mvninstall 2m 9s the patch passed
        +1 compile 8m 10s the patch passed with JDK v1.8.0_66
        +1 javac 8m 10s the patch passed
        +1 compile 8m 44s the patch passed with JDK v1.7.0_91
        +1 javac 8m 44s the patch passed
        -1 checkstyle 0m 58s Patch generated 2 new checkstyle issues in root (total was 173, now 174).
        +1 mvnsite 1m 35s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 54s the patch passed
        +1 javadoc 1m 16s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 28s the patch passed with JDK v1.7.0_91
        +1 unit 7m 17s hadoop-common in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        +1 unit 7m 29s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        80m 34s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779168/HDFS-9576.00.patch
        JIRA Issue HDFS-9576
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b76948b5e54b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / df83230
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13982/artifact/patchprocess/diff-checkstyle-root.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13982/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: .
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13982/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 6s trunk passed +1 compile 7m 45s trunk passed with JDK v1.8.0_66 +1 compile 8m 38s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 56s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 36s trunk passed +1 javadoc 1m 14s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 9s the patch passed +1 compile 8m 10s the patch passed with JDK v1.8.0_66 +1 javac 8m 10s the patch passed +1 compile 8m 44s the patch passed with JDK v1.7.0_91 +1 javac 8m 44s the patch passed -1 checkstyle 0m 58s Patch generated 2 new checkstyle issues in root (total was 173, now 174). +1 mvnsite 1m 35s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 54s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 28s the patch passed with JDK v1.7.0_91 +1 unit 7m 17s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 7m 29s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 80m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779168/HDFS-9576.00.patch JIRA Issue HDFS-9576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b76948b5e54b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / df83230 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13982/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13982/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13982/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Attaching initial patch.

        It actually doesn't make sense to add position info when tracing write requests, because they are always sequential. The patch adds a position field in every read request (stateful or positional). An alternative is to avoid the filed for stateful reads, and meanwhile also trace seek calls. But in probabilistic tracing, we might miss seek calls.

        Colin P. McCabe, Stack Could you take a look? Thanks a lot.

        Show
        zhz Zhe Zhang added a comment - Attaching initial patch. It actually doesn't make sense to add position info when tracing write requests, because they are always sequential. The patch adds a position field in every read request (stateful or positional). An alternative is to avoid the filed for stateful reads, and meanwhile also trace seek calls. But in probabilistic tracing, we might miss seek calls. Colin P. McCabe , Stack Could you take a look? Thanks a lot.

          People

          • Assignee:
            zhz Zhe Zhang
            Reporter:
            zhz Zhe Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development