Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: erasure-coding
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Native erasure coder prefers to direct ByteBuffer for performance consideration. To prepare for it, this change uses ByteBuffer through the codes in implementing striping position read. It will also fix avoiding unnecessary data copying between striping read chunk buffers and decode input buffers.

      1. HDFS-8901.v11.patch
        29 kB
        Youwei Wang
      2. HDFS-8901.v12.patch
        38 kB
        Youwei Wang
      3. HDFS-8901.v13.patch
        5 kB
        Youwei Wang
      4. HDFS-8901.v14.patch
        30 kB
        Youwei Wang
      5. HDFS-8901.v15.patch
        35 kB
        SammiChen
      6. HDFS-8901.v16.patch
        35 kB
        SammiChen
      7. HDFS-8901-v10.patch
        32 kB
        Kai Zheng
      8. HDFS-8901-v17.patch
        35 kB
        SammiChen
      9. HDFS-8901-v18.patch
        43 kB
        SammiChen
      10. HDFS-8901-v19.patch
        43 kB
        SammiChen
      11. HDFS-8901-v2.patch
        28 kB
        Kai Zheng
      12. HDFS-8901-v20.patch
        37 kB
        SammiChen
      13. HDFS-8901-v3.patch
        27 kB
        Kai Zheng
      14. HDFS-8901-v4.patch
        32 kB
        Kai Zheng
      15. HDFS-8901-v5.patch
        30 kB
        Kai Zheng
      16. HDFS-8901-v6.patch
        29 kB
        Kai Zheng
      17. HDFS-8901-v7.patch
        29 kB
        Kai Zheng
      18. HDFS-8901-v8.patch
        31 kB
        Li Bo
      19. HDFS-8901-v9.patch
        32 kB
        Li Bo
      20. initial-poc.patch
        26 kB
        Kai Zheng

        Issue Links

          Activity

          Hide
          drankye Kai Zheng added a comment -

          Uploaded the initial codes to show the idea.

          Show
          drankye Kai Zheng added a comment - Uploaded the initial codes to show the idea.
          Hide
          drankye Kai Zheng added a comment -

          Updated a patch that depends on HDFS-8905

          Show
          drankye Kai Zheng added a comment - Updated a patch that depends on HDFS-8905
          Hide
          walter.k.su Walter Su added a comment -

          (from description) this change uses ByteBuffer through the codes in implementing striping position read. It will also fix avoiding unnecessary data copying between striping read chunk buffers and decode input buffers.

          I think, for PositionStripeReader, the data copy from buf[](given by caller) to decodeInputs[][] is unavoidable. An AlignedStripe actually merged multiple real stripes. I mean if cellsize is 64kb, a real stripe is 6*64kb, an AlignedStripe could be n*6*64kb (assume given buf[] is larger than that). decodeInputs[0][n*64k] is copied from n positions from buf[].

          I understand native erasure coder prefers direct ByteBuffer. So we can make decodeInputs[] a direct ByteBuffer array. Again we copy data from buf[] to decodeInputs[] (off-heap bytebuffers).

          So I think the changes to PositionStripeReader are good, and changes to DFSInputStream are unnecessary.

          Show
          walter.k.su Walter Su added a comment - (from description) this change uses ByteBuffer through the codes in implementing striping position read. It will also fix avoiding unnecessary data copying between striping read chunk buffers and decode input buffers. I think, for PositionStripeReader, the data copy from buf[](given by caller) to decodeInputs[][] is unavoidable. An AlignedStripe actually merged multiple real stripes. I mean if cellsize is 64kb, a real stripe is 6*64kb, an AlignedStripe could be n*6*64kb (assume given buf[] is larger than that). decodeInputs[0][n*64k] is copied from n positions from buf[]. I understand native erasure coder prefers direct ByteBuffer. So we can make decodeInputs[] a direct ByteBuffer array. Again we copy data from buf[] to decodeInputs[] (off-heap bytebuffers). So I think the changes to PositionStripeReader are good, and changes to DFSInputStream are unnecessary.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Walter for the review!

          1. If the read buffer from upper call can contain n (equal or larger than 1) cells, then the n cells based on the read buffer will be used as references into the decode buffers, thus avoids data copy. Sure not all data copy are avoided, in the worst case, if the user provided read buffer isn't the right buffer type desired by the erasure coder, then all cells will need one time copy.

          2. This change is mainly to convert byte array to bytebuffer for positional read. Using appropriate bytebuffer type according to erasure coder will be handled in HDFS-8668 family.

          3. The change into DFSInputStream is because this patch was based on HDFS-8905. You're right the relying isn't essential.

          Show
          drankye Kai Zheng added a comment - Thanks Walter for the review! 1. If the read buffer from upper call can contain n (equal or larger than 1) cells, then the n cells based on the read buffer will be used as references into the decode buffers, thus avoids data copy. Sure not all data copy are avoided, in the worst case, if the user provided read buffer isn't the right buffer type desired by the erasure coder, then all cells will need one time copy. 2. This change is mainly to convert byte array to bytebuffer for positional read. Using appropriate bytebuffer type according to erasure coder will be handled in HDFS-8668 family. 3. The change into DFSInputStream is because this patch was based on HDFS-8905 . You're right the relying isn't essential.
          Hide
          walter.k.su Walter Su added a comment -

          in the worst case, if the user provided read buffer isn't the right buffer type desired by the erasure coder, then all cells will need one time copy.

          I think the buffer given by caller is always byte[]? There's no bytebuffer version of pread API.

          public int read(long position, byte[] buffer, int offset, int length)
          

          Besides, Even if the caller provides a bytebuffer, the data copy from byte[] buffer to decodeInputs still needed.

          Assume schema is 3+2, then

          byte[] buf is      64k 64k 64k 64k 64k 64k 64k 64k 64k ...
          decodeInputs[0] is 64k         64k         64k         ...
          decodeInputs[1] is     64k         64k         64k     ...
          decodeInputs[2] is         64k         64k         64k ...
          

          We can't feed encoder the original byte[] buffer and only call encoding one time.
          We can encode in one time if we provide decodeInputs, then we need data copy.
          We can also encode many times stripe by stripe like stateful read, then data copy isn't needed.

          We are not pursuing fewer times of data copy since the performance is bounded by IO speed and coding speed.

          Show
          walter.k.su Walter Su added a comment - in the worst case, if the user provided read buffer isn't the right buffer type desired by the erasure coder, then all cells will need one time copy. I think the buffer given by caller is always byte[]? There's no bytebuffer version of pread API. public int read( long position, byte [] buffer, int offset, int length) Besides, Even if the caller provides a bytebuffer, the data copy from byte[] buffer to decodeInputs still needed. Assume schema is 3+2, then byte[] buf is 64k 64k 64k 64k 64k 64k 64k 64k 64k ... decodeInputs[0] is 64k 64k 64k ... decodeInputs[1] is 64k 64k 64k ... decodeInputs[2] is 64k 64k 64k ... We can't feed encoder the original byte[] buffer and only call encoding one time. We can encode in one time if we provide decodeInputs, then we need data copy. We can also encode many times stripe by stripe like stateful read, then data copy isn't needed. We are not pursuing fewer times of data copy since the performance is bounded by IO speed and coding speed.
          Hide
          walter.k.su Walter Su added a comment -

          encode decode

          Show
          walter.k.su Walter Su added a comment - encode decode
          Hide
          drankye Kai Zheng added a comment -

          There's no bytebuffer version of pread API.

          Yeah right. For the byte array version, the passed byte array will be wrapped into a heap bytebuffer. When Java coder is used, then the heap bytebuffer will match the bytebuffer type the coder desires.

          We can also encode many times stripe by stripe like stateful read, then data copy isn't needed.

          In your illustration, given the buf is of 64k 64k 64k 64k 64k 64k 64k 64k 64k ... (+9 cells), in this change, the whole buffer is divided into +9 cells wrapping into +9 bytebuffers, these bytebuffers can be referenced into decodeInputs and used for decoding. These derived buffers are not affected in content. The +9 cells are divided into 3 strips, each strip being decoded a time. Is there anything missed here? Please clarify. Thanks.

          We are not pursuing fewer times of data copy since the performance is bounded by IO speed and coding speed.

          Indicated by our latest benchmark tests, coding computation isn't the concern. In high speed network and disk, in some case (before hitting the limit) IO may also not be the bottleneck. I agree data copy isn't the key goal we're working here, I just did it and don't find the overhead in the way.

          Show
          drankye Kai Zheng added a comment - There's no bytebuffer version of pread API. Yeah right. For the byte array version, the passed byte array will be wrapped into a heap bytebuffer. When Java coder is used, then the heap bytebuffer will match the bytebuffer type the coder desires. We can also encode many times stripe by stripe like stateful read, then data copy isn't needed. In your illustration, given the buf is of 64k 64k 64k 64k 64k 64k 64k 64k 64k ... (+9 cells), in this change, the whole buffer is divided into +9 cells wrapping into +9 bytebuffers, these bytebuffers can be referenced into decodeInputs and used for decoding. These derived buffers are not affected in content. The +9 cells are divided into 3 strips, each strip being decoded a time. Is there anything missed here? Please clarify. Thanks. We are not pursuing fewer times of data copy since the performance is bounded by IO speed and coding speed. Indicated by our latest benchmark tests, coding computation isn't the concern. In high speed network and disk, in some case (before hitting the limit) IO may also not be the bottleneck. I agree data copy isn't the key goal we're working here, I just did it and don't find the overhead in the way.
          Hide
          walter.k.su Walter Su added a comment -

          The +9 cells are divided into 3 strips, each strip being decoded a time.

          If that's the plan, we can remove PositionStripeReader, and use StatefulStripeReader for pread. StatefulStripeReader is good at reading stripe by stripe.

          Show
          walter.k.su Walter Su added a comment - The +9 cells are divided into 3 strips, each strip being decoded a time. If that's the plan, we can remove PositionStripeReader, and use StatefulStripeReader for pread. StatefulStripeReader is good at reading stripe by stripe.
          Hide
          drankye Kai Zheng added a comment -

          If that's the plan, ...

          Yeah, it was planned and the patch worked the way.

          we can remove PositionStripeReader, and use StatefulStripeReader for pread. StatefulStripeReader is good at reading stripe by stripe.

          It's a good idea and same thought with me . In fact I did it in HDFS-8957, where I combined the two into a same path around that place. That patch is relying on this one and ready for review. Maybe you could also take a look? Thanks.

          Show
          drankye Kai Zheng added a comment - If that's the plan, ... Yeah, it was planned and the patch worked the way. we can remove PositionStripeReader, and use StatefulStripeReader for pread. StatefulStripeReader is good at reading stripe by stripe. It's a good idea and same thought with me . In fact I did it in HDFS-8957 , where I combined the two into a same path around that place. That patch is relying on this one and ready for review. Maybe you could also take a look? Thanks.
          Hide
          drankye Kai Zheng added a comment -

          Updated the patch avoiding using codes from HDFS-8905 to make this one independent.

          Show
          drankye Kai Zheng added a comment - Updated the patch avoiding using codes from HDFS-8905 to make this one independent.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 10s trunk passed
          +1 compile 2m 9s trunk passed with JDK v1.8.0_66
          +1 compile 1m 58s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 46s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 4m 33s trunk passed
          +1 javadoc 1m 55s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 47s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 42s the patch passed
          +1 compile 2m 1s the patch passed with JDK v1.8.0_66
          +1 javac 2m 1s the patch passed
          +1 compile 1m 56s the patch passed with JDK v1.7.0_91
          +1 javac 1m 56s the patch passed
          -1 checkstyle 0m 27s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 158, now 159).
          +1 mvnsite 1m 44s the patch passed
          +1 mvneclipse 0m 30s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 59s the patch passed
          +1 javadoc 1m 49s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 41s the patch passed with JDK v1.7.0_91
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 174m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 4m 45s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 80m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 2m 50s Patch generated 56 ASF License warnings.
          313m 28s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.server.namenode.TestDeleteRace
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.TestDFSStripedInputStream
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.server.namenode.TestMetaSave
            hadoop.hdfs.server.namenode.ha.TestHAAppend
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead
            org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            org.apache.hadoop.hdfs.TestClientReportBadBlock
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.TestParallelUnixDomainRead
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.TestQuota
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDFSStripedInputStream
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestLeaseRecovery
          JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.tools.TestGetConf
            org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.tools.TestGetGroups
            org.apache.hadoop.hdfs.TestClientReportBadBlock
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.tools.TestDFSAdmin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775730/HDFS-8901-v3.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f7979e9dee1a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e84d6ca
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13767/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13767/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 10s trunk passed +1 compile 2m 9s trunk passed with JDK v1.8.0_66 +1 compile 1m 58s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 4m 33s trunk passed +1 javadoc 1m 55s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 47s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 42s the patch passed +1 compile 2m 1s the patch passed with JDK v1.8.0_66 +1 javac 2m 1s the patch passed +1 compile 1m 56s the patch passed with JDK v1.7.0_91 +1 javac 1m 56s the patch passed -1 checkstyle 0m 27s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 158, now 159). +1 mvnsite 1m 44s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 59s the patch passed +1 javadoc 1m 49s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 41s the patch passed with JDK v1.7.0_91 +1 unit 1m 8s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 174m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 4m 45s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 80m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 2m 50s Patch generated 56 ASF License warnings. 313m 28s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.hdfs.server.namenode.ha.TestHAAppend JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead   org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   org.apache.hadoop.hdfs.TestClientReportBadBlock   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.TestParallelUnixDomainRead JDK v1.7.0_91 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestQuota   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestLeaseRecovery JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.tools.TestGetConf   org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.tools.TestGetGroups   org.apache.hadoop.hdfs.TestClientReportBadBlock   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.tools.TestDFSAdmin Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775730/HDFS-8901-v3.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f7979e9dee1a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e84d6ca findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13767/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13767/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13767/console This message was automatically generated.
          Show
          zhz Zhe Zhang added a comment - Some EC related test failures look possibly related to the change, like https://builds.apache.org/job/PreCommit-HDFS-Build/13767/testReport/org.apache.hadoop.hdfs/TestDFSStripedInputStream/testPreadWithDNFailure/
          Hide
          drankye Kai Zheng added a comment -

          Thanks Zhe Zhang for the check.
          Updated the patch:
          1. Fixed the test failure by bringing back necessary codes from HDFS-8905;
          2. Cleaned up some old check style issues;
          3. Refined a bit the TestDFSStripedInputStream test.

          Show
          drankye Kai Zheng added a comment - Thanks Zhe Zhang for the check. Updated the patch: 1. Fixed the test failure by bringing back necessary codes from HDFS-8905 ; 2. Cleaned up some old check style issues; 3. Refined a bit the TestDFSStripedInputStream test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 10m 7s trunk passed
          +1 compile 2m 16s trunk passed with JDK v1.8.0_66
          +1 compile 2m 11s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 2m 1s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 4m 44s trunk passed
          +1 javadoc 2m 16s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 4s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 7s the patch passed
          +1 compile 3m 23s the patch passed with JDK v1.8.0_66
          +1 javac 3m 23s the patch passed
          +1 compile 2m 25s the patch passed with JDK v1.7.0_91
          +1 javac 2m 26s the patch passed
          -1 checkstyle 0m 30s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 158, now 150).
          +1 mvnsite 2m 9s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 6m 21s the patch passed
          +1 javadoc 2m 23s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 0s the patch passed with JDK v1.7.0_91
          +1 unit 1m 16s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 94m 31s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 4m 21s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 266m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 4m 41s Patch generated 56 ASF License warnings.
          429m 13s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.TestAclsEndToEnd
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead
            org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestRecoverStripedFile
            org.apache.hadoop.hdfs.TestDFSStripedOutputStream
            org.apache.hadoop.hdfs.TestAppendSnapshotTruncate
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.TestParallelUnixDomainRead
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
            hadoop.hdfs.TestFileLengthOnClusterRestart
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestWriteConfigurationToDFS
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestDFSInputStream
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestFileConcurrentReader
            hadoop.hdfs.qjournal.TestNNWithQJM
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
            org.apache.hadoop.hdfs.TestParallelShortCircuitRead
            org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
            org.apache.hadoop.hdfs.TestAppendSnapshotTruncate
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            org.apache.hadoop.hdfs.TestClientReportBadBlock
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.TestParallelUnixDomainRead
            org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776057/HDFS-8901-v4.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5af28dde697d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 65f3952
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13789/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13789/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 10m 7s trunk passed +1 compile 2m 16s trunk passed with JDK v1.8.0_66 +1 compile 2m 11s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 2m 1s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 4m 44s trunk passed +1 javadoc 2m 16s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 4s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 7s the patch passed +1 compile 3m 23s the patch passed with JDK v1.8.0_66 +1 javac 3m 23s the patch passed +1 compile 2m 25s the patch passed with JDK v1.7.0_91 +1 javac 2m 26s the patch passed -1 checkstyle 0m 30s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 158, now 150). +1 mvnsite 2m 9s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 6m 21s the patch passed +1 javadoc 2m 23s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 0s the patch passed with JDK v1.7.0_91 +1 unit 1m 16s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 94m 31s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 4m 21s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 266m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 4m 41s Patch generated 56 ASF License warnings. 429m 13s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead   org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestRecoverStripedFile   org.apache.hadoop.hdfs.TestDFSStripedOutputStream   org.apache.hadoop.hdfs.TestAppendSnapshotTruncate   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.TestParallelUnixDomainRead JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestWriteConfigurationToDFS   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestDFSInputStream   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead   org.apache.hadoop.hdfs.TestParallelShortCircuitRead   org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding   org.apache.hadoop.hdfs.TestAppendSnapshotTruncate   org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   org.apache.hadoop.hdfs.TestClientReportBadBlock   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.TestParallelUnixDomainRead   org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776057/HDFS-8901-v4.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5af28dde697d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 65f3952 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13789/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13789/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13789/console This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Rebased.

          Show
          drankye Kai Zheng added a comment - Rebased.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 10m 43s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 7m 43s trunk passed
          +1 compile 1m 37s trunk passed with JDK v1.8.0_74
          +1 compile 1m 25s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 39s trunk passed
          +1 javadoc 1m 43s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 17s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 22s the patch passed
          +1 compile 1m 44s the patch passed with JDK v1.8.0_74
          +1 javac 1m 44s the patch passed
          +1 compile 1m 27s the patch passed with JDK v1.7.0_95
          +1 javac 1m 27s the patch passed
          -1 checkstyle 0m 25s hadoop-hdfs-project: patch generated 6 new + 141 unchanged - 8 fixed = 147 total (was 149)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) with tabs.
          +1 findbugs 4m 5s the patch passed
          +1 javadoc 1m 38s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 15s the patch passed with JDK v1.7.0_95
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74.
          -1 unit 177m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 3m 41s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 258m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 asflicense 3m 44s Patch generated 1 ASF License warnings.
          495m 32s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead
            org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.TestClientReportBadBlock
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
            org.apache.hadoop.hdfs.TestParallelUnixDomainRead
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.namenode.TestFSNamesystemMBean
            hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
            org.apache.hadoop.hdfs.TestParallelShortCircuitRead
            org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.server.namenode.TestEditLog
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestAppendSnapshotTruncate
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.TestClientReportBadBlock
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
            org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            org.apache.hadoop.hdfs.TestParallelUnixDomainRead



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793679/HDFS-8901-v5.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 648e53fb0659 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6529c875
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14828/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14828/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 43s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 7m 43s trunk passed +1 compile 1m 37s trunk passed with JDK v1.8.0_74 +1 compile 1m 25s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 39s trunk passed +1 javadoc 1m 43s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 17s trunk passed with JDK v1.7.0_95 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 1m 44s the patch passed with JDK v1.8.0_74 +1 javac 1m 44s the patch passed +1 compile 1m 27s the patch passed with JDK v1.7.0_95 +1 javac 1m 27s the patch passed -1 checkstyle 0m 25s hadoop-hdfs-project: patch generated 6 new + 141 unchanged - 8 fixed = 147 total (was 149) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 21s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) with tabs. +1 findbugs 4m 5s the patch passed +1 javadoc 1m 38s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 15s the patch passed with JDK v1.7.0_95 +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. -1 unit 177m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 3m 41s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 258m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 asflicense 3m 44s Patch generated 1 ASF License warnings. 495m 32s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead   org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.TestClientReportBadBlock   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart   org.apache.hadoop.hdfs.TestParallelUnixDomainRead JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.TestFSNamesystemMBean   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead   org.apache.hadoop.hdfs.TestParallelShortCircuitRead   org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.server.namenode.TestEditLog   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestAppendSnapshotTruncate   org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.TestClientReportBadBlock   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart   org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   org.apache.hadoop.hdfs.TestParallelUnixDomainRead Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793679/HDFS-8901-v5.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 648e53fb0659 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6529c875 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14828/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14828/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14828/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Rebased.

          Show
          drankye Kai Zheng added a comment - Rebased.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 53s trunk passed
          +1 compile 1m 21s trunk passed with JDK v1.8.0_77
          +1 compile 1m 27s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 47s trunk passed
          +1 javadoc 1m 28s trunk passed with JDK v1.8.0_77
          +1 javadoc 2m 16s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 1m 16s the patch passed with JDK v1.8.0_77
          +1 javac 1m 16s the patch passed
          +1 compile 1m 22s the patch passed with JDK v1.7.0_95
          +1 javac 1m 22s the patch passed
          -1 checkstyle 0m 26s hadoop-hdfs-project: patch generated 6 new + 141 unchanged - 8 fixed = 147 total (was 149)
          +1 mvnsite 1m 25s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) with tabs.
          +1 findbugs 4m 12s the patch passed
          +1 javadoc 1m 22s the patch passed with JDK v1.8.0_77
          +1 javadoc 2m 12s the patch passed with JDK v1.7.0_95
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          -1 unit 121m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 3m 57s hadoop-hdfs-client in the patch failed with JDK v1.7.0_95.
          -1 unit 218m 55s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 3m 32s Patch does not generate ASF License warnings.
          386m 46s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestLocalDFS
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead
            org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.TestClientReportBadBlock
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
            org.apache.hadoop.hdfs.TestParallelUnixDomainRead
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.util.TestByteArrayManager
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.web.TestWebHdfsTokens
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead
            org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestAppendSnapshotTruncate
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.TestClientReportBadBlock
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
            org.apache.hadoop.hdfs.TestParallelUnixDomainRead



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797262/HDFS-8901-v6.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 276667d74703 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 21eb428
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15088/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15088/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 53s trunk passed +1 compile 1m 21s trunk passed with JDK v1.8.0_77 +1 compile 1m 27s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 47s trunk passed +1 javadoc 1m 28s trunk passed with JDK v1.8.0_77 +1 javadoc 2m 16s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 1m 16s the patch passed with JDK v1.8.0_77 +1 javac 1m 16s the patch passed +1 compile 1m 22s the patch passed with JDK v1.7.0_95 +1 javac 1m 22s the patch passed -1 checkstyle 0m 26s hadoop-hdfs-project: patch generated 6 new + 141 unchanged - 8 fixed = 147 total (was 149) +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 23s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) with tabs. +1 findbugs 4m 12s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_77 +1 javadoc 2m 12s the patch passed with JDK v1.7.0_95 +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. -1 unit 121m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 3m 57s hadoop-hdfs-client in the patch failed with JDK v1.7.0_95. -1 unit 218m 55s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 3m 32s Patch does not generate ASF License warnings. 386m 46s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.TestHFlush   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestLocalDFS   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead   org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.TestClientReportBadBlock   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart   org.apache.hadoop.hdfs.TestParallelUnixDomainRead JDK v1.7.0_95 Failed junit tests hadoop.hdfs.util.TestByteArrayManager   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.balancer.TestBalancer   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead   org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestAppendSnapshotTruncate   org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.TestClientReportBadBlock   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart   org.apache.hadoop.hdfs.TestParallelUnixDomainRead Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797262/HDFS-8901-v6.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 276667d74703 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 21eb428 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15088/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15088/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15088/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Will check the test failures and fix the check styles.

          Show
          drankye Kai Zheng added a comment - Will check the test failures and fix the check styles.
          Hide
          drankye Kai Zheng added a comment -

          Fixed a forever loop bug in the patch that causes the many failures, and the reported check styles. Also rebased.

          Show
          drankye Kai Zheng added a comment - Fixed a forever loop bug in the patch that causes the many failures, and the reported check styles. Also rebased.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 10m 35s trunk passed
          +1 compile 2m 39s trunk passed with JDK v1.8.0_77
          +1 compile 2m 16s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 2m 20s trunk passed
          +1 mvneclipse 0m 45s trunk passed
          +1 findbugs 5m 38s trunk passed
          +1 javadoc 2m 37s trunk passed with JDK v1.8.0_77
          +1 javadoc 3m 29s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 4s the patch passed
          +1 compile 2m 40s the patch passed with JDK v1.8.0_77
          +1 javac 2m 40s the patch passed
          +1 compile 2m 16s the patch passed with JDK v1.7.0_95
          +1 javac 2m 16s the patch passed
          +1 checkstyle 0m 43s hadoop-hdfs-project: patch generated 0 new + 141 unchanged - 8 fixed = 141 total (was 149)
          +1 mvnsite 2m 16s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 6m 5s the patch passed
          +1 javadoc 2m 22s the patch passed with JDK v1.8.0_77
          +1 javadoc 3m 30s the patch passed with JDK v1.7.0_95
          +1 unit 1m 38s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          -1 unit 101m 3s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          +1 unit 1m 23s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 74m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 46s Patch does not generate ASF License warnings.
          238m 31s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.TestReconstructStripedFile
            hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.TestSecureNameNode
            hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestFileCreation
            org.apache.hadoop.hdfs.TestDFSClientRetries
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797646/HDFS-8901-v7.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9e904ec4431a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e82f961
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15108/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15108/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15108/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15108/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15108/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15108/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 10m 35s trunk passed +1 compile 2m 39s trunk passed with JDK v1.8.0_77 +1 compile 2m 16s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 45s trunk passed +1 mvnsite 2m 20s trunk passed +1 mvneclipse 0m 45s trunk passed +1 findbugs 5m 38s trunk passed +1 javadoc 2m 37s trunk passed with JDK v1.8.0_77 +1 javadoc 3m 29s trunk passed with JDK v1.7.0_95 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 4s the patch passed +1 compile 2m 40s the patch passed with JDK v1.8.0_77 +1 javac 2m 40s the patch passed +1 compile 2m 16s the patch passed with JDK v1.7.0_95 +1 javac 2m 16s the patch passed +1 checkstyle 0m 43s hadoop-hdfs-project: patch generated 0 new + 141 unchanged - 8 fixed = 141 total (was 149) +1 mvnsite 2m 16s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 6m 5s the patch passed +1 javadoc 2m 22s the patch passed with JDK v1.8.0_77 +1 javadoc 3m 30s the patch passed with JDK v1.7.0_95 +1 unit 1m 38s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. -1 unit 101m 3s hadoop-hdfs in the patch failed with JDK v1.8.0_77. +1 unit 1m 23s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 74m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 46s Patch does not generate ASF License warnings. 238m 31s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestFileCreation   org.apache.hadoop.hdfs.TestDFSClientRetries   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797646/HDFS-8901-v7.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9e904ec4431a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e82f961 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15108/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15108/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15108/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15108/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15108/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15108/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          Thanks Kai Zheng for the good work, it looks like the test case failure is related. Please take a look at it.

          org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead.testReadWithRemoteBlockReader

          java.lang.UnsupportedOperationException: readDirect unsupported in RemoteBlockReader
          	at org.apache.hadoop.hdfs.RemoteBlockReader.read(RemoteBlockReader.java:481)
          	at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1204)
          	at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1138)
          
          Show
          rakeshr Rakesh R added a comment - Thanks Kai Zheng for the good work, it looks like the test case failure is related. Please take a look at it. org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead.testReadWithRemoteBlockReader java.lang.UnsupportedOperationException: readDirect unsupported in RemoteBlockReader at org.apache.hadoop.hdfs.RemoteBlockReader.read(RemoteBlockReader.java:481) at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1204) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1138)
          Hide
          drankye Kai Zheng added a comment -

          Thanks Rakesh R for identifying the failure. Yes I will investigate it.

          Show
          drankye Kai Zheng added a comment - Thanks Rakesh R for identifying the failure. Yes I will investigate it.
          Hide
          libo-intel Li Bo added a comment -

          rebase the patch to latest code;
          fix the unsupported method problem

          Show
          libo-intel Li Bo added a comment - rebase the patch to latest code; fix the unsupported method problem
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 22s Maven dependency ordering for branch
          +1 mvninstall 6m 53s trunk passed
          +1 compile 1m 18s trunk passed with JDK v1.8.0_91
          +1 compile 1m 24s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 36s trunk passed
          +1 javadoc 1m 23s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 9s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 1m 9s the patch passed with JDK v1.8.0_91
          +1 javac 1m 9s the patch passed
          +1 compile 1m 17s the patch passed with JDK v1.7.0_95
          +1 javac 1m 17s the patch passed
          +1 checkstyle 0m 26s hadoop-hdfs-project: patch generated 0 new + 165 unchanged - 9 fixed = 165 total (was 174)
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 8s the patch passed
          +1 javadoc 1m 25s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 7s the patch passed with JDK v1.7.0_95
          +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_91.
          -1 unit 60m 59s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
          +1 unit 1m 5s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 61m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          160m 24s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.TestFileAppend
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestParallelShortCircuitLegacyRead



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802148/HDFS-8901-v8.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 838720e9269e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 355325b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15357/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15357/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 22s Maven dependency ordering for branch +1 mvninstall 6m 53s trunk passed +1 compile 1m 18s trunk passed with JDK v1.8.0_91 +1 compile 1m 24s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 36s trunk passed +1 javadoc 1m 23s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 9s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 9s the patch passed with JDK v1.8.0_91 +1 javac 1m 9s the patch passed +1 compile 1m 17s the patch passed with JDK v1.7.0_95 +1 javac 1m 17s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project: patch generated 0 new + 165 unchanged - 9 fixed = 165 total (was 174) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 8s the patch passed +1 javadoc 1m 25s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 7s the patch passed with JDK v1.7.0_95 +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_91. -1 unit 60m 59s hadoop-hdfs in the patch failed with JDK v1.8.0_91. +1 unit 1m 5s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 61m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 160m 24s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.TestFileAppend JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestParallelShortCircuitLegacyRead Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802148/HDFS-8901-v8.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 838720e9269e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 355325b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15357/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15357/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          libo-intel Li Bo added a comment -

          fix the bugs in TestParalledShortCircuitLegacyRead

          Show
          libo-intel Li Bo added a comment - fix the bugs in TestParalledShortCircuitLegacyRead
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 6s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 5m 46s trunk passed with JDK v1.8.0_91
          +1 compile 6m 51s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 8s trunk passed
          +1 mvnsite 2m 19s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 5m 5s trunk passed
          +1 javadoc 2m 18s trunk passed with JDK v1.8.0_91
          +1 javadoc 3m 13s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 59s the patch passed
          +1 compile 5m 36s the patch passed with JDK v1.8.0_91
          +1 javac 5m 36s the patch passed
          +1 compile 6m 40s the patch passed with JDK v1.7.0_95
          +1 javac 6m 40s the patch passed
          +1 checkstyle 1m 9s root: patch generated 0 new + 298 unchanged - 9 fixed = 298 total (was 307)
          +1 mvnsite 2m 28s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 54s the patch passed
          +1 javadoc 2m 21s the patch passed with JDK v1.8.0_91
          +1 javadoc 3m 16s the patch passed with JDK v1.7.0_95
          +1 unit 7m 57s hadoop-common in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_91.
          +1 unit 57m 59s hadoop-hdfs in the patch passed with JDK v1.8.0_91.
          +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 unit 54m 59s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          198m 53s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802643/HDFS-8901-v9.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 994e49eb1d21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2835f14
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15384/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15384/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 6s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 5m 46s trunk passed with JDK v1.8.0_91 +1 compile 6m 51s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 8s trunk passed +1 mvnsite 2m 19s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 5m 5s trunk passed +1 javadoc 2m 18s trunk passed with JDK v1.8.0_91 +1 javadoc 3m 13s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 59s the patch passed +1 compile 5m 36s the patch passed with JDK v1.8.0_91 +1 javac 5m 36s the patch passed +1 compile 6m 40s the patch passed with JDK v1.7.0_95 +1 javac 6m 40s the patch passed +1 checkstyle 1m 9s root: patch generated 0 new + 298 unchanged - 9 fixed = 298 total (was 307) +1 mvnsite 2m 28s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 54s the patch passed +1 javadoc 2m 21s the patch passed with JDK v1.8.0_91 +1 javadoc 3m 16s the patch passed with JDK v1.7.0_95 +1 unit 7m 57s hadoop-common in the patch passed with JDK v1.8.0_91. +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_91. +1 unit 57m 59s hadoop-hdfs in the patch passed with JDK v1.8.0_91. +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 54m 59s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 198m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802643/HDFS-8901-v9.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 994e49eb1d21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2835f14 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15384/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15384/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Bo for the update, and fixing of the test failure.

          The fix by implementing the read(ByteBuffer buf) method looks good, but I wonder if we could remove the long deprecated class entirely at all. Will raise this separately.

          diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java
          index 22d4e23..d7c5da9 100644
          --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java
          +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java
          @@ -480,7 +480,21 @@ void sendReadResult(Peer peer, Status statusCode) {
           
             @Override
             public int read(ByteBuffer buf) throws IOException {
          -    throw new UnsupportedOperationException("readDirect unsupported in BlockReaderRemote");
          +    int toRead = buf.remaining();
          +    int nRead;
          +    if (buf.isDirect()) {
          +      byte[] bytes = new byte[toRead];
          +      nRead = read(bytes, 0, toRead);
          +      if (nRead > 0) {
          +        buf.put(bytes, 0, nRead);
          +      }
          +    } else {
          +      nRead = read(buf.array(), buf.arrayOffset(), toRead);
          +      if (nRead > 0) {
          +        buf.position(buf.position() + nRead);
          +      }
          +    }
          +    return nRead;
             }
          
          Show
          drankye Kai Zheng added a comment - Thanks Bo for the update, and fixing of the test failure. The fix by implementing the read(ByteBuffer buf) method looks good, but I wonder if we could remove the long deprecated class entirely at all. Will raise this separately. diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java index 22d4e23..d7c5da9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java @@ -480,7 +480,21 @@ void sendReadResult(Peer peer, Status statusCode) { @Override public int read(ByteBuffer buf) throws IOException { - throw new UnsupportedOperationException( "readDirect unsupported in BlockReaderRemote" ); + int toRead = buf.remaining(); + int nRead; + if (buf.isDirect()) { + byte [] bytes = new byte [toRead]; + nRead = read(bytes, 0, toRead); + if (nRead > 0) { + buf.put(bytes, 0, nRead); + } + } else { + nRead = read(buf.array(), buf.arrayOffset(), toRead); + if (nRead > 0) { + buf.position(buf.position() + nRead); + } + } + return nRead; }
          Hide
          drankye Kai Zheng added a comment -

          Rebased the patch.

          Show
          drankye Kai Zheng added a comment - Rebased the patch.
          Hide
          drankye Kai Zheng added a comment -

          Uma Maheswara Rao G, could you help trigger the Jenkins building? This should be in as requested elsewhere to match with the native coder.

          Show
          drankye Kai Zheng added a comment - Uma Maheswara Rao G , could you help trigger the Jenkins building? This should be in as requested elsewhere to match with the native coder.
          Hide
          drankye Kai Zheng added a comment -

          Since HDFS-10548 was in this will need to be rebased. Youwei Wang, would you help with this? Thanks!

          Show
          drankye Kai Zheng added a comment - Since HDFS-10548 was in this will need to be rebased. Youwei Wang , would you help with this? Thanks!
          Hide
          hayabusa Youwei Wang added a comment -

          Patch HDFS-8901 version 11 uploaded.
          Based on commit id: 7cac7655fd84ac394250705b31e3927fe548e34a

          Show
          hayabusa Youwei Wang added a comment - Patch HDFS-8901 version 11 uploaded. Based on commit id: 7cac7655fd84ac394250705b31e3927fe548e34a
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HDFS-8901 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820124/HDFS-8901.v11.patch
          JIRA Issue HDFS-8901
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16187/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HDFS-8901 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820124/HDFS-8901.v11.patch JIRA Issue HDFS-8901 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16187/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hayabusa Youwei Wang added a comment -

          HDFS-8901 Patch version 12 uploaded.
          Date: 2016-7-27 10:51:58
          Based on commitid: 49969b16cdba0f251b9f8bf3d8df9906e38b5c61

          Show
          hayabusa Youwei Wang added a comment - HDFS-8901 Patch version 12 uploaded. Date: 2016-7-27 10:51:58 Based on commitid: 49969b16cdba0f251b9f8bf3d8df9906e38b5c61
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 40s Maven dependency ordering for branch
          +1 mvninstall 8m 20s trunk passed
          +1 compile 9m 3s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 2m 52s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 5m 21s trunk passed
          +1 javadoc 2m 6s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 2s the patch passed
          +1 compile 7m 21s the patch passed
          +1 javac 7m 21s the patch passed
          -0 checkstyle 1m 32s root: The patch generated 16 new + 305 unchanged - 14 fixed = 321 total (was 319)
          +1 mvnsite 2m 25s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 59s the patch passed
          +1 javadoc 2m 15s the patch passed
          +1 unit 8m 25s hadoop-common in the patch passed.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          -1 unit 77m 43s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          142m 3s



          Reason Tests
          Failed junit tests hadoop.cli.TestHDFSCLI
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.hdfs.TestDecommissionWithStriped
            hadoop.hdfs.TestParallelRead
            hadoop.hdfs.TestDFSStripedInputStream
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.datanode.TestLargeBlockReport
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.TestReconstructStripedFile
            hadoop.hdfs.TestParallelUnixDomainRead



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820353/HDFS-8901.v12.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f1ba08aaeca8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2d8d183
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16205/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16205/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16205/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16205/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 40s Maven dependency ordering for branch +1 mvninstall 8m 20s trunk passed +1 compile 9m 3s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 2m 52s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 5m 21s trunk passed +1 javadoc 2m 6s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 2s the patch passed +1 compile 7m 21s the patch passed +1 javac 7m 21s the patch passed -0 checkstyle 1m 32s root: The patch generated 16 new + 305 unchanged - 14 fixed = 321 total (was 319) +1 mvnsite 2m 25s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 59s the patch passed +1 javadoc 2m 15s the patch passed +1 unit 8m 25s hadoop-common in the patch passed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 77m 43s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 142m 3s Reason Tests Failed junit tests hadoop.cli.TestHDFSCLI   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.TestParallelRead   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.datanode.TestLargeBlockReport   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestParallelUnixDomainRead Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820353/HDFS-8901.v12.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f1ba08aaeca8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2d8d183 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16205/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16205/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16205/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16205/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hayabusa Youwei Wang added a comment - - edited

          New patch submitted.
          File name: HDFS-8901.v13.patch
          Based on commitid: 34ccaa8367f048ed9f56038efe7b3202c436b6e6

          Show
          hayabusa Youwei Wang added a comment - - edited New patch submitted. File name: HDFS-8901 .v13.patch Based on commitid: 34ccaa8367f048ed9f56038efe7b3202c436b6e6
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 3s trunk passed
          +1 compile 0m 30s trunk passed
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 34s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 27s the patch passed
          +1 javac 0m 27s the patch passed
          -0 checkstyle 0m 13s hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 5 new + 75 unchanged - 0 fixed = 80 total (was 75)
          +1 mvnsite 0m 30s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 30s the patch passed
          +1 javadoc 0m 17s the patch passed
          +1 unit 0m 52s hadoop-hdfs-client in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          16m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821255/HDFS-8901.v13.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 62a0b084f8c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 34ccaa8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16273/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16273/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16273/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 3s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed -0 checkstyle 0m 13s hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 5 new + 75 unchanged - 0 fixed = 80 total (was 75) +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 17s the patch passed +1 unit 0m 52s hadoop-hdfs-client in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 16m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821255/HDFS-8901.v13.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 62a0b084f8c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34ccaa8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16273/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16273/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16273/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 1m 24s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 6s trunk passed
          +1 javadoc 1m 16s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 1m 23s the patch passed
          +1 javac 1m 23s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project: The patch generated 4 new + 89 unchanged - 0 fixed = 93 total (was 89)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 1m 10s the patch passed
          +1 unit 0m 53s hadoop-hdfs-client in the patch passed.
          -1 unit 75m 34s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          102m 46s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.namenode.TestEditLog



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821280/HDFS-8901.v14.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ad8d02b861b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 34ccaa8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16274/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16274/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16274/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16274/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 1m 24s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 6s trunk passed +1 javadoc 1m 16s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 1m 23s the patch passed +1 javac 1m 23s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project: The patch generated 4 new + 89 unchanged - 0 fixed = 93 total (was 89) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 10s the patch passed +1 unit 0m 53s hadoop-hdfs-client in the patch passed. -1 unit 75m 34s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 102m 46s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.TestEditLog Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821280/HDFS-8901.v14.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ad8d02b861b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34ccaa8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16274/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16274/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16274/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16274/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Hi Youwei,

          Thanks for your update. Please note we should support both direct ByteBuffer and on-heap ByteBuffer, thus calling aBuffer.array() isn't proper. I would suggest you resume this effort based on the previous patch, instead of reworking this at all.

          Show
          drankye Kai Zheng added a comment - Hi Youwei, Thanks for your update. Please note we should support both direct ByteBuffer and on-heap ByteBuffer, thus calling aBuffer.array() isn't proper. I would suggest you resume this effort based on the previous patch, instead of reworking this at all.
          Hide
          hayabusa Youwei Wang added a comment -

          Patch for HDFS-8901 Version 14 Uploaded.
          Date: 2016-8-2 15:49:56
          Based on commitID:
          6890d5b472320fa7592ed1b08b623c55a27089c6

          Show
          hayabusa Youwei Wang added a comment - Patch for HDFS-8901 Version 14 Uploaded. Date: 2016-8-2 15:49:56 Based on commitID: 6890d5b472320fa7592ed1b08b623c55a27089c6
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 35s Maven dependency ordering for branch
          +1 mvninstall 6m 46s trunk passed
          +1 compile 6m 47s trunk passed
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 2m 20s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 4m 24s trunk passed
          +1 javadoc 2m 0s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 55s the patch passed
          +1 compile 6m 41s the patch passed
          +1 javac 6m 41s the patch passed
          +1 checkstyle 1m 27s root: The patch generated 0 new + 302 unchanged - 9 fixed = 302 total (was 311)
          +1 mvnsite 2m 17s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 47s the patch passed
          +1 javadoc 1m 59s the patch passed
          -1 unit 7m 37s hadoop-common in the patch failed.
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          -1 unit 57m 49s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          114m 0s



          Reason Tests
          Failed junit tests hadoop.net.TestClusterTopology
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821527/HDFS-8901.v14.patch
          JIRA Issue HDFS-8901
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6750986d13a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6890d5b
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16286/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16286/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16286/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 35s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 6m 47s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 2m 20s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 4m 24s trunk passed +1 javadoc 2m 0s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 55s the patch passed +1 compile 6m 41s the patch passed +1 javac 6m 41s the patch passed +1 checkstyle 1m 27s root: The patch generated 0 new + 302 unchanged - 9 fixed = 302 total (was 311) +1 mvnsite 2m 17s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 47s the patch passed +1 javadoc 1m 59s the patch passed -1 unit 7m 37s hadoop-common in the patch failed. +1 unit 0m 56s hadoop-hdfs-client in the patch passed. -1 unit 57m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 114m 0s Reason Tests Failed junit tests hadoop.net.TestClusterTopology   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821527/HDFS-8901.v14.patch JIRA Issue HDFS-8901 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6750986d13a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6890d5b Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16286/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16286/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16286/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          The latest result looks good. Could anyone help do more reviewing? Thanks! Note it's performance related.

          Show
          drankye Kai Zheng added a comment - The latest result looks good. Could anyone help do more reviewing? Thanks! Note it's performance related.
          Hide
          zhz Zhe Zhang added a comment -

          I'm catching up on the history of this JIRA.

          Kai Zheng Back to Walter's question, currently positional read always passes in a byte array. Could you explain a little more which data copy this patch will help avoid?

          Show
          zhz Zhe Zhang added a comment - I'm catching up on the history of this JIRA. Kai Zheng Back to Walter's question, currently positional read always passes in a byte array. Could you explain a little more which data copy this patch will help avoid?
          Hide
          drankye Kai Zheng added a comment -

          Thanks Zhe Zhang for the look! I thought you meant this saying in the description, right?

          It will also fix avoiding unnecessary data copying between striping read chunk buffers and decode input buffers.

          The above saying should be talking about the changes like the following (too long ago said, have to think about this much):

          -        chunk.copyTo(decodeInputs[i]);
          +        if (chunk.useChunkBuffer()) {
          +          chunk.getChunkBuffer().copyTo(decodeInputs[i]);
          +        }
          

          Note the new codes has a condition to justify the copyTo call.

          Show
          drankye Kai Zheng added a comment - Thanks Zhe Zhang for the look! I thought you meant this saying in the description, right? It will also fix avoiding unnecessary data copying between striping read chunk buffers and decode input buffers. The above saying should be talking about the changes like the following (too long ago said, have to think about this much): - chunk.copyTo(decodeInputs[i]); + if (chunk.useChunkBuffer()) { + chunk.getChunkBuffer().copyTo(decodeInputs[i]); + } Note the new codes has a condition to justify the copyTo call.
          Hide
          zhz Zhe Zhang added a comment -

          Sorry about the delay. I just finished reviewing the patch. It needs a rebase, most likely due to HDFS-8905.

          1. The below code can be moved to BlockReaderUtil:
            +        //Behave exactly as the readAll() call
            +        ByteBuffer tmp = buf.duplicate();
            +        tmp.limit(tmp.position() + len);
            +        tmp = tmp.slice();
            ...
            
          2. This is a little confusing and actually IntelliJ can't find where ret is declared. Can we change to regular assignments?
            int nread = 0, ret;
            
          3. Just to clarify, is the plan to add a ByteBuffer version of the below positional read public API?
             public int read(long position, byte[] buffer, int offset, int length)
            
          4. Why was this not needed before but is needed now?
            +          strategy.getReadBuffer().flip();
            
          5. Below formatting looks odd. Maybe an IDE setting issue?
            +  private static void calculateChunkPositionsInBuf(int cellSize,
            +                                                   AlignedStripe[] stripes,
            +                                                   StripingCell[] cells,
            +                                                   ByteBuffer bu
            
          6. So with the changes in DFSInputStream and DFSStripedInputStream, every StripingChunk should use ByteBuffer instead of byte array right? Can we unify the usage of byteBuffer and chunkBuffer? Looks like ChunkByteBuffer can just handle a special case of a single ByteBuffer?
          7. Just to clarify, we will improve performance of the below operation only after implementing the new public API for ByteBuffer-based positional read right? Because otherwise chunk.getChunkBuffer will be an indirect buffer backed by a regular on-heap array.
            -        chunk.copyTo(decodeInputs[i]);
            +        if (chunk.useChunkBuffer()) {
            +          chunk.getChunkBuffer().copyTo(decodeInputs[i]);
            +        }
            
          Show
          zhz Zhe Zhang added a comment - Sorry about the delay. I just finished reviewing the patch. It needs a rebase, most likely due to HDFS-8905 . The below code can be moved to BlockReaderUtil : + //Behave exactly as the readAll() call + ByteBuffer tmp = buf.duplicate(); + tmp.limit(tmp.position() + len); + tmp = tmp.slice(); ... This is a little confusing and actually IntelliJ can't find where ret is declared. Can we change to regular assignments? int nread = 0, ret; Just to clarify, is the plan to add a ByteBuffer version of the below positional read public API? public int read( long position, byte [] buffer, int offset, int length) Why was this not needed before but is needed now? + strategy.getReadBuffer().flip(); Below formatting looks odd. Maybe an IDE setting issue? + private static void calculateChunkPositionsInBuf( int cellSize, + AlignedStripe[] stripes, + StripingCell[] cells, + ByteBuffer bu So with the changes in DFSInputStream and DFSStripedInputStream , every StripingChunk should use ByteBuffer instead of byte array right? Can we unify the usage of byteBuffer and chunkBuffer ? Looks like ChunkByteBuffer can just handle a special case of a single ByteBuffer ? Just to clarify, we will improve performance of the below operation only after implementing the new public API for ByteBuffer -based positional read right? Because otherwise chunk.getChunkBuffer will be an indirect buffer backed by a regular on-heap array. - chunk.copyTo(decodeInputs[i]); + if (chunk.useChunkBuffer()) { + chunk.getChunkBuffer().copyTo(decodeInputs[i]); + }
          Hide
          Sammi SammiChen added a comment -

          Hi Zhe,

          Thanks for your great effort to review the patch. Very good suggestions. Here is my thoughts.

          1. the piece of code will be moved to BlockReaderUtil, create a new readAll function for it.
          2. will be handled.
          3. Yes. we are planning to add a ByteBuffer version read API
          4. good question, I'm double checking the code. And will share my finding later.
          5. will be handled.
          6. Every StripingChunk will have either a ChunkByteBuffer or a ByteBuffer to track different buffer source. ChunkByteBuffer will be used when reusing user input buffer as the striping read buffer during positional read. And ByteBuffer will be used when read using stateful strip reader.
          7. the memory copy is avoid in case chunk.useChunkBuffer is not true, that's an improvement.

          Show
          Sammi SammiChen added a comment - Hi Zhe, Thanks for your great effort to review the patch. Very good suggestions. Here is my thoughts. 1. the piece of code will be moved to BlockReaderUtil, create a new readAll function for it. 2. will be handled. 3. Yes. we are planning to add a ByteBuffer version read API 4. good question, I'm double checking the code. And will share my finding later. 5. will be handled. 6. Every StripingChunk will have either a ChunkByteBuffer or a ByteBuffer to track different buffer source. ChunkByteBuffer will be used when reusing user input buffer as the striping read buffer during positional read. And ByteBuffer will be used when read using stateful strip reader. 7. the memory copy is avoid in case chunk.useChunkBuffer is not true, that's an improvement.
          Hide
          Sammi SammiChen added a comment -

          changes based on v14 patch,
          1. rebase the patch
          2. fix item 1,2,5 from Zhe's suggestions
          3. add new test case testStatefulReadWithDNFailure

          Show
          Sammi SammiChen added a comment - changes based on v14 patch, 1. rebase the patch 2. fix item 1,2,5 from Zhe's suggestions 3. add new test case testStatefulReadWithDNFailure
          Hide
          Sammi SammiChen added a comment -

          For question 4, why flip is not needed before? The reason is previously, positional reader use a byte array wrapped buffer as read buffer. Then later code use byte array to access the fetched data. And for stateful reader, previously it has already reset the buffer position by {{ decodeInputs[i].position(0) }} before decoding.

          And one complement for question 6. ByteBuffer is also used in decode function in positional reader, to hold parity content.

          Show
          Sammi SammiChen added a comment - For question 4, why flip is not needed before? The reason is previously, positional reader use a byte array wrapped buffer as read buffer. Then later code use byte array to access the fetched data. And for stateful reader, previously it has already reset the buffer position by {{ decodeInputs [i] .position(0) }} before decoding. And one complement for question 6. ByteBuffer is also used in decode function in positional reader, to hold parity content.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 24s trunk passed
          +1 compile 8m 43s trunk passed
          +1 checkstyle 1m 35s trunk passed
          +1 mvnsite 2m 40s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 5m 31s trunk passed
          +1 javadoc 2m 20s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 6s the patch passed
          +1 compile 7m 26s the patch passed
          +1 javac 7m 26s the patch passed
          -0 checkstyle 1m 34s root: The patch generated 2 new + 286 unchanged - 10 fixed = 288 total (was 296)
          +1 mvnsite 2m 25s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 59s the patch passed
          +1 javadoc 2m 0s the patch passed
          +1 unit 8m 18s hadoop-common in the patch passed.
          +1 unit 1m 3s hadoop-hdfs-client in the patch passed.
          +1 unit 77m 14s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          139m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-8901
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825653/HDFS-8901.v15.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f60576d29393 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bfb4d95
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16547/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16547/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16547/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 24s trunk passed +1 compile 8m 43s trunk passed +1 checkstyle 1m 35s trunk passed +1 mvnsite 2m 40s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 5m 31s trunk passed +1 javadoc 2m 20s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 6s the patch passed +1 compile 7m 26s the patch passed +1 javac 7m 26s the patch passed -0 checkstyle 1m 34s root: The patch generated 2 new + 286 unchanged - 10 fixed = 288 total (was 296) +1 mvnsite 2m 25s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 59s the patch passed +1 javadoc 2m 0s the patch passed +1 unit 8m 18s hadoop-common in the patch passed. +1 unit 1m 3s hadoop-hdfs-client in the patch passed. +1 unit 77m 14s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 139m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-8901 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825653/HDFS-8901.v15.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f60576d29393 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bfb4d95 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16547/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16547/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16547/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Sammi SammiChen added a comment -

          Fix one style checker reported issue. Another reported issue is a false alert.

          Show
          Sammi SammiChen added a comment - Fix one style checker reported issue. Another reported issue is a false alert.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 2m 2s Maven dependency ordering for branch
          +1 mvninstall 9m 3s trunk passed
          +1 compile 9m 4s trunk passed
          +1 checkstyle 1m 54s trunk passed
          +1 mvnsite 3m 9s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 5m 28s trunk passed
          +1 javadoc 2m 4s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 56s the patch passed
          +1 compile 6m 51s the patch passed
          +1 javac 6m 51s the patch passed
          -0 checkstyle 1m 26s root: The patch generated 1 new + 286 unchanged - 10 fixed = 287 total (was 296)
          +1 mvnsite 2m 16s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 48s the patch passed
          +1 javadoc 2m 1s the patch passed
          +1 unit 8m 10s hadoop-common in the patch passed.
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          -1 unit 71m 35s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          136m 14s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-8901
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825953/HDFS-8901.v16.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0926fa7955bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c258171
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16558/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16558/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16558/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16558/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 2s Maven dependency ordering for branch +1 mvninstall 9m 3s trunk passed +1 compile 9m 4s trunk passed +1 checkstyle 1m 54s trunk passed +1 mvnsite 3m 9s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 5m 28s trunk passed +1 javadoc 2m 4s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 56s the patch passed +1 compile 6m 51s the patch passed +1 javac 6m 51s the patch passed -0 checkstyle 1m 26s root: The patch generated 1 new + 286 unchanged - 10 fixed = 287 total (was 296) +1 mvnsite 2m 16s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 48s the patch passed +1 javadoc 2m 1s the patch passed +1 unit 8m 10s hadoop-common in the patch passed. +1 unit 0m 56s hadoop-hdfs-client in the patch passed. -1 unit 71m 35s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 136m 14s Reason Tests Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-8901 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825953/HDFS-8901.v16.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0926fa7955bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16558/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16558/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16558/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16558/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Sammi SammiChen added a comment -

          I rerun the unit test TestErasureCodeBenchmarkThroughput locally with latest trunk code plus this patch. The unit test succeeds.

          Show
          Sammi SammiChen added a comment - I rerun the unit test TestErasureCodeBenchmarkThroughput locally with latest trunk code plus this patch. The unit test succeeds.
          Hide
          drankye Kai Zheng added a comment -

          Hey Zhe Zhang,

          Do you think SammiChen and the latest patch addressed your questions? I'm wondering if we could get this in soon and then proceed on another one HDFS-8957, to catch up with some release. Thanks!

          Show
          drankye Kai Zheng added a comment - Hey Zhe Zhang , Do you think SammiChen and the latest patch addressed your questions? I'm wondering if we could get this in soon and then proceed on another one HDFS-8957 , to catch up with some release. Thanks!
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Sammi for the update! Sorry it's taking me long to review. I think we are really close.

          The non-EC part:

          1. In my previous comment I suggested moving the readAll code to BlockReaderUtil. Actually thinking more about it, BlockReader already provides int read(ByteBuffer buf). Shouldn't the input stream level read rely on the BlockReader lever ByteBuffer read?
            -        int nread = reader.readAll(buf, offset, len);
            +        int nread = BlockReaderUtil.readAll(reader, buf, len);
            
          2. Nit: targetEnd should probably be initialized immediately after bytesToRead, before the try clause
            +      long targetEnd;
            +      int bytesToRead = (int) Math.min(remaining,
            +          blk.getBlockSize() - targetStart);
                   try {
            +        targetEnd = targetStart + bytesToRead - 1;
            
          3. hedgedFetchBlockByteRange currently still allocates a new buffer (bb = ByteBuffer.allocate(len);) and copies the result into buf. As a follow-on I guess we can think about directly using part of buf?
          4. Is it necessary to slice() the dup buffer in pread?

          The EC-specific part:

          1. For question 4, why flip is not needed before? The reason is previously, positional reader use a byte array wrapped buffer as read buffer. Then later code use byte array to access the fetched data. And for stateful reader, previously it has already reset the buffer position by {{ decodeInputs[i].position(0) }} before decoding.

            Thanks for clarifying this. Then shouldn't we do the flip at decode time?

          2. Every StripingChunk will have either a ChunkByteBuffer or a ByteBuffer to track different buffer source. ChunkByteBuffer will be used when reusing user input buffer as the striping read buffer during positional read. And ByteBuffer will be used when read using stateful strip reader.

            I still think they can be unified. In both cases the chunk will actually be read into a single ByteBuffer. It's just that stateful read will read a chunk into a contiguous segment of the ByteBuffer, but positional read will read the chunk into multiple segments of the ByteBuffer. From the StripingChunk's angle, I think we should just give it a list of ByteBuffer's. When reading a StripingChunk, we fill the ByteBuffer's one by one. In other words, we should treat StripingChunk#byteBuffer in this patch as a "slice" in chunkBuffer. I think we can do this consolidation as a follow-on, and most of your handlings of chunkBuffer are correct.

          3. I think the logic in the below code is not very safe. It works now because a) finalizeDecodeInputs is only used by positional read; b) In positional read, public StripingChunk(ByteBuffer buf) constructor is only used for parity and out-of-range chunks (the data won't be in application buffer anyway). Whether to copy chunk data into decodeInputs should depend on whether decodeInputs is a duplicate or slice ByteBuffer from the application buffer. Can we remove this change in this patch and always copyTo? After the above follow-on we can re-enable the optimization.
            -        chunk.copyTo(decodeInputs[i]);
            +        if (chunk.useChunkBuffer()) {
            +          chunk.getChunkBuffer().copyTo(decodeInputs[i]);
            +        }
            
          4. If we don't want to affect the position or limit of target, should we just duplicate it? It could have a non-zero position before entering this call.
                void copyTo(ByteBuffer target) {
                  for (ByteBuffer slice : slices) {
                    target.put(slice);
                  }
                  target.flip();
                }
            
          Show
          zhz Zhe Zhang added a comment - Thanks Sammi for the update! Sorry it's taking me long to review. I think we are really close. The non-EC part: In my previous comment I suggested moving the readAll code to BlockReaderUtil . Actually thinking more about it, BlockReader already provides int read(ByteBuffer buf) . Shouldn't the input stream level read rely on the BlockReader lever ByteBuffer read? - int nread = reader.readAll(buf, offset, len); + int nread = BlockReaderUtil.readAll(reader, buf, len); Nit: targetEnd should probably be initialized immediately after bytesToRead , before the try clause + long targetEnd; + int bytesToRead = ( int ) Math .min(remaining, + blk.getBlockSize() - targetStart); try { + targetEnd = targetStart + bytesToRead - 1; hedgedFetchBlockByteRange currently still allocates a new buffer ( bb = ByteBuffer.allocate(len); ) and copies the result into buf . As a follow-on I guess we can think about directly using part of buf ? Is it necessary to slice() the dup buffer in pread ? The EC-specific part: For question 4, why flip is not needed before? The reason is previously, positional reader use a byte array wrapped buffer as read buffer. Then later code use byte array to access the fetched data. And for stateful reader, previously it has already reset the buffer position by {{ decodeInputs [i] .position(0) }} before decoding. Thanks for clarifying this. Then shouldn't we do the flip at decode time? Every StripingChunk will have either a ChunkByteBuffer or a ByteBuffer to track different buffer source. ChunkByteBuffer will be used when reusing user input buffer as the striping read buffer during positional read. And ByteBuffer will be used when read using stateful strip reader. I still think they can be unified. In both cases the chunk will actually be read into a single ByteBuffer . It's just that stateful read will read a chunk into a contiguous segment of the ByteBuffer , but positional read will read the chunk into multiple segments of the ByteBuffer . From the StripingChunk 's angle, I think we should just give it a list of ByteBuffer 's. When reading a StripingChunk , we fill the ByteBuffer 's one by one. In other words, we should treat StripingChunk#byteBuffer in this patch as a "slice" in chunkBuffer . I think we can do this consolidation as a follow-on, and most of your handlings of chunkBuffer are correct. I think the logic in the below code is not very safe. It works now because a) finalizeDecodeInputs is only used by positional read; b) In positional read, public StripingChunk(ByteBuffer buf) constructor is only used for parity and out-of-range chunks (the data won't be in application buffer anyway). Whether to copy chunk data into decodeInputs should depend on whether decodeInputs is a duplicate or slice ByteBuffer from the application buffer. Can we remove this change in this patch and always copyTo ? After the above follow-on we can re-enable the optimization. - chunk.copyTo(decodeInputs[i]); + if (chunk.useChunkBuffer()) { + chunk.getChunkBuffer().copyTo(decodeInputs[i]); + } If we don't want to affect the position or limit of target, should we just duplicate it? It could have a non-zero position before entering this call. void copyTo(ByteBuffer target) { for (ByteBuffer slice : slices) { target.put(slice); } target.flip(); }
          Hide
          Sammi SammiChen added a comment -

          Thanks Zhe for take a lot of time review patch and give so many good advices!

          The non-EC part:
          In my previous comment I suggested moving the readAll code to BlockReaderUtil. Actually thinking more about it, BlockReader already provides int read(ByteBuffer buf). Shouldn't the input stream level read rely on the BlockReader lever ByteBuffer read?

          -        int nread = reader.readAll(buf, offset, len);
          +        int nread = BlockReaderUtil.readAll(reader, buf, len);
          

          I thought the {{ BlockReaderUtil.readAll }} implementation just demonstrates the idea, that is leverage BlockReader#read(ByteBuffer buf) to read data into buf. Please enlighten me If I'm wrong.

          Nit: targetEnd should probably be initialized immediately after bytesToRead, before the try clause

          Handled in the new patch

          hedgedFetchBlockByteRange currently still allocates a new buffer (bb = ByteBuffer.allocate(len);) and copies the result into buf. As a follow-on I guess we can think about directly using part of buf?

          Good suggestion, consider do a follow-on later.

          Is it necessary to slice() the dup buffer in pread?

          Good advice, addressed in the new patch.

          The EC-specific part:
          Thanks for clarifying this. Then shouldn't we do the flip at decode time?

          Good advice, addressed in the new patch.

          I still think they can be unified. In both cases the chunk will actually be read into a single ByteBuffer. It's just that stateful read will read a chunk into a contiguous segment of the ByteBuffer, but positional read will read the chunk into multiple segments of the ByteBuffer. From the StripingChunk's angle, I think we should just give it a list of ByteBuffer's. When reading a StripingChunk, we fill the ByteBuffer's one by one. In other words, we should treat StripingChunk#byteBuffer in this patch as a "slice" in chunkBuffer. I think we can do this consolidation as a follow-on, and most of your handlings of chunkBuffer are correct.

          Catch your point now. Will Consider a follow-on later.

          1. I think the logic in the below code is not very safe. It works now because a) finalizeDecodeInputs is only used by positional read; b) In positional read, public StripingChunk(ByteBuffer buf) constructor is only used for parity and out-of-range chunks (the data won't be in application buffer anyway). Whether to copy chunk data into decodeInputs should depend on whether decodeInputs is a duplicate or slice ByteBuffer from the application buffer. Can we remove this change in this patch and always copyTo? After the above follow-on we can re-enable the optimization.
            -        chunk.copyTo(decodeInputs[i]);
            +        if (chunk.useChunkBuffer()) {
            +          chunk.getChunkBuffer().copyTo(decodeInputs[i]);
            +        }
            

          I understand your concern. Given currently finalizeDecodeInputs is only used by positional read decode process, decodeInputs buffer has two sources, one is application buffer, another is internal buffer allocated to hold the out-of-range chunks and parity. When the buffer is formal type, it don't need copy, because element of decodeInputs already point to the buffer. When the buffer is internal allocated buffer, chunk.useChunkBuffer() will be true, and then copy the data. I run the TestDFSStripedInputStream, go through each condition branch, make sure the decode result is correct.

          If we don't want to affect the position or limit of target, should we just duplicate it? It could have a non-zero position before entering this call.

              void copyTo(ByteBuffer target) {
                for (ByteBuffer slice : slices) {
                  target.put(slice);
                }
                target.flip();
              }
          

          copyTo is used to copy a chunk data to the target buffer for next step decoding. After the call, target buffer is ready for read. So we do want to affect the position of target buffer.

          Show
          Sammi SammiChen added a comment - Thanks Zhe for take a lot of time review patch and give so many good advices! The non-EC part: In my previous comment I suggested moving the readAll code to BlockReaderUtil . Actually thinking more about it, BlockReader already provides int read(ByteBuffer buf) . Shouldn't the input stream level read rely on the BlockReader lever ByteBuffer read? - int nread = reader.readAll(buf, offset, len); + int nread = BlockReaderUtil.readAll(reader, buf, len); I thought the {{ BlockReaderUtil.readAll }} implementation just demonstrates the idea, that is leverage BlockReader#read(ByteBuffer buf) to read data into buf. Please enlighten me If I'm wrong. Nit: targetEnd should probably be initialized immediately after bytesToRead , before the try clause Handled in the new patch hedgedFetchBlockByteRange currently still allocates a new buffer ( bb = ByteBuffer.allocate(len); ) and copies the result into buf . As a follow-on I guess we can think about directly using part of buf ? Good suggestion, consider do a follow-on later. Is it necessary to slice() the dup buffer in pread ? Good advice, addressed in the new patch. The EC-specific part: Thanks for clarifying this. Then shouldn't we do the flip at decode time? Good advice, addressed in the new patch. I still think they can be unified. In both cases the chunk will actually be read into a single ByteBuffer . It's just that stateful read will read a chunk into a contiguous segment of the ByteBuffer , but positional read will read the chunk into multiple segments of the ByteBuffer . From the StripingChunk 's angle, I think we should just give it a list of ByteBuffer 's. When reading a StripingChunk , we fill the ByteBuffer 's one by one. In other words, we should treat StripingChunk#byteBuffer in this patch as a "slice" in chunkBuffer . I think we can do this consolidation as a follow-on, and most of your handlings of chunkBuffer are correct. Catch your point now. Will Consider a follow-on later. I think the logic in the below code is not very safe. It works now because a) finalizeDecodeInputs is only used by positional read; b) In positional read, public StripingChunk(ByteBuffer buf) constructor is only used for parity and out-of-range chunks (the data won't be in application buffer anyway). Whether to copy chunk data into decodeInputs should depend on whether decodeInputs is a duplicate or slice ByteBuffer from the application buffer. Can we remove this change in this patch and always copyTo ? After the above follow-on we can re-enable the optimization. - chunk.copyTo(decodeInputs[i]); + if (chunk.useChunkBuffer()) { + chunk.getChunkBuffer().copyTo(decodeInputs[i]); + } I understand your concern. Given currently finalizeDecodeInputs is only used by positional read decode process, decodeInputs buffer has two sources, one is application buffer, another is internal buffer allocated to hold the out-of-range chunks and parity. When the buffer is formal type, it don't need copy, because element of decodeInputs already point to the buffer. When the buffer is internal allocated buffer, chunk.useChunkBuffer() will be true, and then copy the data. I run the TestDFSStripedInputStream , go through each condition branch, make sure the decode result is correct. If we don't want to affect the position or limit of target, should we just duplicate it? It could have a non-zero position before entering this call. void copyTo(ByteBuffer target) { for (ByteBuffer slice : slices) { target.put(slice); } target.flip(); } copyTo is used to copy a chunk data to the target buffer for next step decoding. After the call, target buffer is ready for read. So we do want to affect the position of target buffer.
          Hide
          Sammi SammiChen added a comment -

          attach patch v17, improve the code based on Zhe's good suggestions

          Show
          Sammi SammiChen added a comment - attach patch v17, improve the code based on Zhe's good suggestions
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 7m 4s trunk passed
          +1 compile 7m 37s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 2m 37s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 4m 32s trunk passed
          +1 javadoc 2m 0s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 59s the patch passed
          +1 compile 7m 21s the patch passed
          +1 javac 7m 21s the patch passed
          -0 checkstyle 1m 27s root: The patch generated 1 new + 286 unchanged - 10 fixed = 287 total (was 296)
          +1 mvnsite 2m 27s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 2s the patch passed
          +1 javadoc 2m 2s the patch passed
          -1 unit 7m 25s hadoop-common in the patch failed.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          -1 unit 62m 53s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          120m 21s



          Reason Tests
          Failed junit tests hadoop.security.TestGroupsCaching
            hadoop.hdfs.TestPread



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-8901
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826606/HDFS-8901-v17.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f8933e7ff518 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 08f55cc
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16606/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16606/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16606/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16606/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16606/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 4s trunk passed +1 compile 7m 37s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 2m 37s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 4m 32s trunk passed +1 javadoc 2m 0s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 59s the patch passed +1 compile 7m 21s the patch passed +1 javac 7m 21s the patch passed -0 checkstyle 1m 27s root: The patch generated 1 new + 286 unchanged - 10 fixed = 287 total (was 296) +1 mvnsite 2m 27s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 2s the patch passed +1 javadoc 2m 2s the patch passed -1 unit 7m 25s hadoop-common in the patch failed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 62m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 120m 21s Reason Tests Failed junit tests hadoop.security.TestGroupsCaching   hadoop.hdfs.TestPread Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-8901 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826606/HDFS-8901-v17.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f8933e7ff518 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 08f55cc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16606/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16606/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16606/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16606/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16606/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the update SammiChen.

          1. TestPread failure looks related. It was not failing for v16 patch.
          2. I thought the BlockReaderUtil.readAll implementation just demonstrates the idea, that is leverage BlockReader#read(ByteBuffer buf) to read data into buf.

            Looking back at the discussion history, my previous suggestion to move it to BlockReaderUtil was misleading. Sorry about that.
            So my concern here is not about correctness but about code structure. BlockReaderUtil was designed to contain static util / helper methods for different block readers. That's why its scope was within its package. We shouldn't make it public unnecessarily. If we need a ByteBuffer version of readAll, I think we should add the API to the BlockReader interface.

          3. In general, I feel that dealing with ByteBuffer causes a lot of complexities in understanding the expected position and limit before and after method calls. If possible, it'd be very helpful to specify those expectations in Javadocs. For example, copyTo can specify that it will take care of flipping target to a ready-to-read state.

          Otherwise LGTM. +1 once the above are addressed / clarified. Great work Sammi and Kai!

          Show
          zhz Zhe Zhang added a comment - Thanks for the update SammiChen . TestPread failure looks related. It was not failing for v16 patch. I thought the BlockReaderUtil.readAll implementation just demonstrates the idea, that is leverage BlockReader#read(ByteBuffer buf) to read data into buf. Looking back at the discussion history, my previous suggestion to move it to BlockReaderUtil was misleading. Sorry about that. So my concern here is not about correctness but about code structure. BlockReaderUtil was designed to contain static util / helper methods for different block readers. That's why its scope was within its package. We shouldn't make it public unnecessarily. If we need a ByteBuffer version of readAll , I think we should add the API to the BlockReader interface. In general, I feel that dealing with ByteBuffer causes a lot of complexities in understanding the expected position and limit before and after method calls. If possible, it'd be very helpful to specify those expectations in Javadocs. For example, copyTo can specify that it will take care of flipping target to a ready-to-read state. Otherwise LGTM. +1 once the above are addressed / clarified. Great work Sammi and Kai!
          Hide
          Sammi SammiChen added a comment -

          1. TestPread failure is fixed.
          2.

          Looking back at the discussion history, my previous suggestion to move it to BlockReaderUtil was misleading. Sorry about that.
          So my concern here is not about correctness but about code structure. BlockReaderUtil was designed to contain static util / helper methods for different block readers. That's why its scope was within its package. We shouldn't make it public unnecessarily. If we need a ByteBuffer version of readAll, I think we should add the API to the BlockReader interface.

          Sounds very reasonable. A new API BlockReaderl#readAll(ByteBuffer buf, int len) is introduced to replace the previous one in BlockReaderUtil
          3.

          1. In general, I feel that dealing with ByteBuffer causes a lot of complexities in understanding the expected position and limit before and after method calls. If possible, it'd be very helpful to specify those expectations in Javadocs. For example, copyTo can specify that it will take care of flipping target to a ready-to-read state.

          Note is added to copyTo as a heads-up that target will be of read-to-read state after the call. And I agree with the complexities opinion.

          4. also fixed some old checkstyle issues.

          Thanks Zhe! I think we are very close now.

          Show
          Sammi SammiChen added a comment - 1. TestPread failure is fixed. 2. Looking back at the discussion history, my previous suggestion to move it to BlockReaderUtil was misleading. Sorry about that. So my concern here is not about correctness but about code structure. BlockReaderUtil was designed to contain static util / helper methods for different block readers. That's why its scope was within its package. We shouldn't make it public unnecessarily. If we need a ByteBuffer version of readAll , I think we should add the API to the BlockReader interface. Sounds very reasonable. A new API BlockReaderl#readAll(ByteBuffer buf, int len) is introduced to replace the previous one in BlockReaderUtil 3. In general, I feel that dealing with ByteBuffer causes a lot of complexities in understanding the expected position and limit before and after method calls. If possible, it'd be very helpful to specify those expectations in Javadocs. For example, copyTo can specify that it will take care of flipping target to a ready-to-read state. Note is added to copyTo as a heads-up that target will be of read-to-read state after the call. And I agree with the complexities opinion. 4. also fixed some old checkstyle issues. Thanks Zhe! I think we are very close now.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 23s Maven dependency ordering for branch
          +1 mvninstall 6m 53s trunk passed
          +1 compile 7m 19s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 2m 29s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 4m 32s trunk passed
          +1 javadoc 1m 58s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 52s the patch passed
          +1 compile 6m 44s the patch passed
          +1 javac 6m 44s the patch passed
          -0 checkstyle 1m 26s root: The patch generated 3 new + 317 unchanged - 18 fixed = 320 total (was 335)
          +1 mvnsite 2m 15s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 7s the patch passed
          +1 javadoc 2m 0s the patch passed
          +1 unit 8m 1s hadoop-common in the patch passed.
          +1 unit 1m 3s hadoop-hdfs-client in the patch passed.
          -1 unit 60m 42s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          118m 44s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-8901
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826799/HDFS-8901-v18.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ab7d97a3e796 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0690f09
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16616/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16616/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16616/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16616/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 23s Maven dependency ordering for branch +1 mvninstall 6m 53s trunk passed +1 compile 7m 19s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 2m 29s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 4m 32s trunk passed +1 javadoc 1m 58s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 6m 44s the patch passed +1 javac 6m 44s the patch passed -0 checkstyle 1m 26s root: The patch generated 3 new + 317 unchanged - 18 fixed = 320 total (was 335) +1 mvnsite 2m 15s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 7s the patch passed +1 javadoc 2m 0s the patch passed +1 unit 8m 1s hadoop-common in the patch passed. +1 unit 1m 3s hadoop-hdfs-client in the patch passed. -1 unit 60m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 118m 44s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-8901 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826799/HDFS-8901-v18.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ab7d97a3e796 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0690f09 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16616/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16616/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16616/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16616/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Sammi SammiChen added a comment -

          1. fix TestSnapshotFileLength test failure
          2. fix 2 checkstyle issues. the remaining one is a false alert.

          Show
          Sammi SammiChen added a comment - 1. fix TestSnapshotFileLength test failure 2. fix 2 checkstyle issues. the remaining one is a false alert.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Sammi. I think all my previous comments have been addressed. But I see some issues / questions in the changes from v17 to v18/19:

          1. Why do we need to flip? In the previous code, arraycopy just starts copying from the current position of result.
            -            System.arraycopy(result.array(), result.position(), buf, offset,
            -                len);
            +            result.flip();
            
          2. Could you also explain why TestPread and TestSnapshotFileLength failed for v17 and v18 patches respectively?
          3. Need to be removed:
            +          //buffer.position(buffer.position() + bytesToRead);
            

          Thanks for adding the ByteBuffer version of readAll. Due to the size of the current patch, can we create a separate JIRA just to add this API? It will go through quickly and we can focus on the main logic on this JIRA.

          Thanks for the great effort here.

          Show
          zhz Zhe Zhang added a comment - Thanks Sammi. I think all my previous comments have been addressed. But I see some issues / questions in the changes from v17 to v18/19: Why do we need to flip? In the previous code, arraycopy just starts copying from the current position of result . - System .arraycopy(result.array(), result.position(), buf, offset, - len); + result.flip(); Could you also explain why TestPread and TestSnapshotFileLength failed for v17 and v18 patches respectively? Need to be removed: + //buffer.position(buffer.position() + bytesToRead); Thanks for adding the ByteBuffer version of readAll. Due to the size of the current patch, can we create a separate JIRA just to add this API? It will go through quickly and we can focus on the main logic on this JIRA. Thanks for the great effort here.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 25s Maven dependency ordering for branch
          +1 mvninstall 8m 18s trunk passed
          +1 compile 8m 6s trunk passed
          +1 checkstyle 1m 36s trunk passed
          +1 mvnsite 2m 45s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 5m 4s trunk passed
          +1 javadoc 2m 17s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 17s the patch passed
          +1 compile 8m 21s the patch passed
          +1 javac 8m 21s the patch passed
          -0 checkstyle 1m 36s root: The patch generated 1 new + 317 unchanged - 18 fixed = 318 total (was 335)
          +1 mvnsite 2m 40s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 30s the patch passed
          +1 javadoc 1m 59s the patch passed
          +1 unit 8m 47s hadoop-common in the patch passed.
          +1 unit 1m 4s hadoop-hdfs-client in the patch passed.
          -1 unit 60m 14s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          125m 16s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-8901
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826857/HDFS-8901-v19.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 03a151f01ca7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 05f5c0f
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16619/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16619/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16619/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16619/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 25s Maven dependency ordering for branch +1 mvninstall 8m 18s trunk passed +1 compile 8m 6s trunk passed +1 checkstyle 1m 36s trunk passed +1 mvnsite 2m 45s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 5m 4s trunk passed +1 javadoc 2m 17s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 17s the patch passed +1 compile 8m 21s the patch passed +1 javac 8m 21s the patch passed -0 checkstyle 1m 36s root: The patch generated 1 new + 317 unchanged - 18 fixed = 318 total (was 335) +1 mvnsite 2m 40s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 30s the patch passed +1 javadoc 1m 59s the patch passed +1 unit 8m 47s hadoop-common in the patch passed. +1 unit 1m 4s hadoop-hdfs-client in the patch passed. -1 unit 60m 14s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 125m 16s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-8901 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826857/HDFS-8901-v19.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 03a151f01ca7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 05f5c0f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16619/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16619/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16619/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16619/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Hi SammiChen & Zhe Zhang,

          Thanks for the discussion. Sorry for the late. I have other thought about adding the new readAll API to BlockReader.

          Thanks for adding the ByteBuffer version of readAll. Due to the size of the current patch, can we create a separate JIRA just to add this API? It will go through quickly and we can focus on the main logic on this JIRA.

          Looking at HDFS-10839 that does this work, it looks like to be more natural to add the API as readFully(ByteBuffer buf) in ByteBufferReadable interface. However that would involve a large change and may take some time to happen. Could we leave this work as follow-on to improve and move on this without doing that?

          Show
          drankye Kai Zheng added a comment - Hi SammiChen & Zhe Zhang , Thanks for the discussion. Sorry for the late. I have other thought about adding the new readAll API to BlockReader . Thanks for adding the ByteBuffer version of readAll. Due to the size of the current patch, can we create a separate JIRA just to add this API? It will go through quickly and we can focus on the main logic on this JIRA. Looking at HDFS-10839 that does this work, it looks like to be more natural to add the API as readFully(ByteBuffer buf) in ByteBufferReadable interface. However that would involve a large change and may take some time to happen. Could we leave this work as follow-on to improve and move on this without doing that?
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the comment Kai. I agree we can leave the readAll implementation as a follow-on. I'm OK with the approach in v14 patch which puts the logic in DFSInputStream. After we determine and fix the readAll logic we can do another follow-on to use the appropriate readAll call.

          Show
          zhz Zhe Zhang added a comment - Thanks for the comment Kai. I agree we can leave the readAll implementation as a follow-on. I'm OK with the approach in v14 patch which puts the logic in DFSInputStream . After we determine and fix the readAll logic we can do another follow-on to use the appropriate readAll call.
          Hide
          Sammi SammiChen added a comment -

          Thanks Zhe! Thanks Kai!

          Why do we need to flip? In the previous code, arraycopy just starts copying from the current position of result.

          I changed readAll(ByteBuffer buf, int len) implementation. In Previously implementation, position of buf will not be advanced by len after the all. I think it is not a traditional way when ByteBuffer parameter is used. SO I changed it. Now buf will be advanced by len after the call. That's why result should flip before read.

          Could you also explain why TestPread and TestSnapshotFileLength failed for v17 and v18 patches respectively?

          Previously user input buffer is slice()ed in pread, and then the sliced buffer is passed into the callee. The sliced buffer has its independent position and length. This piece of code is optimized in pread, get rid of the slice(), directly pass the user input buffer to callee. So user buffer's position and length need to be handled carefully now. TestPread and TestSnapshotFileLength failed because user buffer's position and read/write length doesn't took cared correctly in some execution path.

          Need to be removed:

          Oops, will handle it.

          Based on our current discussion, I will reuse the approach in v14 patch to handle the readAll logic.

          Show
          Sammi SammiChen added a comment - Thanks Zhe! Thanks Kai! Why do we need to flip? In the previous code, arraycopy just starts copying from the current position of result. I changed readAll(ByteBuffer buf, int len) implementation. In Previously implementation, position of buf will not be advanced by len after the all. I think it is not a traditional way when ByteBuffer parameter is used. SO I changed it. Now buf will be advanced by len after the call. That's why result should flip before read. Could you also explain why TestPread and TestSnapshotFileLength failed for v17 and v18 patches respectively? Previously user input buffer is slice()ed in pread, and then the sliced buffer is passed into the callee. The sliced buffer has its independent position and length. This piece of code is optimized in pread, get rid of the slice(), directly pass the user input buffer to callee. So user buffer's position and length need to be handled carefully now. TestPread and TestSnapshotFileLength failed because user buffer's position and read/write length doesn't took cared correctly in some execution path. Need to be removed: Oops, will handle it. Based on our current discussion, I will reuse the approach in v14 patch to handle the readAll logic.
          Hide
          Sammi SammiChen added a comment -

          1. remove "//buffer.position(buffer.position() + bytesToRead)"
          2. use inline code logic to replace ByteBuffer version of readAll function, as did in v14.

          Show
          Sammi SammiChen added a comment - 1. remove "//buffer.position(buffer.position() + bytesToRead)" 2. use inline code logic to replace ByteBuffer version of readAll function, as did in v14.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 6m 55s trunk passed
          +1 checkstyle 1m 26s trunk passed
          +1 mvnsite 2m 23s trunk passed
          +1 mvneclipse 0m 47s trunk passed
          +1 findbugs 4m 37s trunk passed
          +1 javadoc 2m 8s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 13s the patch passed
          +1 compile 8m 40s the patch passed
          +1 javac 8m 40s the patch passed
          -0 checkstyle 1m 34s root: The patch generated 1 new + 275 unchanged - 18 fixed = 276 total (was 293)
          +1 mvnsite 2m 38s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 6m 6s the patch passed
          +1 javadoc 2m 16s the patch passed
          -1 unit 7m 53s hadoop-common in the patch failed.
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
          +1 unit 76m 53s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          137m 20s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-8901
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827055/HDFS-8901-v20.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f8fb3650756d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 07650bc
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16634/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16634/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16634/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16634/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 6m 55s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 2m 23s trunk passed +1 mvneclipse 0m 47s trunk passed +1 findbugs 4m 37s trunk passed +1 javadoc 2m 8s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 13s the patch passed +1 compile 8m 40s the patch passed +1 javac 8m 40s the patch passed -0 checkstyle 1m 34s root: The patch generated 1 new + 275 unchanged - 18 fixed = 276 total (was 293) +1 mvnsite 2m 38s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 6s the patch passed +1 javadoc 2m 16s the patch passed -1 unit 7m 53s hadoop-common in the patch failed. +1 unit 1m 2s hadoop-hdfs-client in the patch passed. +1 unit 76m 53s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 137m 20s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-8901 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827055/HDFS-8901-v20.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f8fb3650756d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 07650bc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16634/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16634/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16634/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16634/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Sammi SammiChen added a comment -

          1. Run hadoop.net.TestDNS locally, it passed without any failure. Also the patch is irrelevant to DNS.
          2. The reported checkstyle issue seems a false alert.

          Show
          Sammi SammiChen added a comment - 1. Run hadoop.net.TestDNS locally, it passed without any failure. Also the patch is irrelevant to DNS. 2. The reported checkstyle issue seems a false alert.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Zhe Zhang. Can we have this now if it addressed all your questions? Not sure about it, though much things to follow on to improve.

          Show
          drankye Kai Zheng added a comment - Thanks Zhe Zhang . Can we have this now if it addressed all your questions? Not sure about it, though much things to follow on to improve.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the update SammiChen! v20 patch LGTM. +1 and I'll commit shortly.

          Show
          zhz Zhe Zhang added a comment - Thanks for the update SammiChen ! v20 patch LGTM. +1 and I'll commit shortly.
          Hide
          zhz Zhe Zhang added a comment -

          Committed to trunk. Thanks SammiChen and Kai Zheng for the contribution! Looking forward to seeing improved performance on native buffer.

          Show
          zhz Zhe Zhang added a comment - Committed to trunk. Thanks SammiChen and Kai Zheng for the contribution! Looking forward to seeing improved performance on native buffer.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10410 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10410/)
          HDFS-8901. Use ByteBuffer in striping positional read. Contributed by (zhz: rev 401db4fc65140979fe7665983e36905e886df971)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10410 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10410/ ) HDFS-8901 . Use ByteBuffer in striping positional read. Contributed by (zhz: rev 401db4fc65140979fe7665983e36905e886df971) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          Hide
          Sammi SammiChen added a comment -

          Thanks Zhe for contribute so much effort to review the patch and give excellent advices!

          Show
          Sammi SammiChen added a comment - Thanks Zhe for contribute so much effort to review the patch and give excellent advices!

            People

            • Assignee:
              Sammi SammiChen
              Reporter:
              drankye Kai Zheng
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development