Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10930

Refactor: Wrap Datanode IO related operations

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Datanode IO (Disk/Network) related operations and instrumentations are currently spilled in many classes such as DataNode.java, BlockReceiver.java, BlockSender.java, FsDatasetImpl.java, FsVolumeImpl.java, DirectoryScanner.java, BlockScanner.java, FsDatasetAsyncDiskService.java, LocalReplica.java, LocalReplicaPipeline.java, Storage.java, etc.

      This ticket is opened to consolidate IO related operations for easy instrumentation, metrics collection, logging and trouble shooting.

      1. HDFS-10930.01.patch
        48 kB
        Xiaoyu Yao
      2. HDFS-10930.02.patch
        55 kB
        Xiaoyu Yao
      3. HDFS-10930.03.patch
        53 kB
        Xiaoyu Yao
      4. HDFS-10930.04.patch
        61 kB
        Xiaoyu Yao
      5. HDFS-10930.05.patch
        62 kB
        Xiaoyu Yao
      6. HDFS-10930.06.patch
        63 kB
        Xiaoyu Yao
      7. HDFS-10930.07.patch
        62 kB
        Xiaoyu Yao
      8. HDFS-10930.08.patch
        62 kB
        Xiaoyu Yao
      9. HDFS-10930.09.patch
        62 kB
        Xiaoyu Yao
      10. HDFS-10930.10.patch
        65 kB
        Xiaoyu Yao
      11. HDFS-10930.11.patch
        65 kB
        Xiaoyu Yao
      12. HDFS-10930.barnch-2.00.patch
        57 kB
        Xiaoyu Yao
      13. HDFS-10930-branch-2.00.patch
        57 kB
        Arpit Agarwal
      14. HDFS-10930-branch-2.001.patch
        57 kB
        Xiaoyu Yao
      15. HDFS-10930-branch-2.002.patch
        58 kB
        Xiaoyu Yao

        Issue Links

          Activity

          Hide
          manojg Manoj Govindassamy added a comment - - edited

          Thanks Arpit Agarwal. Interestingly, only the "write data to disk" warning message has gone away. BlockReceiver continues to have other slowness logging like below even after these getting added to metrics.

          Slow flushOrSync took ..
          Slow BlockReceiver write packet to mirror took ..
          Slow manageWriterOsCache took ..
          Slow PacketResponder send ack to upstream took ..

          Show
          manojg Manoj Govindassamy added a comment - - edited Thanks Arpit Agarwal . Interestingly, only the "write data to disk" warning message has gone away. BlockReceiver continues to have other slowness logging like below even after these getting added to metrics. Slow flushOrSync took .. Slow BlockReceiver write packet to mirror took .. Slow manageWriterOsCache took .. Slow PacketResponder send ack to upstream took ..
          Hide
          manojg Manoj Govindassamy added a comment - - edited

          I did check ReplicaOutputStreams and WrappedFileOutputStream before and I saw the latencies getting added to DataNodeVolumeMetrics. But, I couldn't find the check against datanodeSlowLogThresholdMs(dfs.datanode.slow.io.warning.threshold.ms) and the warning logging. May be I am missing something here. Will dig more.

          Show
          manojg Manoj Govindassamy added a comment - - edited I did check ReplicaOutputStreams and WrappedFileOutputStream before and I saw the latencies getting added to DataNodeVolumeMetrics . But, I couldn't find the check against datanodeSlowLogThresholdMs( dfs.datanode.slow.io.warning.threshold.ms ) and the warning logging. May be I am missing something here. Will dig more.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Manoj Govindassamy, the log message was removed by HDFS-10958. HDFS-10959 exposes per volume perf counters via JMX.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Manoj Govindassamy , the log message was removed by HDFS-10958 . HDFS-10959 exposes per volume perf counters via JMX.
          Hide
          xyao Xiaoyu Yao added a comment -

          Manoj Govindassamy, the Slow logging was moved to ReplicaOutputStreams.java, please check the writeToDisk()/sync()/flush() methods there.

          Show
          xyao Xiaoyu Yao added a comment - Manoj Govindassamy , the Slow logging was moved to ReplicaOutputStreams.java, please check the writeToDisk()/sync()/flush() methods there.
          Hide
          manojg Manoj Govindassamy added a comment -

          Xiaoyu Yao, Arpit Agarwal,

          When writing data to disk and whenever the time taken is over the threshold limit (default 300ms) we used to get this WARN message. Fix for this jira removed this logging. Is this message removal intentional ?

                    if (duration > datanodeSlowLogThresholdMs) {
                      LOG.warn("Slow BlockReceiver write data to disk cost:" + duration
                          + "ms (threshold=" + datanodeSlowLogThresholdMs + "ms)");
                    }
          
          Show
          manojg Manoj Govindassamy added a comment - Xiaoyu Yao , Arpit Agarwal , When writing data to disk and whenever the time taken is over the threshold limit (default 300ms) we used to get this WARN message. Fix for this jira removed this logging. Is this message removal intentional ? if (duration > datanodeSlowLogThresholdMs) { LOG.warn("Slow BlockReceiver write data to disk cost:" + duration + "ms (threshold=" + datanodeSlowLogThresholdMs + "ms)"); }
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Committed to branch-2, along with HDFS-11253.

          Show
          arpitagarwal Arpit Agarwal added a comment - Committed to branch-2, along with HDFS-11253 .
          Hide
          xyao Xiaoyu Yao added a comment -

          Looks good to me. Thanks Arpit Agarwal for taking care of the branch-2 patch/commit.

          Show
          xyao Xiaoyu Yao added a comment - Looks good to me. Thanks Arpit Agarwal for taking care of the branch-2 patch/commit.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          The local test-patch run for branch-2 (using HDFS-10930-branch-2.002.patch) came back clean.

          Vote Subsystem Runtime Comment
          +1 @author 0m 0s The patch does not contain any @author
                tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or
                modified test files.
          +1 mvninstall 3m 25s branch-2 passed
          +1 compile 0m 27s branch-2 passed
          +1 checkstyle 0m 24s branch-2 passed
          +1 mvnsite 0m 31s branch-2 passed
          +1 mvneclipse 0m 11s branch-2 passed
          +1 findbugs 0m 58s branch-2 passed
          +1 javadoc 0m 38s branch-2 passed
          +1 mvninstall 0m 26s the patch passed
          +1 compile 0m 23s the patch passed
          +1 javac 0m 23s the patch passed
          -1 checkstyle 0m 20s hadoop-hdfs-project/hadoop-hdfs: The
                patch generated 3 new + 530 unchanged -
                13 fixed = 533 total (was 543)
          +1 mvnsite 0m 28s the patch passed
          +1 mvneclipse 0m 8s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 2s the patch passed
          +1 javadoc 0m 35s the patch passed
          +1 unit 201m 40s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 12s The patch does not generate ASF License
                warnings.
              212m 42s

          The checkstyle issues are existing method length warnings.

          Will push this to branch-2 along with HDFS-11253 to address the earlier comment.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited The local test-patch run for branch-2 (using HDFS-10930 -branch-2.002.patch) came back clean. Vote Subsystem Runtime Comment +1 @author 0m 0s The patch does not contain any @author       tags. +1 test4tests 0m 0s The patch appears to include 6 new or       modified test files. +1 mvninstall 3m 25s branch-2 passed +1 compile 0m 27s branch-2 passed +1 checkstyle 0m 24s branch-2 passed +1 mvnsite 0m 31s branch-2 passed +1 mvneclipse 0m 11s branch-2 passed +1 findbugs 0m 58s branch-2 passed +1 javadoc 0m 38s branch-2 passed +1 mvninstall 0m 26s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed -1 checkstyle 0m 20s hadoop-hdfs-project/hadoop-hdfs: The       patch generated 3 new + 530 unchanged -       13 fixed = 533 total (was 543) +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 8s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 35s the patch passed +1 unit 201m 40s hadoop-hdfs in the patch passed. +1 asflicense 0m 12s The patch does not generate ASF License       warnings.     212m 42s The checkstyle issues are existing method length warnings. Will push this to branch-2 along with HDFS-11253 to address the earlier comment.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review. That 's a good catch. We should also close the checksumIn as well to handle the exception case.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review. That 's a good catch. We should also close the checksumIn as well to handle the exception case.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the branch-2 patch Xiaoyu Yao. Just one comment on the latest branch-2 patch.

          1. The following exception block in the BlockSender constructor should call blockIn.close().
                  ris = new ReplicaInputStreams(blockIn, checksumIn, volumeRef);
                } catch (IOException ioe) {
                  IOUtils.closeStream(this);
                  throw ioe;
                }
            

          +1 otherwise. Kicked off a test-patch run locally.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the branch-2 patch Xiaoyu Yao . Just one comment on the latest branch-2 patch. The following exception block in the BlockSender constructor should call blockIn.close(). ris = new ReplicaInputStreams(blockIn, checksumIn, volumeRef); } catch (IOException ioe) { IOUtils.closeStream( this ); throw ioe; } +1 otherwise. Kicked off a test-patch run locally.
          Hide
          andrew.wang Andrew Wang added a comment -

          Once there's a PR, precommit bot can't process patch files attached to JIRA. So, you may need to use a dummy JIRA and post the patch there to get a run.

          Show
          andrew.wang Andrew Wang added a comment - Once there's a PR, precommit bot can't process patch files attached to JIRA. So, you may need to use a dummy JIRA and post the patch there to get a run.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user xiaoyuyao closed the pull request at:

          https://github.com/apache/hadoop/pull/170

          Show
          githubbot ASF GitHub Bot added a comment - Github user xiaoyuyao closed the pull request at: https://github.com/apache/hadoop/pull/170
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user xiaoyuyao commented on the issue:

          https://github.com/apache/hadoop/pull/170

          close as change commit to trunk.

          Show
          githubbot ASF GitHub Bot added a comment - Github user xiaoyuyao commented on the issue: https://github.com/apache/hadoop/pull/170 close as change commit to trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10951 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10951/)
          HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed (xyao: rev df983b524ab68ea0c70cee9033bfff2d28052cbf)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaOutputStreams.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipeline.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplicaInPipeline.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10951 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10951/ ) HDFS-10930 . Refactor: Wrap Datanode IO related operations. Contributed (xyao: rev df983b524ab68ea0c70cee9033bfff2d28052cbf) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaOutputStreams.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipeline.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplicaInPipeline.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HDFS-10930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/170
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17780/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HDFS-10930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/170 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17780/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment - - edited

          Thanks Arpit Agarwal for the review. I just commit the patch to trunk and attach a new branch-2 patch.

          Show
          xyao Xiaoyu Yao added a comment - - edited Thanks Arpit Agarwal for the review. I just commit the patch to trunk and attach a new branch-2 patch.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal.

          The Github PR 170 matches with patch v10, which fixed the find bugs issue with try-with-resource.
          Patch v11 just fixed two javadoc issue where the @return comments are not complete in LocalReplica.java.

          I will commit patch v11 shortly based on Arpit Agarwal's +1, the Jenkins results of v10 patch and manually rerun of all the failed unit tests (all passed locally). The 2nd checkstyle issue does not exist in latest patch. The 1st and 3nd one is not caused by this patch.

          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java:513:  private int receivePacket() throws IOException {:3: Method length is 308 lines (max allowed is 150).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:23:import java.io.FileDescriptor;:8: Unused import - java.io.FileDescriptor.
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:193:  BlockSender(ExtendedBlock block, long startOffset, long length,:3: Method length is 214 lines (max allowed is 150).
          
          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal . The Github PR 170 matches with patch v10, which fixed the find bugs issue with try-with-resource. Patch v11 just fixed two javadoc issue where the @return comments are not complete in LocalReplica.java. I will commit patch v11 shortly based on Arpit Agarwal 's +1, the Jenkins results of v10 patch and manually rerun of all the failed unit tests (all passed locally). The 2nd checkstyle issue does not exist in latest patch. The 1st and 3nd one is not caused by this patch. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java:513: private int receivePacket() throws IOException {:3: Method length is 308 lines (max allowed is 150). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:23: import java.io.FileDescriptor;:8: Unused import - java.io.FileDescriptor. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:193: BlockSender(ExtendedBlock block, long startOffset, long length,:3: Method length is 214 lines (max allowed is 150).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 37s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 50s the patch passed
          -1 javadoc 0m 37s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 105m 33s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          126m 1s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/170
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 487db3fcbf92 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4dd4f3a
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17777/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17777/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17777/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17777/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17777/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 37s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 50s the patch passed -1 javadoc 0m 37s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 105m 33s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 126m 1s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDirectoryScanner Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/170 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 487db3fcbf92 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4dd4f3a Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17777/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17777/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17777/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17777/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17777/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Xiaoyu Yao.

          +1 for the v11 patch. I assumed the only delta wrt v10 is the workaround for the findbugs false positive in validateIntegrityAndSetLength.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Xiaoyu Yao . +1 for the v11 patch. I assumed the only delta wrt v10 is the workaround for the findbugs false positive in validateIntegrityAndSetLength.
          Hide
          xyao Xiaoyu Yao added a comment -

          Seems like Jenkins finally picked up the new Github PR

          GITHUB PR #170 is being downloaded at Tue Dec  6 16:18:15 UTC 2016 from
          https://github.com/apache/hadoop/pull/170
          Patch from GITHUB PR #170 is being downloaded at Tue Dec  6 16:18:16 UTC 2016 from
          https://github.com/apache/hadoop/pull/170.patch
          HDFS-10930 appears to be a Github PR. Switching Modes.
          
          Show
          xyao Xiaoyu Yao added a comment - Seems like Jenkins finally picked up the new Github PR GITHUB PR #170 is being downloaded at Tue Dec 6 16:18:15 UTC 2016 from https: //github.com/apache/hadoop/pull/170 Patch from GITHUB PR #170 is being downloaded at Tue Dec 6 16:18:16 UTC 2016 from https: //github.com/apache/hadoop/pull/170.patch HDFS-10930 appears to be a Github PR. Switching Modes.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a new patch to see if this can trigger Jenkins run on the correct patch. I've tested locally with checkstyle/findbugs/unit tests. Plan to commit patch v11 if Jenkins still can't run with the correct patch.

          Show
          xyao Xiaoyu Yao added a comment - Attach a new patch to see if this can trigger Jenkins run on the correct patch. I've tested locally with checkstyle/findbugs/unit tests. Plan to commit patch v11 if Jenkins still can't run with the correct patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 9m 3s trunk passed
          +1 compile 0m 54s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 38s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 102m 31s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          124m 40s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 720] is not discharged
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestEncryptionZones
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9a1b538bf396 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8e63fa9
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17769/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17769/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17769/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17769/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17769/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 9m 3s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 38s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 102m 31s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 124m 40s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java: [line 720] is not discharged Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestEncryptionZones Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9a1b538bf396 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8e63fa9 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17769/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17769/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17769/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17769/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17769/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user xiaoyuyao opened a pull request:

          https://github.com/apache/hadoop/pull/170

          HDFS-10930

          HDFS-10930.10

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/xiaoyuyao/hadoop HDFS-10930

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/170.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #170


          commit 500f665ab940f174314005eb2eb68388c96babbc
          Author: Xiaoyu Yao <xyao@apache.org>
          Date: 2016-11-12T15:43:37Z

          HDFS-10930.05.patch

          commit 08d5f43e54a5b659953ed78921e38eb3c7119bfe
          Author: Xiaoyu Yao <xyao@apache.org>
          Date: 2016-11-15T02:41:48Z

          HDFS-10930.06.patch

          commit eacfb8451a5ca64c185ca073e3ab41573c32cd33
          Author: Xiaoyu Yao <xyao@apache.org>
          Date: 2016-11-29T03:09:36Z

          HDFS-10930.07.patch

          commit 0df4aff0d15a270a912f2eb10952220c41bb6fde
          Author: Xiaoyu Yao <xyao@apache.org>
          Date: 2016-12-06T05:03:28Z

          HDFS-10930.10.patch


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user xiaoyuyao opened a pull request: https://github.com/apache/hadoop/pull/170 HDFS-10930 HDFS-10930 .10 You can merge this pull request into a Git repository by running: $ git pull https://github.com/xiaoyuyao/hadoop HDFS-10930 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #170 commit 500f665ab940f174314005eb2eb68388c96babbc Author: Xiaoyu Yao <xyao@apache.org> Date: 2016-11-12T15:43:37Z HDFS-10930 .05.patch commit 08d5f43e54a5b659953ed78921e38eb3c7119bfe Author: Xiaoyu Yao <xyao@apache.org> Date: 2016-11-15T02:41:48Z HDFS-10930 .06.patch commit eacfb8451a5ca64c185ca073e3ab41573c32cd33 Author: Xiaoyu Yao <xyao@apache.org> Date: 2016-11-29T03:09:36Z HDFS-10930 .07.patch commit 0df4aff0d15a270a912f2eb10952220c41bb6fde Author: Xiaoyu Yao <xyao@apache.org> Date: 2016-12-06T05:03:28Z HDFS-10930 .10.patch
          Hide
          xyao Xiaoyu Yao added a comment -

          I've tested it locally and the try with resource finally solve the findbugs issue here. Hope we will have a clean Jenkins run.

          Show
          xyao Xiaoyu Yao added a comment - I've tested it locally and the try with resource finally solve the findbugs issue here. Hope we will have a clean Jenkins run.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 39s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 468 unchanged - 12 fixed = 471 total (was 480)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 38s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 62m 4s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          82m 14s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 720] is not discharged
          Failed junit tests hadoop.hdfs.TestDFSClientRetries



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a123de5ab40e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dcedb72
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17766/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17766/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17766/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17766/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17766/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17766/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 39s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 468 unchanged - 12 fixed = 471 total (was 480) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 38s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 62m 4s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 82m 14s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java: [line 720] is not discharged Failed junit tests hadoop.hdfs.TestDFSClientRetries Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a123de5ab40e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dcedb72 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17766/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17766/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17766/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17766/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17766/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17766/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Change to use IOUtils.cleanup

                IOUtils.cleanup(null, checksumIn, blockIn, ris);
          
          Show
          xyao Xiaoyu Yao added a comment - Change to use IOUtils.cleanup IOUtils.cleanup( null , checksumIn, blockIn, ris);
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 8m 31s trunk passed
          +1 compile 0m 54s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 12s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 10s trunk passed
          +1 javadoc 0m 45s trunk passed
          +1 mvninstall 0m 58s the patch passed
          +1 compile 1m 0s the patch passed
          +1 javac 1m 0s the patch passed
          -0 checkstyle 0m 40s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479)
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 38s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          +1 unit 63m 52s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          86m 35s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 720] is not discharged



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 11a75c09f0fc 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dcedb72
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17762/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17762/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17762/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17762/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17762/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 8m 31s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 58s the patch passed +1 compile 1m 0s the patch passed +1 javac 1m 0s the patch passed -0 checkstyle 0m 40s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 38s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) +1 unit 63m 52s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 86m 35s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java: [line 720] is not discharged Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11a75c09f0fc 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dcedb72 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17762/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17762/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17762/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17762/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17762/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment - - edited

          Revert and reattach a new patch to correct the typo in commit message reported by Brahma Reddy Battula.

          The only delta from v07->v08 is the findbugs fix discussed in HDFS-11205. Plan to close HDFS-11205 and fix it here once we get a clean Jenkins run.

          Show
          xyao Xiaoyu Yao added a comment - - edited Revert and reattach a new patch to correct the typo in commit message reported by Brahma Reddy Battula . The only delta from v07->v08 is the findbugs fix discussed in HDFS-11205 . Plan to close HDFS-11205 and fix it here once we get a clean Jenkins run.
          Hide
          brahmareddy Brahma Reddy Battula added a comment - - edited

          Regarding Trunk commit.
          1) Seems mistakenly given jiraID,should be HDFS-10930..?

          HADOOP-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao.

          2) can you look following findbug?

              FindBugs :
          
                 module:hadoop-hdfs-project/hadoop-hdfs 
                 org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 720] is not discharged 
          
          Show
          brahmareddy Brahma Reddy Battula added a comment - - edited Regarding Trunk commit. 1) Seems mistakenly given jiraID,should be HDFS-10930 ..? HADOOP-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao. 2) can you look following findbug? FindBugs : module:hadoop-hdfs-project/hadoop-hdfs org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 720] is not discharged
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Sigh.. I am not sure how to get Jenkins to stop looking at the pull request. We may have to run unit tests locally.

          Show
          arpitagarwal Arpit Agarwal added a comment - Sigh.. I am not sure how to get Jenkins to stop looking at the pull request. We may have to run unit tests locally.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 8s HDFS-10930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17721/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 8s HDFS-10930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17721/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Rename to trigger Jenkins for branch-2.

          Show
          xyao Xiaoyu Yao added a comment - Rename to trigger Jenkins for branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HDFS-10930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17720/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HDFS-10930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17720/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the branch-2 patch, pending Jenkins. Thanks Xiaoyu Yao.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the branch-2 patch, pending Jenkins. Thanks Xiaoyu Yao .
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Rename Xiaoyu's patch for Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - Rename Xiaoyu's patch for Jenkins.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Reopening for branch-2 Jenkins run.

          Show
          arpitagarwal Arpit Agarwal added a comment - Reopening for branch-2 Jenkins run.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach an initial branch-2 patch for review.

          Show
          xyao Xiaoyu Yao added a comment - Attach an initial branch-2 patch for review.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review and commit. I will post a branch-2 patch tomorrow.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review and commit. I will post a branch-2 patch tomorrow.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I committed this to trunk. Thanks for this improvement Xiaoyu Yao!

          branch-2 threw up a number of conflicts so we may need a separate patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - I committed this to trunk. Thanks for this improvement Xiaoyu Yao ! branch-2 threw up a number of conflicts so we may need a separate patch.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Jenkins seems to be hopelessly confused as it's picking up the now deleted pull request.

          Given the delta from v6 to v7 patch is just the findbugs+checkstyle fix, I plan to commit this fix later tonight based on the earlier Jenkins run. The unit test failure is unrelated and does not reproduce locally.

          Show
          arpitagarwal Arpit Agarwal added a comment - Jenkins seems to be hopelessly confused as it's picking up the now deleted pull request. Given the delta from v6 to v7 patch is just the findbugs+checkstyle fix, I plan to commit this fix later tonight based on the earlier Jenkins run. The unit test failure is unrelated and does not reproduce locally.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 8s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 7s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 36s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 65m 33s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          101m 38s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 720] is not discharged
          Failed junit tests hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 549b8f0dd329 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3b9d3ac
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17701/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17701/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17701/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17701/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17701/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17701/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 8s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 7s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 36s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 65m 33s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 101m 38s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java: [line 720] is not discharged Failed junit tests hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 549b8f0dd329 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3b9d3ac Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17701/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17701/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17701/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17701/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17701/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17701/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Remove the PR link and resubmit for Jenkins.

          Show
          xyao Xiaoyu Yao added a comment - Remove the PR link and resubmit for Jenkins.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user xiaoyuyao closed the pull request at:

          https://github.com/apache/hadoop/pull/160

          Show
          githubbot ASF GitHub Bot added a comment - Github user xiaoyuyao closed the pull request at: https://github.com/apache/hadoop/pull/160
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the v07 patch pending Jenkins. Thanks Xiaoyu Yao!

          Looks like this Jenkins run picked up the pull request. We'll need to either update the PR or just unlink it so Jenkins uses the attached patch file.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the v07 patch pending Jenkins. Thanks Xiaoyu Yao ! Looks like this Jenkins run picked up the pull request. We'll need to either update the PR or just unlink it so Jenkins uses the attached patch file.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 1s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 15s trunk passed
          +1 compile 0m 51s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 42s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 53s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 37s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 107m 46s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          127m 48s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 718] is not discharged
          Failed junit tests hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7f9620b26d5a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a2b1ff0
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17681/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17681/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 1s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 15s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 53s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 37s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 107m 46s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 127m 48s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java: [line 718] is not discharged Failed junit tests hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7f9620b26d5a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a2b1ff0 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17681/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17681/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review. I've updated the patch based on the feedback for the findbugs issue.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review. I've updated the patch based on the feedback for the findbugs issue.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The findbugs issue looks valid. It can be addressed by restoring the close calls in the finally block if the ris is null.

               } finally {
                 if (ris != null) {
                   ris.close();
          +      } else {
          +        IOUtils.closeStream(checksumIn);
          +        IOUtils.closeStream(blockIn);
                 }
          

          The function will need some more changes to restore scoping for checksumIn and blockIn.

          Show
          arpitagarwal Arpit Agarwal added a comment - The findbugs issue looks valid. It can be addressed by restoring the close calls in the finally block if the ris is null. } finally { if (ris != null ) { ris.close(); + } else { + IOUtils.closeStream(checksumIn); + IOUtils.closeStream(blockIn); } The function will need some more changes to restore scoping for checksumIn and blockIn.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 13s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 1m 9s the patch passed
          +1 compile 1m 53s the patch passed
          +1 javac 1m 53s the patch passed
          -0 checkstyle 1m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 481 unchanged - 12 fixed = 484 total (was 493)
          +1 mvnsite 2m 3s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 4m 8s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 1m 30s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 59m 56s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          88m 37s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 718] is not discharged
          Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 483a4d7f4076 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5af572b
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17574/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17574/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17574/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17574/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17574/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17574/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 13s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 1m 9s the patch passed +1 compile 1m 53s the patch passed +1 javac 1m 53s the patch passed -0 checkstyle 1m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 481 unchanged - 12 fixed = 484 total (was 493) +1 mvnsite 2m 3s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 4m 8s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 1m 30s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 59m 56s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 88m 37s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java: [line 718] is not discharged Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 483a4d7f4076 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5af572b Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17574/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17574/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17574/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17574/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17574/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17574/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          +1 pending Jenkins (just started a manual build).

          I'll also hold off committing for a couple of days.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited +1 pending Jenkins (just started a manual build). I'll also hold off committing for a couple of days.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user xiaoyuyao commented on the issue:

          https://github.com/apache/hadoop/pull/160

          Update PR to fix the findbugs and unit test failure.

          Show
          githubbot ASF GitHub Bot added a comment - Github user xiaoyuyao commented on the issue: https://github.com/apache/hadoop/pull/160 Update PR to fix the findbugs and unit test failure.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 34s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 481 unchanged - 12 fixed = 484 total (was 493)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
          -1 javadoc 0m 42s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 82m 14s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          102m 49s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            ris is null guaranteed to be dereferenced in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) on exception path Dereferenced at BlockPoolSlice.java:be dereferenced in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) on exception path Dereferenced at BlockPoolSlice.java:[line 753]
            Possible null pointer dereference of ris in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) Dereferenced at BlockPoolSlice.java:ris in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) Dereferenced at BlockPoolSlice.java:[line 753]
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 718] is not discharged
          Failed junit tests hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          GITHUB PR https://github.com/apache/hadoop/pull/160
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 80ca84118f70 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 99c2bbd
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17563/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17563/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17563/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17563/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17563/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17563/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 34s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 481 unchanged - 12 fixed = 484 total (was 493) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) -1 javadoc 0m 42s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 82m 14s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 102m 49s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   ris is null guaranteed to be dereferenced in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) on exception path Dereferenced at BlockPoolSlice.java:be dereferenced in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) on exception path Dereferenced at BlockPoolSlice.java: [line 753]   Possible null pointer dereference of ris in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) Dereferenced at BlockPoolSlice.java:ris in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) Dereferenced at BlockPoolSlice.java: [line 753]   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java: [line 718] is not discharged Failed junit tests hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 GITHUB PR https://github.com/apache/hadoop/pull/160 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 80ca84118f70 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 99c2bbd Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17563/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17563/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17563/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17563/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17563/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17563/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review. Attach v06 patch that addresses the comments. Summary of delta from v05:

          1. Fix the unit test failure TestWriteToReplica#testReplicaMapAfterDatanodeRestart which shares the same cause of the Findbugs issues with the change below

          BlockPoolSlice.java Line 753
          ++ if (ris != null) {
  
                 ris.close();
          
++ }
          

          2. Will open follow up JIRAs on the remaining issues as Arpit Agarwal suggested.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review. Attach v06 patch that addresses the comments. Summary of delta from v05: 1. Fix the unit test failure TestWriteToReplica#testReplicaMapAfterDatanodeRestart which shares the same cause of the Findbugs issues with the change below BlockPoolSlice.java Line 753 ++ if (ris != null ) {
 ris.close(); 
++ } 2. Will open follow up JIRAs on the remaining issues as Arpit Agarwal suggested.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Mirroring comment from Github:

          Thanks Xiaoyu Yao. The patch looks good overall. Findbugs flagged a few warnings and the TestWriteToReplica failure looks related.

          There's a few remaining static methods like Storage#rename, and DatanodeUtil#createTmpFile that aren't as easy to factor out. We can look at them in a subsequent Jira. In a subsequent change we'll want to move the refactored methods into interfaces for testability.

          Show
          arpitagarwal Arpit Agarwal added a comment - Mirroring comment from Github: Thanks Xiaoyu Yao . The patch looks good overall. Findbugs flagged a few warnings and the TestWriteToReplica failure looks related. There's a few remaining static methods like Storage#rename, and DatanodeUtil#createTmpFile that aren't as easy to factor out. We can look at them in a subsequent Jira. In a subsequent change we'll want to move the refactored methods into interfaces for testability.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user xiaoyuyao opened a pull request:

          https://github.com/apache/hadoop/pull/160

          HDFS-10930.05.patch

          Create a PR for easy review.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/xiaoyuyao/hadoop HDFS-10930

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/160.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #160


          commit 43c3f2eb9ffecd2ed4831148900d3213f3a8671a
          Author: Xiaoyu Yao <xyao@apache.org>
          Date: 2016-11-12T15:43:37Z

          HDFS-10930.05.patch


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user xiaoyuyao opened a pull request: https://github.com/apache/hadoop/pull/160 HDFS-10930 .05.patch Create a PR for easy review. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xiaoyuyao/hadoop HDFS-10930 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/160.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #160 commit 43c3f2eb9ffecd2ed4831148900d3213f3a8671a Author: Xiaoyu Yao <xyao@apache.org> Date: 2016-11-12T15:43:37Z HDFS-10930 .05.patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 5m 8s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 8m 18s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 56s trunk passed
          +1 javadoc 0m 42s trunk passed
          +1 mvninstall 0m 57s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 481 unchanged - 12 fixed = 484 total (was 493)
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 2m 9s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
          -1 javadoc 0m 41s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 79m 13s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          105m 55s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            ris is null guaranteed to be dereferenced in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) on exception path Dereferenced at BlockPoolSlice.java:be dereferenced in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) on exception path Dereferenced at BlockPoolSlice.java:[line 753]
            Possible null pointer dereference of ris in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) Dereferenced at BlockPoolSlice.java:ris in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) Dereferenced at BlockPoolSlice.java:[line 753]
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:[line 718] is not discharged
          Failed junit tests hadoop.hdfs.tools.TestDelegationTokenFetcher
            hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
            hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838627/HDFS-10930.05.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bb79cba63472 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3d26717
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17535/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17535/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17535/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17535/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17535/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17535/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 5m 8s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 8m 18s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 481 unchanged - 12 fixed = 484 total (was 493) +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 2m 9s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) -1 javadoc 0m 41s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 79m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 105m 55s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   ris is null guaranteed to be dereferenced in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) on exception path Dereferenced at BlockPoolSlice.java:be dereferenced in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) on exception path Dereferenced at BlockPoolSlice.java: [line 753]   Possible null pointer dereference of ris in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) Dereferenced at BlockPoolSlice.java:ris in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) Dereferenced at BlockPoolSlice.java: [line 753]   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java:clean up java.io.InputStream on checked exception Obligation to clean up resource created at BlockPoolSlice.java: [line 718] is not discharged Failed junit tests hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838627/HDFS-10930.05.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bb79cba63472 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3d26717 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17535/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17535/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17535/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17535/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17535/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17535/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Update the patch to fix the check style, find bugs and unit test issues.

          Show
          xyao Xiaoyu Yao added a comment - Update the patch to fix the check style, find bugs and unit test issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 15s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 482 unchanged - 12 fixed = 500 total (was 494)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          -1 whitespace 0m 0s The patch 1 line(s) with tabs.
          -1 findbugs 1m 58s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 39s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
          -1 unit 58m 40s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          78m 28s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Non-virtual method call in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) passes null for non-null parameter of new org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaInputStreams(InputStream, InputStream, FsVolumeReference) At BlockPoolSlice.java:long) passes null for non-null parameter of new org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaInputStreams(InputStream, InputStream, FsVolumeReference) At BlockPoolSlice.java:[line 720]
          Failed junit tests hadoop.hdfs.TestDFSShell
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.web.TestWebHdfsFileSystemContract
            hadoop.hdfs.TestDataTransferProtocol
            hadoop.hdfs.TestReadWhileWriting
            hadoop.fs.TestSWebHdfsFileContextMainOperations
            hadoop.fs.TestHDFSFileContextMainOperations
            hadoop.hdfs.TestFileAppend2
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.TestGetFileChecksum
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.TestFileAppendRestart
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.fs.TestWebHdfsFileContextMainOperations
            hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.fs.permission.TestStickyBit
            hadoop.hdfs.TestLeaseRecovery
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.TestSnapshotPathINodes
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.hdfs.TestFileAppend3
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
            hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy
            hadoop.hdfs.TestPipelines
            hadoop.hdfs.TestAppendDifferentChecksum
            hadoop.hdfs.TestFileAppend4
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.TestHDFSFileSystemContract
            hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
            hadoop.hdfs.server.namenode.ha.TestDNFencing



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-10930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838499/HDFS-10930.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9b6450e18dfc 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8848a8a
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17526/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17526/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 15s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 482 unchanged - 12 fixed = 500 total (was 494) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch 1 line(s) with tabs. -1 findbugs 1m 58s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 39s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 58m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 78m 28s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Non-virtual method call in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File, long) passes null for non-null parameter of new org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaInputStreams(InputStream, InputStream, FsVolumeReference) At BlockPoolSlice.java:long) passes null for non-null parameter of new org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaInputStreams(InputStream, InputStream, FsVolumeReference) At BlockPoolSlice.java: [line 720] Failed junit tests hadoop.hdfs.TestDFSShell   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.TestDataTransferProtocol   hadoop.hdfs.TestReadWhileWriting   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.fs.TestHDFSFileContextMainOperations   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.namenode.ha.TestQuotasWithHA   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.fs.TestWebHdfsFileContextMainOperations   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.fs.permission.TestStickyBit   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.TestSnapshotPathINodes   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.TestFileAppend3   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs   hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.hdfs.TestFileAppend4   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.server.namenode.ha.TestDNFencing Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-10930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838499/HDFS-10930.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9b6450e18dfc 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8848a8a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17526/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17526/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Update the patch, clean up the instrumentation from the refactor work.

          Show
          xyao Xiaoyu Yao added a comment - Update the patch, clean up the instrumentation from the refactor work.
          Hide
          xyao Xiaoyu Yao added a comment -

          The instrumentation wrappers are moved into non-static methods of LocalReplica/LocalReplicaInPipeline/ReplicaInputStreams/ReplicaOutputstreams.

          Show
          xyao Xiaoyu Yao added a comment - The instrumentation wrappers are moved into non-static methods of LocalReplica/LocalReplicaInPipeline/ReplicaInputStreams/ReplicaOutputstreams.
          Hide
          xyao Xiaoyu Yao added a comment -

          Based on offline discussion with Arpit Agarwal, update the patch to wrap the instrumentation inside original datanode IO related classes with minimal static wrappers for better testability.

          Show
          xyao Xiaoyu Yao added a comment - Based on offline discussion with Arpit Agarwal , update the patch to wrap the instrumentation inside original datanode IO related classes with minimal static wrappers for better testability.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Daryn Sharp for the feedback. I've put my answers to the questions inline. Feel free to share your thoughts of other approaches.

          do we really need to move non-redundant and non-general use methods into a new class just to add a log line for duration?

          Refactor and adding trace is just the first step. It helps us add instrumentation and per volume I/O metrics in other follow up JIRAs.

          I'm not a huge fan of kitchen sink classes of static methods because it's counter to polymorphism and it usually makes tests, particularly mocks, wicked hard to write.

          DatanodeFaultInjection and mocks exist in datanode unit tests. The majority of the extracted static methods in DatanodeIO class wraps low level OS calls so that we can easily add datanode I/O instrumentation and collect finer grain metrics without churning datanode code everywhere. The test/mock can still be added at higher level from the callers' perspective with the verification the metrics update.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Daryn Sharp for the feedback. I've put my answers to the questions inline. Feel free to share your thoughts of other approaches. do we really need to move non-redundant and non-general use methods into a new class just to add a log line for duration? Refactor and adding trace is just the first step. It helps us add instrumentation and per volume I/O metrics in other follow up JIRAs. I'm not a huge fan of kitchen sink classes of static methods because it's counter to polymorphism and it usually makes tests, particularly mocks, wicked hard to write. DatanodeFaultInjection and mocks exist in datanode unit tests. The majority of the extracted static methods in DatanodeIO class wraps low level OS calls so that we can easily add datanode I/O instrumentation and collect finer grain metrics without churning datanode code everywhere. The test/mock can still be added at higher level from the callers' perspective with the verification the metrics update.
          Hide
          daryn Daryn Sharp added a comment -

          Took a cursory look, do we really need to move non-redundant and non-general use methods into a new class just to add a log line for duration? I'm not a huge fan of kitchen sink classes of static methods because it's counter to polymorphism and it usually makes tests, particularly mocks, wicked hard to write.

          Show
          daryn Daryn Sharp added a comment - Took a cursory look, do we really need to move non-redundant and non-general use methods into a new class just to add a log line for duration? I'm not a huge fan of kitchen sink classes of static methods because it's counter to polymorphism and it usually makes tests, particularly mocks, wicked hard to write.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 7m 35s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 365 unchanged - 4 fixed = 371 total (was 369)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 39s hadoop-hdfs-project_hadoop-hdfs generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
          -1 unit 59m 47s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          79m 58s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Dead store to duration in org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaOutputStreams.flushOrSync(boolean, DataNodeMetrics) At ReplicaOutputStreams.java:org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaOutputStreams.flushOrSync(boolean, DataNodeMetrics) At ReplicaOutputStreams.java:[line 156]
          Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.TestFileConcurrentReader
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.hdfs.TestWriteRead
            hadoop.hdfs.TestFileAppend3
            hadoop.hdfs.server.datanode.TestTransferRbw



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832772/HDFS-10930.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ac05ceda085e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8a09bf7
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17111/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17111/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17111/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17111/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17111/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17111/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 7m 35s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 365 unchanged - 4 fixed = 371 total (was 369) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 39s hadoop-hdfs-project_hadoop-hdfs generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) -1 unit 59m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 79m 58s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Dead store to duration in org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaOutputStreams.flushOrSync(boolean, DataNodeMetrics) At ReplicaOutputStreams.java:org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaOutputStreams.flushOrSync(boolean, DataNodeMetrics) At ReplicaOutputStreams.java: [line 156] Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.hdfs.TestWriteRead   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.server.datanode.TestTransferRbw Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832772/HDFS-10930.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ac05ceda085e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8a09bf7 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17111/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17111/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17111/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17111/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17111/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17111/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 30s trunk passed
          +1 compile 0m 58s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 58s the patch passed
          +1 compile 0m 56s the patch passed
          +1 javac 0m 56s the patch passed
          -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 11 new + 220 unchanged - 4 fixed = 231 total (was 224)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 15s the patch passed
          -1 javadoc 0m 44s hadoop-hdfs-project_hadoop-hdfs generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
          -1 unit 65m 53s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          88m 48s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestFileConcurrentReader
            hadoop.hdfs.server.datanode.TestTransferRbw
            hadoop.hdfs.TestFileAppend3
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.hdfs.TestWriteRead
            hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831591/HDFS-10930.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e4bd9cc6f79a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e57fa81
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17062/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17062/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 30s trunk passed +1 compile 0m 58s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 58s the patch passed +1 compile 0m 56s the patch passed +1 javac 0m 56s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 11 new + 220 unchanged - 4 fixed = 231 total (was 224) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 15s the patch passed -1 javadoc 0m 44s hadoop-hdfs-project_hadoop-hdfs generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) -1 unit 65m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 88m 48s Reason Tests Failed junit tests hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.server.datanode.TestTransferRbw   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.hdfs.TestWriteRead   hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831591/HDFS-10930.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e4bd9cc6f79a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e57fa81 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17062/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17062/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a initial patch that refactors IO related operations into DatanodeIO.java class. I will open follow up JIRAs on instrumentation, per volume IO related metrics collection, etc based on that.

          Show
          xyao Xiaoyu Yao added a comment - Attach a initial patch that refactors IO related operations into DatanodeIO.java class. I will open follow up JIRAs on instrumentation, per volume IO related metrics collection, etc based on that.
          Hide
          xyao Xiaoyu Yao added a comment -

          No, the goal is not to merge all these classes above into one mega-class. The proposal is to wrap IO related datanode operations for easy instrumentation, metric collection, logging and trouble shooting of datanode performance issues.

          Show
          xyao Xiaoyu Yao added a comment - No, the goal is not to merge all these classes above into one mega-class. The proposal is to wrap IO related datanode operations for easy instrumentation, metric collection, logging and trouble shooting of datanode performance issues.
          Hide
          daryn Daryn Sharp added a comment -

          I hope I'm misunderstanding this jira. Are you actually proposing to merge all those classes into a single mega-class? If yes, that gets a big . Else please elaborate.

          Show
          daryn Daryn Sharp added a comment - I hope I'm misunderstanding this jira. Are you actually proposing to merge all those classes into a single mega-class? If yes, that gets a big . Else please elaborate.

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development