Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9267

TestDiskError should get stored replicas through FsDatasetTestUtils.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:

      Description

      TestDiskError#testReplicationError scans local directories to verify blocks and metadata files, which leaks the details of FsDataset implementation.

      This JIRA will abstract the "scanning" operation to FsDatasetTestUtils.

      1. HDFS-9267.00.patch
        5 kB
        Lei (Eddy) Xu
      2. HDFS-9267.01.patch
        4 kB
        Lei (Eddy) Xu
      3. HDFS-9267.02.patch
        4 kB
        Lei (Eddy) Xu
      4. HDFS-9267.03.patch
        26 kB
        Lei (Eddy) Xu
      5. HDFS-9267.04.patch
        29 kB
        Lei (Eddy) Xu
      6. HDFS-9267.05.patch
        29 kB
        Lei (Eddy) Xu
      7. HDFS-9267.06.patch
        4 kB
        Lei (Eddy) Xu

        Issue Links

          Activity

          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Rebase to trunk.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Rebase to trunk.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 8m 2s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 25s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 1m 3s Pre-build of native portion
          -1 hdfs tests 49m 27s Tests failed in hadoop-hdfs.
              72m 52s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.server.datanode.TestDataNodeInitStorage
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767693/HDFS-9267.01.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 6c8b6f3
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13093/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13093/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13093/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13093/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13093/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 8m 2s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 25s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 1m 3s Pre-build of native portion -1 hdfs tests 49m 27s Tests failed in hadoop-hdfs.     72m 52s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.datanode.TestDataNodeInitStorage   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767693/HDFS-9267.01.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 6c8b6f3 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13093/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13093/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13093/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13093/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13093/console This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Fix checkstyle warnings.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Fix checkstyle warnings.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 8m 41s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 8m 52s There were no new javac warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 32s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 1m 8s Pre-build of native portion
          -1 hdfs tests 53m 36s Tests failed in hadoop-hdfs.
              79m 16s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
          Timed out tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767845/HDFS-9267.02.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / d759b4b
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13112/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13112/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13112/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13112/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 8m 41s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 52s There were no new javac warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 32s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 1m 8s Pre-build of native portion -1 hdfs tests 53m 36s Tests failed in hadoop-hdfs.     79m 16s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter Timed out tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767845/HDFS-9267.02.patch Optional Tests javac unit findbugs checkstyle git revision trunk / d759b4b Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13112/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13112/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13112/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13112/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Hmm. Good idea overall. Do you think it would be better to have an Iterator<Replica> here? That collection of all the replicas in the dataset could get pretty big in theory.

          Show
          cmccabe Colin P. McCabe added a comment - Hmm. Good idea overall. Do you think it would be better to have an Iterator<Replica> here? That collection of all the replicas in the dataset could get pretty big in theory.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Colin P. McCabe thanks a lot for the suggestions.

          Do you think it would be better to have an Iterator<Replica> here?

          In getStoredReplicas(), it needs to scan and load all on-disk replicas into a local replicaMap, to verify the contents on the disk. Returning a iterator of this local replicaMap has the same space complexity as returning a Collection, because this replicaMap is still referred by the iterator. Also it is less readable to implement isEmpty() using iterator (i.e., using it.hasNext()) in the following code:

          while (!utils.getStoredReplicas(bpid).isEmpty()) {
              Thread.sleep(100);
          }
          

          That collection of all the replicas in the dataset could get pretty big in theory.

          FsDatasetTestUtils is only used by HDFS unit tests, which should not have millions of blocks in one test. Will Hbase or other projects use this function? If the space is a concern, we could write a replica Scanner in the future.

          What do you think?

          Show
          eddyxu Lei (Eddy) Xu added a comment - Colin P. McCabe thanks a lot for the suggestions. Do you think it would be better to have an Iterator<Replica> here? In getStoredReplicas() , it needs to scan and load all on-disk replicas into a local replicaMap , to verify the contents on the disk. Returning a iterator of this local replicaMap has the same space complexity as returning a Collection, because this replicaMap is still referred by the iterator. Also it is less readable to implement isEmpty() using iterator (i.e., using it.hasNext() ) in the following code: while (!utils.getStoredReplicas(bpid).isEmpty()) { Thread .sleep(100); } That collection of all the replicas in the dataset could get pretty big in theory. FsDatasetTestUtils is only used by HDFS unit tests, which should not have millions of blocks in one test. Will Hbase or other projects use this function? If the space is a concern, we could write a replica Scanner in the future. What do you think?
          Hide
          cmccabe Colin P. McCabe added a comment -

          I don't see why you have to load all replicas into a map. An iterator could just store the file name of the previous replica, and know how to look up the next one.

          Also it is less readable to implement isEmpty() using iterator (i.e., using it.hasNext()) in the following code:

          If you know that the number of elements will be small (or 0) then just load them all into a list using Guava's Lists class or whatever.

          while (!Lists.newArrayList(utils.getStoredReplicas(bpid)).isEmpty()) {
              Thread.sleep(100);
          }
          

          The Iterator interface will work for large-scale tests and small-scale ones equally well, the collection one only works for small-scale tests. It's better to use an iterator.

          Show
          cmccabe Colin P. McCabe added a comment - I don't see why you have to load all replicas into a map. An iterator could just store the file name of the previous replica, and know how to look up the next one. Also it is less readable to implement isEmpty() using iterator (i.e., using it.hasNext()) in the following code: If you know that the number of elements will be small (or 0) then just load them all into a list using Guava's Lists class or whatever. while (!Lists.newArrayList(utils.getStoredReplicas(bpid)).isEmpty()) { Thread .sleep(100); } The Iterator interface will work for large-scale tests and small-scale ones equally well, the collection one only works for small-scale tests. It's better to use an iterator.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          I don't see why you have to load all replicas into a map.

          It is for re-using the code in FsVolume#getVolumeMap() to load on-disk files into Replicas.

          I will implement a Replica iterator then. Thanks a lot.

          Show
          eddyxu Lei (Eddy) Xu added a comment - I don't see why you have to load all replicas into a map. It is for re-using the code in FsVolume#getVolumeMap() to load on-disk files into Replicas . I will implement a Replica iterator then. Thanks a lot.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Colin P. McCabe.

          I updated the patch to provide an ReplicaIterator class and refactor the BlockPoolSlice to use it. The reason that using ReplicaIterator instead of java.util.Iterator is that it can IOException in next().

          Could you give some feedbacks? Thanks a lot.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Colin P. McCabe . I updated the patch to provide an ReplicaIterator class and refactor the BlockPoolSlice to use it. The reason that using ReplicaIterator instead of java.util.Iterator is that it can IOException in next() . Could you give some feedbacks? Thanks a lot.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 3m 20s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_60
          +1 compile 0m 34s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 18s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 1m 15s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 56s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_60
          +1 javac 0m 37s the patch passed
          +1 compile 0m 33s the patch passed with JDK v1.7.0_79
          +1 javac 0m 33s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 12s hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs issues.
          +1 javadoc 1m 14s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 6s the patch passed with JDK v1.7.0_79
          -1 unit 129m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 182m 48s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 2m 27s Patch generated 57 ASF License warnings.
          337m 19s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice$BlockPoolSliceReplicaIterator$DirIterator.next() can't throw NoSuchElementException At BlockPoolSlice.java:At BlockPoolSlice.java:[line 456]
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.TestReplication
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.namenode.ha.TestHAAppend
          JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestReplicaIterator
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
            org.apache.hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade
            org.apache.hadoop.hdfs.server.namenode.TestFileTruncate
            org.apache.hadoop.hdfs.TestPersistBlocks
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.TestReplication
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestReplaceDatanodeOnFailure
          JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestReplicaIterator
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
            org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade
            org.apache.hadoop.hdfs.TestPersistBlocks
            org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-07
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771134/HDFS-9267.03.patch
          JIRA Issue HDFS-9267
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux cbb42c78e378 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / bf6aa30
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13426/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13426/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 3m 20s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_60 +1 compile 0m 34s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 18s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 15s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 56s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 41s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_60 +1 javac 0m 37s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_79 +1 javac 0m 33s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 12s hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs issues. +1 javadoc 1m 14s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 6s the patch passed with JDK v1.7.0_79 -1 unit 129m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 182m 48s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 2m 27s Patch generated 57 ASF License warnings. 337m 19s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice$BlockPoolSliceReplicaIterator$DirIterator.next() can't throw NoSuchElementException At BlockPoolSlice.java:At BlockPoolSlice.java: [line 456] JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.TestReplication   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.namenode.ha.TestHAAppend JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestReplicaIterator   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   org.apache.hadoop.hdfs.TestDFSUpgradeFromImage   org.apache.hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade   org.apache.hadoop.hdfs.server.namenode.TestFileTruncate   org.apache.hadoop.hdfs.TestPersistBlocks JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestReplaceDatanodeOnFailure JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestReplicaIterator   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   org.apache.hadoop.hdfs.TestDFSUpgradeFromImage   org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade   org.apache.hadoop.hdfs.TestPersistBlocks   org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-07 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771134/HDFS-9267.03.patch JIRA Issue HDFS-9267 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux cbb42c78e378 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / bf6aa30 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13426/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13426/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13426/console This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Will fix these tests in the next patch.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Will fix these tests in the next patch.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          It fixes the test failures for JDK 7.
          Upload the patch to trigger a new jenkins.

          Show
          eddyxu Lei (Eddy) Xu added a comment - It fixes the test failures for JDK 7. Upload the patch to trigger a new jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 7m 45s trunk passed
          +1 compile 0m 40s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 56s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 47s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_66
          +1 javac 0m 39s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_85
          +1 javac 0m 40s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs issues.
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_85
          -1 unit 50m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 50m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
          127m 31s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice$BlockPoolSliceReplicaIterator$DirIterator.next() can't throw NoSuchElementException At BlockPoolSlice.java:At BlockPoolSlice.java:[line 464]
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.namenode.TestBackupNode
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774929/HDFS-9267.04.patch
          JIRA Issue HDFS-9267
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d83796e7129f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9b8e50b
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13698/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13698/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 7m 45s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 48s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_66 +1 javac 0m 39s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_85 +1 javac 0m 40s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs issues. +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_85 -1 unit 50m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 50m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 127m 31s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice$BlockPoolSliceReplicaIterator$DirIterator.next() can't throw NoSuchElementException At BlockPoolSlice.java:At BlockPoolSlice.java: [line 464] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.TestBackupNode JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774929/HDFS-9267.04.patch JIRA Issue HDFS-9267 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d83796e7129f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9b8e50b findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13698/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13698/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13698/console This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Fix findbugs warning.

          Ran the failed tests locally using JDK8 and JDK7, all passed on my laptop.

          Colin P. McCabe would you mind to take another look?

          Show
          eddyxu Lei (Eddy) Xu added a comment - Fix findbugs warning. Ran the failed tests locally using JDK8 and JDK7, all passed on my laptop. Colin P. McCabe would you mind to take another look?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 10m 11s trunk passed
          +1 compile 1m 3s trunk passed with JDK v1.8.0_66
          +1 compile 0m 55s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 2m 27s trunk passed
          +1 javadoc 1m 32s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 30s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 4s the patch passed
          +1 compile 1m 2s the patch passed with JDK v1.8.0_66
          +1 javac 1m 2s the patch passed
          +1 compile 0m 56s the patch passed with JDK v1.7.0_85
          +1 javac 0m 56s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 1m 9s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 41s the patch passed
          +1 javadoc 1m 32s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 30s the patch passed with JDK v1.7.0_85
          -1 unit 80m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 71m 48s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 24s Patch generated 56 ASF License warnings.
          188m 29s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775154/HDFS-9267.05.patch
          JIRA Issue HDFS-9267
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1aea753386a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3c4a34e
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13725/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13725/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 10m 11s trunk passed +1 compile 1m 3s trunk passed with JDK v1.8.0_66 +1 compile 0m 55s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 2m 27s trunk passed +1 javadoc 1m 32s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 30s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 4s the patch passed +1 compile 1m 2s the patch passed with JDK v1.8.0_66 +1 javac 1m 2s the patch passed +1 compile 0m 56s the patch passed with JDK v1.7.0_85 +1 javac 0m 56s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 1m 9s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 41s the patch passed +1 javadoc 1m 32s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 30s the patch passed with JDK v1.7.0_85 -1 unit 80m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 71m 48s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 24s Patch generated 56 ASF License warnings. 188m 29s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775154/HDFS-9267.05.patch JIRA Issue HDFS-9267 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1aea753386a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3c4a34e findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13725/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13725/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13725/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for working on this, Lei (Eddy) Xu. This is a lot more code than I was expecting. Patch 002 is really simple... I wish we could do something that was as simple.

          Does it make sense to do something like what patch 002 is doing and just change

          -292 public Collection<Replica> getStoredReplicas(String bpid) throws IOException {
          +292 public Iterator<Replica> getStoredReplicas(String bpid) throws IOException {
          

          and

          -308	    return ret;
          +308	    return ret.iterator();
          

          Since this is part of the test utils, we might not need to optimize it yet.

          Thanks again for working on this and sorry for the sometime slow reviews.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for working on this, Lei (Eddy) Xu . This is a lot more code than I was expecting. Patch 002 is really simple... I wish we could do something that was as simple. Does it make sense to do something like what patch 002 is doing and just change -292 public Collection<Replica> getStoredReplicas( String bpid) throws IOException { +292 public Iterator<Replica> getStoredReplicas( String bpid) throws IOException { and -308 return ret; +308 return ret.iterator(); Since this is part of the test utils, we might not need to optimize it yet. Thanks again for working on this and sorry for the sometime slow reviews.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Colin P. McCabe

          Thanks a lot for the reviews and the understandings.

          In the newest patch, I changed the getStoredReplicas(...) to return a Iterator<Replica> as you suggested.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Colin P. McCabe Thanks a lot for the reviews and the understandings. In the newest patch, I changed the getStoredReplicas(...) to return a Iterator<Replica> as you suggested.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 9m 3s trunk passed
          +1 compile 0m 54s trunk passed with JDK v1.8.0_66
          +1 compile 0m 49s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 11s trunk passed
          +1 javadoc 1m 22s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 11s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 59s the patch passed
          +1 compile 0m 54s the patch passed with JDK v1.8.0_66
          +1 javac 0m 54s the patch passed
          +1 compile 0m 50s the patch passed with JDK v1.7.0_91
          +1 javac 0m 50s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 23s the patch passed
          +1 javadoc 1m 23s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 17s the patch passed with JDK v1.7.0_91
          -1 unit 71m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 64m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 22s Patch generated 56 ASF License warnings.
          167m 50s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775362/HDFS-9267.06.patch
          JIRA Issue HDFS-9267
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0d5d8a3a2ee2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6b9a5be
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13737/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13737/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 9m 3s trunk passed +1 compile 0m 54s trunk passed with JDK v1.8.0_66 +1 compile 0m 49s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 11s trunk passed +1 javadoc 1m 22s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 11s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 59s the patch passed +1 compile 0m 54s the patch passed with JDK v1.8.0_66 +1 javac 0m 54s the patch passed +1 compile 0m 50s the patch passed with JDK v1.7.0_91 +1 javac 0m 50s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 23s the patch passed +1 javadoc 1m 23s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 17s the patch passed with JDK v1.7.0_91 -1 unit 71m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 64m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 22s Patch generated 56 ASF License warnings. 167m 50s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775362/HDFS-9267.06.patch JIRA Issue HDFS-9267 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0d5d8a3a2ee2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6b9a5be findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13737/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13737/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13737/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1. Thanks, Lei (Eddy) Xu.

          Show
          cmccabe Colin P. McCabe added a comment - +1. Thanks, Lei (Eddy) Xu .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8923 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8923/)
          HDFS-9267. TestDiskError should get stored replicas through (cmccabe: rev e02bbeb8862ee5bca572a0252e8ff3a3699eff5a)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8923 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8923/ ) HDFS-9267 . TestDiskError should get stored replicas through (cmccabe: rev e02bbeb8862ee5bca572a0252e8ff3a3699eff5a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #666 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/666/)
          HDFS-9267. TestDiskError should get stored replicas through (cmccabe: rev e02bbeb8862ee5bca572a0252e8ff3a3699eff5a)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #666 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/666/ ) HDFS-9267 . TestDiskError should get stored replicas through (cmccabe: rev e02bbeb8862ee5bca572a0252e8ff3a3699eff5a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java

            People

            • Assignee:
              eddyxu Lei (Eddy) Xu
              Reporter:
              eddyxu Lei (Eddy) Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development