Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7764

DirectoryScanner shouldn't abort the scan if one directory had an error

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1, 2.8.2
    • Component/s: datanode
    • Labels:
      None

      Description

      If there is an exception while preparing the ScanInfo for the blocks in the directory, DirectoryScanner is immediately throwing exception and coming out of the current scan cycle. The idea of this jira is to discuss & improve the exception handling mechanism.

      DirectoryScanner.java

          for (Entry<Integer, Future<ScanInfoPerBlockPool>> report :
              compilersInProgress.entrySet()) {
            try {
              dirReports[report.getKey()] = report.getValue().get();
            } catch (Exception ex) {
              LOG.error("Error compiling report", ex);
              // Propagate ex to DataBlockScanner to deal with
              throw new RuntimeException(ex);
            }
      
      1. HDFS-7764.patch
        2 kB
        Rakesh R
      2. HDFS-7764-01.patch
        7 kB
        Rakesh R
      3. HDFS-7764-02.patch
        9 kB
        Rakesh R
      4. HDFS-7764-03.patch
        9 kB
        Rakesh R
      5. HDFS-7764-04.patch
        10 kB
        Rakesh R

        Activity

        Hide
        rakeshr Rakesh R added a comment -

        Attached a patch where it will signal futureTask.cancel(true); subsequent tasks when there was an exception occurred earlier. Kindly review the same. Thanks!

        Show
        rakeshr Rakesh R added a comment - Attached a patch where it will signal futureTask.cancel(true); subsequent tasks when there was an exception occurred earlier. Kindly review the same. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12697679/HDFS-7764.patch
        against trunk revision e0ec071.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM

        The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestRollingUpgrade

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9514//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9514//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9514//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697679/HDFS-7764.patch against trunk revision e0ec071. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestRollingUpgrade Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9514//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9514//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9514//console This message is automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Can you explain why we would want to cancel the other pending tasks?

        It seems like having a problem generating a report for one volume should not impact generating reports for the other volumes.

        Did you see this code throw an exception in production?

        Show
        cmccabe Colin P. McCabe added a comment - Can you explain why we would want to cancel the other pending tasks? It seems like having a problem generating a report for one volume should not impact generating reports for the other volumes. Did you see this code throw an exception in production?
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Colin P. McCabe for the interest.

        Can you explain why we would want to cancel the other pending tasks?

        I was reviewing the code and saw public ScanInfoPerBlockPool call() throws Exception can throw exception. My thought was, while iterating if any of the compilersInProgress throws exception then will propogate this exception and skip the current cycle. So as a good coding practise it would be good to cancel all the given future tasks before exiting. Does this make sense to you?

        Did you see this code throw an exception in production?

        Truly not.

        Show
        rakeshr Rakesh R added a comment - Thanks Colin P. McCabe for the interest. Can you explain why we would want to cancel the other pending tasks? I was reviewing the code and saw public ScanInfoPerBlockPool call() throws Exception can throw exception. My thought was, while iterating if any of the compilersInProgress throws exception then will propogate this exception and skip the current cycle. So as a good coding practise it would be good to cancel all the given future tasks before exiting. Does this make sense to you? Did you see this code throw an exception in production? Truly not.
        Hide
        cmccabe Colin P. McCabe added a comment -

        I looked at this quickly, and it does look like the error handling is wrong here. We shouldn't be aborting the whole directory scan because one FileUtil#listFiles threw an exception. On the bright side, we do seem to log the first problem we hit here:

              try {
                files = FileUtil.listFiles(dir);
              } catch (IOException ioe) {
                LOG.warn("Exception occured while compiling report: ", ioe);
                // Ignore this directory and proceed.
                return report;
              }
        

        If you want to improve this, then I would say:

        • change it to use the jdk7 API that distinguishes between various different types of I/O issues rather than just returning null. This is probably as simple as using IOUtils#listDirectory instead of FileUtil.listFiles
        • don't abort the scan on every directory just because one had an error. You will need a unit test for this one.
        Show
        cmccabe Colin P. McCabe added a comment - I looked at this quickly, and it does look like the error handling is wrong here. We shouldn't be aborting the whole directory scan because one FileUtil#listFiles threw an exception. On the bright side, we do seem to log the first problem we hit here: try { files = FileUtil.listFiles(dir); } catch (IOException ioe) { LOG.warn( "Exception occured while compiling report: " , ioe); // Ignore this directory and proceed. return report; } If you want to improve this, then I would say: change it to use the jdk7 API that distinguishes between various different types of I/O issues rather than just returning null. This is probably as simple as using IOUtils#listDirectory instead of FileUtil.listFiles don't abort the scan on every directory just because one had an error. You will need a unit test for this one.
        Hide
        rakeshr Rakesh R added a comment -

        Yes Colin P. McCabe, your point makes sense. Let me try to implement as per your suggestions.

        Show
        rakeshr Rakesh R added a comment - Yes Colin P. McCabe , your point makes sense. Let me try to implement as per your suggestions.
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Colin P. McCabe for the suggestion. I've attached another patch addressing the same. Could you please review it again when you get a chance.

        Show
        rakeshr Rakesh R added a comment - Thanks Colin P. McCabe for the suggestion. I've attached another patch addressing the same. Could you please review it again when you get a chance.
        Hide
        rakeshr Rakesh R added a comment -

        Note:- I've modified the jira subject line and description reflecting the problem statement.

        Show
        rakeshr Rakesh R added a comment - Note:- I've modified the jira subject line and description reflecting the problem statement.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 6s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 3m 21s trunk passed
        +1 compile 0m 33s trunk passed with JDK v1.8.0_60
        +1 compile 0m 30s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 15s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_60
        +1 javadoc 1m 46s trunk passed with JDK v1.7.0_79
        +1 mvninstall 0m 39s the patch passed
        +1 compile 0m 31s the patch passed with JDK v1.8.0_60
        +1 javac 0m 31s the patch passed
        +1 compile 0m 31s the patch passed with JDK v1.7.0_79
        +1 javac 0m 31s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 2s the patch passed
        +1 javadoc 1m 7s the patch passed with JDK v1.8.0_60
        +1 javadoc 1m 46s the patch passed with JDK v1.7.0_79
        -1 unit 51m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
        -1 unit 49m 49s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        -1 asflicense 0m 21s Patch generated 58 ASF License warnings.
        120m 39s



        Reason Tests
        JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
        JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestDFSShell



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770587/HDFS-7764-01.patch
        JIRA Issue HDFS-7764
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux 8519bda7f692 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh
        git revision trunk / 73b9c7b
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13383/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 226MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13383/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 21s trunk passed +1 compile 0m 33s trunk passed with JDK v1.8.0_60 +1 compile 0m 30s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 15s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 5s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 39s the patch passed +1 compile 0m 31s the patch passed with JDK v1.8.0_60 +1 javac 0m 31s the patch passed +1 compile 0m 31s the patch passed with JDK v1.7.0_79 +1 javac 0m 31s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_79 -1 unit 51m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 49m 49s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 21s Patch generated 58 ASF License warnings. 120m 39s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770587/HDFS-7764-01.patch JIRA Issue HDFS-7764 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 8519bda7f692 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh git revision trunk / 73b9c7b Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13383/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13383/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13383/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 4s trunk passed
        +1 compile 0m 54s trunk passed with JDK v1.8.0_66
        +1 compile 0m 51s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 1m 2s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 2m 10s trunk passed
        +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 55s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 44s the patch passed with JDK v1.8.0_66
        +1 javac 0m 44s the patch passed
        +1 compile 0m 45s the patch passed with JDK v1.7.0_85
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 1m 0s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 15s the patch passed
        +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 50s the patch passed with JDK v1.7.0_85
        -1 unit 55m 12s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 58m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 19s Patch generated 56 ASF License warnings.
        143m 57s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.security.TestPermission
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.TestPersistBlocks
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
        JDK v1.7.0_85 Failed junit tests hadoop.security.TestPermission
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.TestBlockReaderLocal
          hadoop.hdfs.TestBlockReaderFactory



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770587/HDFS-7764-01.patch
        JIRA Issue HDFS-7764
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0fb948f14f6c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 611aa77
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13610/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13610/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 4s trunk passed +1 compile 0m 54s trunk passed with JDK v1.8.0_66 +1 compile 0m 51s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 52s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_66 +1 javac 0m 44s the patch passed +1 compile 0m 45s the patch passed with JDK v1.7.0_85 +1 javac 0m 45s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 15s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_85 -1 unit 55m 12s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 58m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 19s Patch generated 56 ASF License warnings. 143m 57s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.security.TestPermission   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.7.0_85 Failed junit tests hadoop.security.TestPermission   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.TestBlockReaderFactory Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770587/HDFS-7764-01.patch JIRA Issue HDFS-7764 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0fb948f14f6c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 611aa77 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13610/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13610/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13610/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Note:- It looks like the test case failures are unrelated to the patch.

        Show
        rakeshr Rakesh R added a comment - Note:- It looks like the test case failures are unrelated to the patch.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Thanks, Rakesh R.

        856	      if (fileNames.size() < 0) {
        857	        return report;
        858	      }
        

        What's the purpose of this if statement? The size of a list can't be less than 0.

        859	      files = new File[fileNames.size()];
        860	      for (int i = 0; i < fileNames.size(); i++) {
        861	        files[i] = new File(dir, fileNames.get(i));
        862	      }
        863	      Arrays.sort(files);
        

        It would be nice to avoid allocating all these new arrays. We don't really need them. We should be able to sort the list with List#sort, and we can turn the String objects into File objects one at a time in the for loop.

        Show
        cmccabe Colin P. McCabe added a comment - Thanks, Rakesh R . 856 if (fileNames.size() < 0) { 857 return report; 858 } What's the purpose of this if statement? The size of a list can't be less than 0. 859 files = new File[fileNames.size()]; 860 for ( int i = 0; i < fileNames.size(); i++) { 861 files[i] = new File(dir, fileNames.get(i)); 862 } 863 Arrays.sort(files); It would be nice to avoid allocating all these new arrays. We don't really need them. We should be able to sort the list with List#sort , and we can turn the String objects into File objects one at a time in the for loop.
        Hide
        rakeshr Rakesh R added a comment -

        Thanks a lot Colin P. McCabe for the useful comments. I've attached new patch addressing the same.

        Show
        rakeshr Rakesh R added a comment - Thanks a lot Colin P. McCabe for the useful comments. I've attached new patch addressing the same.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 1s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 39s trunk passed
        +1 compile 0m 58s trunk passed with JDK v1.8.0_66
        +1 compile 0m 53s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 1m 6s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 2m 24s trunk passed
        +1 javadoc 1m 25s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 16s trunk passed with JDK v1.7.0_85
        +1 mvninstall 1m 6s the patch passed
        +1 compile 1m 1s the patch passed with JDK v1.8.0_66
        +1 javac 1m 1s the patch passed
        +1 compile 0m 55s the patch passed with JDK v1.7.0_85
        +1 javac 0m 55s the patch passed
        -1 checkstyle 0m 20s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 48, now 50).
        +1 mvnsite 1m 14s the patch passed
        +1 mvneclipse 0m 17s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 39s the patch passed
        +1 javadoc 1m 40s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 16s the patch passed with JDK v1.7.0_85
        -1 unit 69m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 65m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 22s Patch generated 58 ASF License warnings.
        169m 53s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          hadoop.hdfs.TestSafeModeWithStripedFile
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          hadoop.hdfs.server.datanode.TestBlockReplacement



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774977/HDFS-7764-02.patch
        JIRA Issue HDFS-7764
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 23194387a460 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1c05393
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13704/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13704/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 39s trunk passed +1 compile 0m 58s trunk passed with JDK v1.8.0_66 +1 compile 0m 53s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 24s trunk passed +1 javadoc 1m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 16s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 6s the patch passed +1 compile 1m 1s the patch passed with JDK v1.8.0_66 +1 javac 1m 1s the patch passed +1 compile 0m 55s the patch passed with JDK v1.7.0_85 +1 javac 0m 55s the patch passed -1 checkstyle 0m 20s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 48, now 50). +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 39s the patch passed +1 javadoc 1m 40s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 16s the patch passed with JDK v1.7.0_85 -1 unit 69m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 65m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 22s Patch generated 58 ASF License warnings. 169m 53s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.TestSafeModeWithStripedFile JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774977/HDFS-7764-02.patch JIRA Issue HDFS-7764 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 23194387a460 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1c05393 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13704/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13704/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13704/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 22s trunk passed
        +1 compile 0m 42s trunk passed with JDK v1.8.0_66
        +1 compile 0m 44s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 2m 5s trunk passed
        +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 52s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 42s the patch passed with JDK v1.8.0_66
        +1 javac 0m 42s the patch passed
        +1 compile 0m 44s the patch passed with JDK v1.7.0_85
        +1 javac 0m 44s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 12s the patch passed
        +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 52s the patch passed with JDK v1.7.0_85
        -1 unit 60m 57s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 58m 33s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 25s Patch generated 56 ASF License warnings.
        148m 8s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.web.TestWebHDFS
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          hadoop.hdfs.server.namenode.TestCheckpoint



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775002/HDFS-7764-03.patch
        JIRA Issue HDFS-7764
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 713bac6b009e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1c05393
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13705/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13705/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 22s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 5s trunk passed +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 51s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_66 +1 javac 0m 42s the patch passed +1 compile 0m 44s the patch passed with JDK v1.7.0_85 +1 javac 0m 44s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 52s the patch passed with JDK v1.7.0_85 -1 unit 60m 57s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 58m 33s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 25s Patch generated 56 ASF License warnings. 148m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.namenode.TestCheckpoint Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775002/HDFS-7764-03.patch JIRA Issue HDFS-7764 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 713bac6b009e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1c05393 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13705/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13705/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13705/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Note: Looks like test failures are not related to my patch, please ignore this.

        Show
        rakeshr Rakesh R added a comment - Note: Looks like test failures are not related to my patch, please ignore this.
        Hide
        rakeshr Rakesh R added a comment -

        Colin P. McCabe, latest patch addresses all the review comments. Kindly review the latest patch when you are available. Thank you very much.

        Show
        rakeshr Rakesh R added a comment - Colin P. McCabe , latest patch addresses all the review comments. Kindly review the latest patch when you are available. Thank you very much.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Thanks for working on this, Rakesh R.

        @@ -737,8 +739,7 @@ private void addDifference(LinkedList<ScanInfo> diffRecord,
                   }
                 } catch (Exception ex) {
                   LOG.error("Error compiling report", ex);
        -          // Propagate ex to DataBlockScanner to deal with
        -          throw new RuntimeException(ex);
        +          // Ignore this exception and continue scanning the other directories
                 }
               }
        

        Hmm. I think we should print the storageID of the volume that had a problem. Also, I'm not sure the comment makes sense since we're not "ignoring" the exception-- we are logging it. Maybe just comment "continue scanning the other volumes?"

        Show
        cmccabe Colin P. McCabe added a comment - Thanks for working on this, Rakesh R . @@ -737,8 +739,7 @@ private void addDifference(LinkedList<ScanInfo> diffRecord, } } catch (Exception ex) { LOG.error( "Error compiling report" , ex); - // Propagate ex to DataBlockScanner to deal with - throw new RuntimeException(ex); + // Ignore this exception and continue scanning the other directories } } Hmm. I think we should print the storageID of the volume that had a problem. Also, I'm not sure the comment makes sense since we're not "ignoring" the exception-- we are logging it. Maybe just comment "continue scanning the other volumes?"
        Hide
        rakeshr Rakesh R added a comment -

        Thank you Colin P. McCabe for the reviews. I've attached another patch addressing the comments. Kindly review it again!

        FsVolumeSpi fsVolumeSpi = volumes.get(index);
        LOG.error("Error compiling report for the volume, StorageId: "
                      + fsVolumeSpi.getStorageID(), ex);
        
        Show
        rakeshr Rakesh R added a comment - Thank you Colin P. McCabe for the reviews. I've attached another patch addressing the comments. Kindly review it again! FsVolumeSpi fsVolumeSpi = volumes.get(index); LOG.error( "Error compiling report for the volume, StorageId: " + fsVolumeSpi.getStorageID(), ex);
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 6s trunk passed
        +1 compile 0m 57s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 47s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 46s the patch passed with JDK v1.8.0_66
        +1 javac 0m 47s the patch passed
        +1 compile 0m 42s the patch passed with JDK v1.7.0_91
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 59s the patch passed
        +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91
        -1 unit 73m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 60m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 31s Patch does not generate ASF License warnings.
        161m 13s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestINodeFile
          hadoop.hdfs.TestDFSStripedOutputStream
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.TestExternalBlockReader
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.TestEncryptionZonesWithKMS
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.TestAclsEndToEnd



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782555/HDFS-7764-04.patch
        JIRA Issue HDFS-7764
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6127258fbef9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9fbd579
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14133/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14133/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14133/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14133/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14133/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14133/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 6s trunk passed +1 compile 0m 57s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 48s the patch passed +1 compile 0m 46s the patch passed with JDK v1.8.0_66 +1 javac 0m 47s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_91 +1 javac 0m 42s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91 -1 unit 73m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 60m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 161m 13s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestINodeFile   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestExternalBlockReader   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestAclsEndToEnd Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782555/HDFS-7764-04.patch JIRA Issue HDFS-7764 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6127258fbef9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9fbd579 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14133/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14133/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14133/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14133/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14133/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14133/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Please ignore test case failures, its not related to the patch.

        Show
        rakeshr Rakesh R added a comment - Please ignore test case failures, its not related to the patch.
        Hide
        rakeshr Rakesh R added a comment -

        Hi Colin P. McCabe, Could you share your opinion about the latest patch when you get a chance. Thanks!

        Show
        rakeshr Rakesh R added a comment - Hi Colin P. McCabe , Could you share your opinion about the latest patch when you get a chance. Thanks!
        Hide
        cmccabe Colin P. McCabe added a comment -

        +1. Thanks, Rakesh R.

        Show
        cmccabe Colin P. McCabe added a comment - +1. Thanks, Rakesh R .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9204 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9204/)
        HDFS-7764. DirectoryScanner shouldn't abort the scan if one directory (cmccabe: rev f67149ab08bb49381def6c535ab4c4610e0a4221)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9204 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9204/ ) HDFS-7764 . DirectoryScanner shouldn't abort the scan if one directory (cmccabe: rev f67149ab08bb49381def6c535ab4c4610e0a4221) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
        Hide
        rakeshr Rakesh R added a comment -

        Thank you Colin P. McCabe for the good support in resolving this issue!

        Show
        rakeshr Rakesh R added a comment - Thank you Colin P. McCabe for the good support in resolving this issue!
        Hide
        cmccabe Colin P. McCabe added a comment -

        Thanks for working on this! Sorry that it took a while to get in.

        Show
        cmccabe Colin P. McCabe added a comment - Thanks for working on this! Sorry that it took a while to get in.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Cherry-picked to branch-2.8.

        Show
        arpitagarwal Arpit Agarwal added a comment - Cherry-picked to branch-2.8.

          People

          • Assignee:
            rakeshr Rakesh R
            Reporter:
            rakeshr Rakesh R
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development