Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8578

On upgrade, Datanode should process all storage/data dirs in parallel

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Right now, during upgrades datanode is processing all the storage dirs sequentially. Assume it takes ~20 mins to process a single storage dir then datanode which has ~10 disks will take around 3hours to come up.

      BlockPoolSliceStorage.java

         for (int idx = 0; idx < getNumStorageDirs(); idx++) {
            doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
            assert getCTime() == nsInfo.getCTime() 
                : "Data-node and name-node CTimes must be the same.";
          }
      

      It would save lots of time during major upgrades if datanode process all storagedirs/disks parallelly.

      Can we make datanode to process all storage dirs parallelly?

      1. h8578_20160218-branch-2.7-addendum.patch
        2 kB
        Vinayakumar B
      2. h8578_20160218.patch
        20 kB
        Tsz Wo Nicholas Sze
      3. h8578_20160216.patch
        18 kB
        Tsz Wo Nicholas Sze
      4. h8578_20160128b.patch
        19 kB
        Tsz Wo Nicholas Sze
      5. h8578_20160128.patch
        18 kB
        Tsz Wo Nicholas Sze
      6. h8578_20160117.patch
        51 kB
        Tsz Wo Nicholas Sze
      7. h8578_20151213.patch
        50 kB
        Tsz Wo Nicholas Sze
      8. h8578_20151212.patch
        50 kB
        Tsz Wo Nicholas Sze
      9. h8578_20151211b.patch
        46 kB
        Tsz Wo Nicholas Sze
      10. HDFS-8578-17.patch
        25 kB
        Vinayakumar B
      11. h8578_20151211.patch
        43 kB
        Tsz Wo Nicholas Sze
      12. HDFS-8578-16.patch
        23 kB
        Vinayakumar B
      13. h8578_20151210.patch
        41 kB
        Tsz Wo Nicholas Sze
      14. HDFS-8578-15.patch
        20 kB
        Vinayakumar B
      15. HDFS-8578-branch-2.7-003.patch
        20 kB
        Vinayakumar B
      16. HDFS-8578-14.patch
        38 kB
        Vinayakumar B
      17. HDFS-8578-branch-2.7-002.patch
        38 kB
        Vinayakumar B
      18. HDFS-8578-13.patch
        38 kB
        Vinayakumar B
      19. HDFS-8578-branch-2.7-001.patch
        38 kB
        Vinayakumar B
      20. HDFS-8578-12.patch
        19 kB
        Vinayakumar B
      21. HDFS-8578-11.patch
        19 kB
        Vinayakumar B
      22. HDFS-8578-10.patch
        18 kB
        Vinayakumar B
      23. HDFS-8578-09.patch
        18 kB
        Vinayakumar B
      24. HDFS-8578-08.patch
        18 kB
        Vinayakumar B
      25. HDFS-8578-07.patch
        17 kB
        Vinayakumar B
      26. HDFS-8578-06.patch
        16 kB
        Vinayakumar B
      27. HDFS-8578-05.patch
        11 kB
        Vinayakumar B
      28. HDFS-8578-04.patch
        11 kB
        Vinayakumar B
      29. HDFS-8578-03.patch
        11 kB
        Vinayakumar B
      30. HDFS-8578-branch-2.6.0.patch
        4 kB
        Raju Bairishetti
      31. HDFS-8578-02.patch
        10 kB
        Vinayakumar B
      32. HDFS-8578-01.patch
        4 kB
        Vinayakumar B

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          No problem. Thanks for checking.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - No problem. Thanks for checking.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Woops. Nevermind. I see the commits there now. Sorry about that

          Show
          ctrezzo Chris Trezzo added a comment - Woops. Nevermind. I see the commits there now. Sorry about that
          Hide
          ctrezzo Chris Trezzo added a comment -

          Tsz Wo Nicholas Sze Vinayakumar B This patch also needs to go into branch-2.8 (which was already cut when this was committed). Same for HDFS-9654 and HDFS-9730.

          Show
          ctrezzo Chris Trezzo added a comment - Tsz Wo Nicholas Sze Vinayakumar B This patch also needs to go into branch-2.8 (which was already cut when this was committed). Same for HDFS-9654 and HDFS-9730 .
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Varun Vasudev for reporting compilation problem and thanks Vinayakumar B for the quick fix.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Varun Vasudev for reporting compilation problem and thanks Vinayakumar B for the quick fix.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Fixed the compilation in branch-2.7

          Show
          vinayrpet Vinayakumar B added a comment - Fixed the compilation in branch-2.7
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attaching Committed addendum patch to fix compilation for reference

          Show
          vinayrpet Vinayakumar B added a comment - Attaching Committed addendum patch to fix compilation for reference
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Varun Vasudev, I will fix it soon.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Varun Vasudev , I will fix it soon.
          Hide
          vvasudev Varun Vasudev added a comment -

          This is causing compilation to fail on branch-2.7.

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-hdfs: Compilation failure: Compilation failure:
          [ERROR] /Users/vvasudev/Workspace/apache/committer-rw/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:[57,30] cannot find symbol
          [ERROR] symbol:   class DFSUtilClient
          [ERROR] location: package org.apache.hadoop.hdfs
          [ERROR] /Users/vvasudev/Workspace/apache/committer-rw/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:[444,17] cannot find symbol
          [ERROR] symbol:   variable DFSUtilClient
          [ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataStorage
          [ERROR] /Users/vvasudev/Workspace/apache/committer-rw/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:[493,17] cannot find symbol
          [ERROR] symbol:   variable DFSUtilClient
          [ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataStorage
          [ERROR] -> [Help 1]
          

          It looks like DFSUtilClient.java is missing in branch-2.7

          HWxxxxx:hadoop vvasudev$ git branch
            YARN-2139
            YARN-3926
            branch-2
          * branch-2.7
            branch-2.8
            trunk
          HWxxxxx:hadoop vvasudev$ git pull
          Already up-to-date.
          HWxxxxx:hadoop vvasudev$  find . -name DFSUtilClient.java
          HWxxxxx:hadoop vvasudev$
          
          Show
          vvasudev Varun Vasudev added a comment - This is causing compilation to fail on branch-2.7. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile ( default -compile) on project hadoop-hdfs: Compilation failure: Compilation failure: [ERROR] /Users/vvasudev/Workspace/apache/committer-rw/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:[57,30] cannot find symbol [ERROR] symbol: class DFSUtilClient [ERROR] location: package org.apache.hadoop.hdfs [ERROR] /Users/vvasudev/Workspace/apache/committer-rw/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:[444,17] cannot find symbol [ERROR] symbol: variable DFSUtilClient [ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataStorage [ERROR] /Users/vvasudev/Workspace/apache/committer-rw/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:[493,17] cannot find symbol [ERROR] symbol: variable DFSUtilClient [ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataStorage [ERROR] -> [Help 1] It looks like DFSUtilClient.java is missing in branch-2.7 HWxxxxx:hadoop vvasudev$ git branch YARN-2139 YARN-3926 branch-2 * branch-2.7 branch-2.8 trunk HWxxxxx:hadoop vvasudev$ git pull Already up-to-date. HWxxxxx:hadoop vvasudev$ find . -name DFSUtilClient.java HWxxxxx:hadoop vvasudev$
          Hide
          vinayrpet Vinayakumar B added a comment -

          I have committed this. Thanks Vinayakumar B for working on this.

          No. Thank You, for making this simple.

          Show
          vinayrpet Vinayakumar B added a comment - I have committed this. Thanks Vinayakumar B for working on this. No. Thank You, for making this simple.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks Vinayakumar B for working on this and thanks Chris Trezzo for reviewing the patches!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks Vinayakumar B for working on this and thanks Chris Trezzo for reviewing the patches!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9343 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9343/)
          HDFS-8578. On upgrade, Datanode should process all storage/data dirs in (szetszwo: rev 66289a3bf403f307844ea0b6ceed35b603d12c0b)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9343 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9343/ ) HDFS-8578 . On upgrade, Datanode should process all storage/data dirs in (szetszwo: rev 66289a3bf403f307844ea0b6ceed35b603d12c0b) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 12s trunk passed
          +1 compile 0m 43s trunk passed with JDK v1.8.0_72
          +1 compile 0m 45s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 4s trunk passed
          +1 javadoc 1m 10s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 53s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.8.0_72
          +1 javac 0m 40s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_95
          +1 javac 0m 42s the patch passed
          -1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: patch generated 3 new + 460 unchanged - 2 fixed = 463 total (was 462)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 16s the patch passed
          +1 javadoc 1m 7s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 52s the patch passed with JDK v1.7.0_95
          -1 unit 58m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 56m 13s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          141m 40s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
          JDK v1.7.0_95 Failed junit tests hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788615/h8578_20160218.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ed2c60d45a64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 748b6c0
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14545/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14545/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 12s trunk passed +1 compile 0m 43s trunk passed with JDK v1.8.0_72 +1 compile 0m 45s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 4s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 53s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 48s the patch passed +1 compile 0m 40s the patch passed with JDK v1.8.0_72 +1 javac 0m 40s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_95 +1 javac 0m 42s the patch passed -1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: patch generated 3 new + 460 unchanged - 2 fixed = 463 total (was 462) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 52s the patch passed with JDK v1.7.0_95 -1 unit 58m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 56m 13s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 141m 40s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics JDK v1.7.0_95 Failed junit tests hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788615/h8578_20160218.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ed2c60d45a64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 748b6c0 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14545/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14545/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8578_20160218.patch: addresses Chris' comment.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8578_20160218.patch: addresses Chris' comment.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Thanks Tsz Wo Nicholas Sze! I now see that the boolean ensures that properties are only written in the case where they have not already been written.

          One small nit that might make it clearer with respect to the boolean:
          In DataStorage.loadStorageDirectory, maybe pull the write properties into the if statement like so:

                //DataStorage.loadStorageDirectory
                if (!doTransition(sd, nsInfo, startOpt, callables, datanode.getConf())){
                  // 3. Update successfully loaded storage.
                  setServiceLayoutVersion(getServiceLayoutVersion());
                  writeProperties(sd);
                }
                return sd;
          

          Outside of this small nit, the patch looks good to me. +1.

          Thanks Tsz Wo Nicholas Sze and Vinayakumar B for all the work.

          Show
          ctrezzo Chris Trezzo added a comment - Thanks Tsz Wo Nicholas Sze ! I now see that the boolean ensures that properties are only written in the case where they have not already been written. One small nit that might make it clearer with respect to the boolean: In DataStorage.loadStorageDirectory, maybe pull the write properties into the if statement like so: //DataStorage.loadStorageDirectory if (!doTransition(sd, nsInfo, startOpt, callables, datanode.getConf())){ // 3. Update successfully loaded storage. setServiceLayoutVersion(getServiceLayoutVersion()); writeProperties(sd); } return sd; Outside of this small nit, the patch looks good to me. +1. Thanks Tsz Wo Nicholas Sze and Vinayakumar B for all the work.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Vinayakumar B, thanks for reviewing the patch!

          Chris Trezzo, do you want to take a look at the latest patch?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Vinayakumar B , thanks for reviewing the patch! Chris Trezzo , do you want to take a look at the latest patch?
          Hide
          vinayrpet Vinayakumar B added a comment -

          +1, Latest patch looks good to me.

          Show
          vinayrpet Vinayakumar B added a comment - +1, Latest patch looks good to me.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The failed tests do not seem related.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The failed tests do not seem related.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 15m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 54s trunk passed
          +1 compile 0m 40s trunk passed with JDK v1.8.0_72
          +1 compile 0m 42s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_72
          +1 javac 0m 37s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_95
          +1 javac 0m 40s the patch passed
          -1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 458 unchanged - 2 fixed = 460 total (was 460)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 45s the patch passed with JDK v1.7.0_95
          -1 unit 69m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 67m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          177m 48s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.TestCacheDirectives
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.TestReconstructStripedFile
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787963/h8578_20160216.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e3ccbebadbf5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8ed07bd
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14501/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14501/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 15m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 54s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_72 +1 compile 0m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 46s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_72 +1 javac 0m 37s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_95 +1 javac 0m 40s the patch passed -1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 458 unchanged - 2 fixed = 460 total (was 460) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_95 -1 unit 69m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 67m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 177m 48s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.TestCacheDirectives JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787963/h8578_20160216.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e3ccbebadbf5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8ed07bd Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14501/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14501/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8578_20160216.patch: sync'ed with trunk.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8578_20160216.patch: sync'ed with trunk.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > ... we change the order of execution for some of the operations. ...

          Chris Trezzo, thanks for the comment. I agree we should not change the order of execution of loading a directory. I think it is preserved by the patch. For DataStorage, doTransition is changed to return a boolean and it returns true when upgrade as shown below.

          //DataStorage.loadStorageDirectory
                if (doTransition(sd, nsInfo, startOpt, callables, datanode.getConf())){
                  return sd;
                }
          
                // 3. Update successfully loaded storage.
                setServiceLayoutVersion(getServiceLayoutVersion());
                writeProperties(sd);
          
                return sd;
          
          //DataStorage.doTransition
              // do upgrade
              if (this.layoutVersion > HdfsServerConstants.DATANODE_LAYOUT_VERSION) {
                if (federationSupported) {
                  // If the existing on-disk layout version supports federation,
                  // simply update the properties.
                  upgradeProperties(sd);
                } else {
                  doUpgradePreFederation(sd, nsInfo, callables, conf);
                }
                return true; // doUgrade already has written properties
              }
          
          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > ... we change the order of execution for some of the operations. ... Chris Trezzo , thanks for the comment. I agree we should not change the order of execution of loading a directory. I think it is preserved by the patch. For DataStorage, doTransition is changed to return a boolean and it returns true when upgrade as shown below. //DataStorage.loadStorageDirectory if (doTransition(sd, nsInfo, startOpt, callables, datanode.getConf())){ return sd; } // 3. Update successfully loaded storage. setServiceLayoutVersion(getServiceLayoutVersion()); writeProperties(sd); return sd; //DataStorage.doTransition // do upgrade if ( this .layoutVersion > HdfsServerConstants.DATANODE_LAYOUT_VERSION) { if (federationSupported) { // If the existing on-disk layout version supports federation, // simply update the properties. upgradeProperties(sd); } else { doUpgradePreFederation(sd, nsInfo, callables, conf); } return true ; // doUgrade already has written properties }
          Hide
          ctrezzo Chris Trezzo added a comment -

          Tsz Wo Nicholas Sze Vinayakumar B

          One major issue I see with this patch is that we change the order of execution for some of the operations. For example, in DataStorage#loadStorageDirectory the hard-linking of directories now happens after the following:

                // 3. Update successfully loaded storage.
                setServiceLayoutVersion(getServiceLayoutVersion());
                writeProperties(sd);
          

          This would present issues in the scenario where the data node fails after the above step, but before hard-linking actually occurs.

          Show
          ctrezzo Chris Trezzo added a comment - Tsz Wo Nicholas Sze Vinayakumar B One major issue I see with this patch is that we change the order of execution for some of the operations. For example, in DataStorage#loadStorageDirectory the hard-linking of directories now happens after the following: // 3. Update successfully loaded storage. setServiceLayoutVersion(getServiceLayoutVersion()); writeProperties(sd); This would present issues in the scenario where the data node fails after the above step, but before hard-linking actually occurs.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Looking through the patch now.

          Show
          ctrezzo Chris Trezzo added a comment - Looking through the patch now.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 51s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 44s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.8.0_66
          +1 javac 0m 36s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.7.0_91
          +1 javac 0m 37s the patch passed
          -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 457 unchanged - 2 fixed = 459 total (was 459)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 2s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91
          -1 unit 64m 28s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 65m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          154m 46s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.namenode.TestFSImageWithXAttr



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784887/h8578_20160128b.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c4b72de859a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 662e17b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14278/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14278/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 51s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 44s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 44s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_66 +1 javac 0m 36s the patch passed +1 compile 0m 37s the patch passed with JDK v1.7.0_91 +1 javac 0m 37s the patch passed -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 457 unchanged - 2 fixed = 459 total (was 459) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91 -1 unit 64m 28s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 65m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 154m 46s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.TestFSImageWithXAttr Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784887/h8578_20160128b.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c4b72de859a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 662e17b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14278/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14278/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14278/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          +1. Latest patch looks excellent
          Chris Trezzo, do you want to take a look?

          Show
          vinayrpet Vinayakumar B added a comment - +1. Latest patch looks excellent Chris Trezzo , do you want to take a look?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thank Vinay. Here is a new patch addressing your comment.

          h8578_20160128b.patch

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thank Vinay. Here is a new patch addressing your comment. h8578_20160128b.patch
          Hide
          vinayrpet Vinayakumar B added a comment -

          patch looks great.

          Simple nits.
          1. typo,doUgrade. I think it was missed in HDFS-9654.
          2. Fix possible checkstyles. Looks very trivial.

          3. Can skip to print this when no upgrade tasks?

          +    LOG.info("loadBlockPoolSliceStorage: " + tasks.size() + " upgrade tasks");
          +    for(UpgradeTask t : tasks) {
          +      try {
          +        success.add(t.future.get());
          +      } catch (ExecutionException e) {
          +        LOG.warn("Failed to add storage directory " + t.dataDir
          +            + " for block pool " + bpid, e);
          +      } catch (InterruptedException e) {
          +        throw DFSUtilClient.toInterruptedIOException("Task interrupted", e);
          +      }
          +    }
          +
          Show
          vinayrpet Vinayakumar B added a comment - patch looks great. Simple nits. 1. typo, doUgrade . I think it was missed in HDFS-9654 . 2. Fix possible checkstyles. Looks very trivial. 3. Can skip to print this when no upgrade tasks? + LOG.info( "loadBlockPoolSliceStorage: " + tasks.size() + " upgrade tasks" ); + for (UpgradeTask t : tasks) { + try { + success.add(t. future .get()); + } catch (ExecutionException e) { + LOG.warn( "Failed to add storage directory " + t.dataDir + + " for block pool " + bpid, e); + } catch (InterruptedException e) { + throw DFSUtilClient.toInterruptedIOException( "Task interrupted" , e); + } + } +
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 23s trunk passed
          +1 compile 0m 39s trunk passed with JDK v1.8.0_66
          +1 compile 0m 40s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 44s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.8.0_66
          +1 javac 0m 36s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_91
          +1 javac 0m 38s the patch passed
          -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 3 new + 458 unchanged - 0 fixed = 461 total (was 458)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 41s the patch passed with JDK v1.7.0_91
          -1 unit 50m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 49m 39s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          124m 24s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestFSEditLogLoader



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784813/h8578_20160128.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b324c40fa89c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 662e17b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14274/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14274/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14274/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14274/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14274/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 23s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_66 +1 compile 0m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 44s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 45s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_66 +1 javac 0m 36s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 3 new + 458 unchanged - 0 fixed = 461 total (was 458) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_91 -1 unit 50m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 49m 39s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 124m 24s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestFSEditLogLoader Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784813/h8578_20160128.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b324c40fa89c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 662e17b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14274/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14274/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14274/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14274/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14274/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          HDFS-9654 is now committed (thanks Vinayakumar B and Chris Trezzo for reviewing it.) The refactoring idea is great. Here is a much smaller patch:

          h8578_20160128.patch

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - HDFS-9654 is now committed (thanks Vinayakumar B and Chris Trezzo for reviewing it.) The refactoring idea is great. Here is a much smaller patch: h8578_20160128.patch
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9198 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9198/)
          HDFS-9654. Code refactoring for HDFS-8578. (szetszwo: rev 662e17b46a0f41ade6a304e12925b70b5d09fc2f)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/UpgradeUtilities.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9198 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9198/ ) HDFS-9654 . Code refactoring for HDFS-8578 . (szetszwo: rev 662e17b46a0f41ade6a304e12925b70b5d09fc2f) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/UpgradeUtilities.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 40s trunk passed
          +1 compile 0m 40s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 51s trunk passed
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 45s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_66
          +1 javac 0m 37s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_91
          +1 javac 0m 38s the patch passed
          -1 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: patch generated 5 new + 534 unchanged - 18 fixed = 539 total (was 552)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 1m 1s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91
          -1 unit 65m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 63m 15s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          154m 19s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782762/h8578_20160117.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cfd4b0db3e1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / da77f42
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14142/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14142/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14142/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14142/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14142/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14142/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 40s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 45s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 45s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_66 +1 javac 0m 37s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed -1 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: patch generated 5 new + 534 unchanged - 18 fixed = 539 total (was 552) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91 -1 unit 65m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 63m 15s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 154m 19s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782762/h8578_20160117.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cfd4b0db3e1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / da77f42 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14142/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14142/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14142/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14142/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14142/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14142/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8578_20160117.patch: addresses the review comments from Vinayakumar B and Chris Trezzo.

          Notes that the patch includes the refactoring code posted in HDFS-9654. Will remove refactoring code once HDFS-9654 has been committed.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8578_20160117.patch: addresses the review comments from Vinayakumar B and Chris Trezzo . Notes that the patch includes the refactoring code posted in HDFS-9654 . Will remove refactoring code once HDFS-9654 has been committed.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          1. Replace addBlockPoolStorage method with getBlockPoolSliceStorage. This just changes the synchronization around adding things to bpStorageMap. The change isn't necessary for the single threaded case, but doesn't hurt so we can commit it before parallelizing.
          2. Add the new createStorageID method to correctly handle upgrading from layout versions earlier then ADD_DATANODE_AND_STORAGE_UUIDS. This seems like a slightly separate issue, but I may be missing something. In any case, we can probably commit this as well without the parallelization.

          Sure, let's do a code refactoring before changing to processing all storage/data dirs in parallel. Filed HDFS-9654.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - 1. Replace addBlockPoolStorage method with getBlockPoolSliceStorage. This just changes the synchronization around adding things to bpStorageMap. The change isn't necessary for the single threaded case, but doesn't hurt so we can commit it before parallelizing. 2. Add the new createStorageID method to correctly handle upgrading from layout versions earlier then ADD_DATANODE_AND_STORAGE_UUIDS. This seems like a slightly separate issue, but I may be missing something. In any case, we can probably commit this as well without the parallelization. Sure, let's do a code refactoring before changing to processing all storage/data dirs in parallel. Filed HDFS-9654 .
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Vinayakumar B and Chris Trezzo, thanks for the comments. Will update the patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Vinayakumar B and Chris Trezzo , thanks for the comments. Will update the patch.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Tsz Wo Nicholas Sze
          Also, we might want to break this into a few subtasks. I could see there being the following separate patches:
          1. Replace addBlockPoolStorage method with getBlockPoolSliceStorage. This just changes the synchronization around adding things to bpStorageMap. The change isn't necessary for the single threaded case, but doesn't hurt so we can commit it before parallelizing.
          2. Add the new createStorageID method to correctly handle upgrading from layout versions earlier then ADD_DATANODE_AND_STORAGE_UUIDS. This seems like a slightly separate issue, but I may be missing something. In any case, we can probably commit this as well without the parallelization.
          3. Add the new ExecutorService to parallelize scan in hard-linking (i.e. doUpgrade). This patch would finally add the executor service and make the scan run in parallel.

          What do you think? Let me know if I am missing something here. It took me a little bit to wrap my head around the patch again. Splitting it up might help get some more eyes on the patch and have it committed sooner.

          Show
          ctrezzo Chris Trezzo added a comment - Tsz Wo Nicholas Sze Also, we might want to break this into a few subtasks. I could see there being the following separate patches: 1. Replace addBlockPoolStorage method with getBlockPoolSliceStorage. This just changes the synchronization around adding things to bpStorageMap. The change isn't necessary for the single threaded case, but doesn't hurt so we can commit it before parallelizing. 2. Add the new createStorageID method to correctly handle upgrading from layout versions earlier then ADD_DATANODE_AND_STORAGE_UUIDS. This seems like a slightly separate issue, but I may be missing something. In any case, we can probably commit this as well without the parallelization. 3. Add the new ExecutorService to parallelize scan in hard-linking (i.e. doUpgrade). This patch would finally add the executor service and make the scan run in parallel. What do you think? Let me know if I am missing something here. It took me a little bit to wrap my head around the patch again. Splitting it up might help get some more eyes on the patch and have it committed sooner.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Hi Tsz Wo Nicholas Sze, thanks for the updated patch! One comment so far:

          I think that we can get rid of the SubmissionService class and directly use an ExecutorService instead. You can use the shutdown and awaitTermination methods to wait for all of the doUpgrade tasks to complete. This way we will not need an extra class or to keep track of the number of tasks submitted.

          You might need to pass one more list of futures into the methods so that when the callables are submitted we can keep track of the futures to fill the success list at the end. I am not totally convinced that we need this yet though.

          Show
          ctrezzo Chris Trezzo added a comment - Hi Tsz Wo Nicholas Sze , thanks for the updated patch! One comment so far: I think that we can get rid of the SubmissionService class and directly use an ExecutorService instead. You can use the shutdown and awaitTermination methods to wait for all of the doUpgrade tasks to complete. This way we will not need an extra class or to keep track of the number of tasks submitted. You might need to pass one more list of futures into the methods so that when the callables are submitted we can keep track of the futures to fill the success list at the end. I am not totally convinced that we need this yet though.
          Hide
          vinayrpet Vinayakumar B added a comment -

          For h8578_20151213.patch

          1. I think executor service should be shutdown once the loading is over.

              SubmissionService(int numThreads) {
                super(Executors.newFixedThreadPool(numThreads));
                this.numThreads = numThreads;
              }
              

          2. I think there is some change in behavior, in case of any upgrade failure for any one of the disk.

              final int taskCount = service.getCount();
              LOG.info("loadBlockPoolSliceStorage: processing " + service);
              for(int i = 0; i < taskCount; i++) {
                success.add(service.takeResult());
              }
          

          In above part of the code after patch, if any disk fails to upgrade, entire upgrade will fail, even though other upgrades in-progress.. But earlier, only that disk was excluded and upgrade of other disks were continued. Need to handle exceptions here and continue with other disks.

          Nit:
          3. One checkstyle comment, line length > 80 also can be corrected.
          Other changes looks fine.

          +1 once addressed.

          Show
          vinayrpet Vinayakumar B added a comment - For h8578_20151213.patch 1. I think executor service should be shutdown once the loading is over. SubmissionService( int numThreads) { super (Executors.newFixedThreadPool(numThreads)); this .numThreads = numThreads; } 2. I think there is some change in behavior, in case of any upgrade failure for any one of the disk. final int taskCount = service.getCount(); LOG.info( "loadBlockPoolSliceStorage: processing " + service); for ( int i = 0; i < taskCount; i++) { success.add(service.takeResult()); } In above part of the code after patch, if any disk fails to upgrade, entire upgrade will fail, even though other upgrades in-progress.. But earlier, only that disk was excluded and upgrade of other disks were continued. Need to handle exceptions here and continue with other disks. Nit: 3. One checkstyle comment, line length > 80 also can be corrected. Other changes looks fine. +1 once addressed.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Allen Wittenauer, thanks a lot for explaining it. I also appreciate you effort on improving Jenkins!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Allen Wittenauer , thanks a lot for explaining it. I also appreciate you effort on improving Jenkins!
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The failed tests in the previous build do not seem related to the patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The failed tests in the previous build do not seem related to the patch.
          Hide
          aw Allen Wittenauer added a comment - - edited

          > 458m 22s

          From https://builds.apache.org/job/PreCommit-HDFS-Build/13854/console:

          Build timed out (after 300 minutes). Marking the build as aborted.
          

          The test run took too long. Jenkins thinks it is killing it before it completes and therefore grabs partial output. But because it is wrapped in a docker container, Yetus still finishes. There's a chance of potentially merging the output of multiple runs, but I'd have to do a lot more research on that.

          One other thing: testreport on Jenkins will only show the last JDK executed, not all of them. So in this case, JDK8 errors are only visible via what Yetus reports back.

          Show
          aw Allen Wittenauer added a comment - - edited > 458m 22s From https://builds.apache.org/job/PreCommit-HDFS-Build/13854/console: Build timed out (after 300 minutes). Marking the build as aborted. The test run took too long. Jenkins thinks it is killing it before it completes and therefore grabs partial output. But because it is wrapped in a docker container, Yetus still finishes. There's a chance of potentially merging the output of multiple runs, but I'd have to do a lot more research on that. One other thing: testreport on Jenkins will only show the last JDK executed, not all of them. So in this case, JDK8 errors are only visible via what Yetus reports back.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 10m 10s trunk passed
          +1 compile 1m 8s trunk passed with JDK v1.8.0_66
          +1 compile 0m 57s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 29s trunk passed
          +1 javadoc 1m 38s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 29s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 8s the patch passed
          +1 compile 1m 7s the patch passed with JDK v1.8.0_66
          +1 javac 1m 7s the patch passed
          +1 compile 0m 56s the patch passed with JDK v1.7.0_91
          +1 javac 0m 56s the patch passed
          -1 checkstyle 0m 21s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 543).
          +1 mvnsite 1m 10s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 43s the patch passed
          +1 javadoc 1m 35s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 31s the patch passed with JDK v1.7.0_91
          -1 unit 80m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 70m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 23s Patch generated 56 ASF License warnings.
          187m 39s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestAclsEndToEnd
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.ha.TestHASafeMode



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777372/h8578_20151213.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7c5ea4cee412 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0c3a53e
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13877/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13877/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 10m 10s trunk passed +1 compile 1m 8s trunk passed with JDK v1.8.0_66 +1 compile 0m 57s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 29s trunk passed +1 javadoc 1m 38s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 29s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 8s the patch passed +1 compile 1m 7s the patch passed with JDK v1.8.0_66 +1 javac 1m 7s the patch passed +1 compile 0m 56s the patch passed with JDK v1.7.0_91 +1 javac 0m 56s the patch passed -1 checkstyle 0m 21s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 543). +1 mvnsite 1m 10s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 43s the patch passed +1 javadoc 1m 35s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 31s the patch passed with JDK v1.7.0_91 -1 unit 80m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 70m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 23s Patch generated 56 ASF License warnings. 187m 39s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestAclsEndToEnd   hadoop.tracing.TestTracing   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.ha.TestHASafeMode Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777372/h8578_20151213.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7c5ea4cee412 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0c3a53e findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13877/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13877/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13877/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          There are a lot of test failures in the previous report. However, if we click the link https://builds.apache.org/job/PreCommit-HDFS-Build/13854/testReport/ , there are only two tests failed.

          Allen Wittenauer, do you know why?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - There are a lot of test failures in the previous report. However, if we click the link https://builds.apache.org/job/PreCommit-HDFS-Build/13854/testReport/ , there are only two tests failed. Allen Wittenauer , do you know why?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 21m 40s trunk passed
          +1 compile 3m 13s trunk passed with JDK v1.8.0_66
          +1 compile 2m 39s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 51s trunk passed
          +1 mvnsite 3m 5s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 5m 49s trunk passed
          +1 javadoc 2m 53s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 24s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 59s the patch passed
          +1 compile 2m 28s the patch passed with JDK v1.8.0_66
          +1 javac 2m 28s the patch passed
          +1 compile 1m 49s the patch passed with JDK v1.7.0_91
          +1 javac 1m 49s the patch passed
          -1 checkstyle 0m 37s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 543).
          +1 mvnsite 2m 14s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          -1 whitespace 0m 0s The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 4m 33s the patch passed
          +1 javadoc 2m 44s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 31s the patch passed with JDK v1.7.0_91
          -1 unit 204m 5s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 177m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 58s Patch generated 56 ASF License warnings.
          458m 22s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestBlockReaderLocal
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.tracing.TestTracing
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.web.TestWebHdfsTokens
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestDFSStripedInputStream
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.datanode.TestHSync
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777325/h8578_20151212.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 42d14de649af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7fb212e
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13854/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13854/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 21m 40s trunk passed +1 compile 3m 13s trunk passed with JDK v1.8.0_66 +1 compile 2m 39s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 51s trunk passed +1 mvnsite 3m 5s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 5m 49s trunk passed +1 javadoc 2m 53s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 24s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 59s the patch passed +1 compile 2m 28s the patch passed with JDK v1.8.0_66 +1 javac 2m 28s the patch passed +1 compile 1m 49s the patch passed with JDK v1.7.0_91 +1 javac 1m 49s the patch passed -1 checkstyle 0m 37s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 543). +1 mvnsite 2m 14s the patch passed +1 mvneclipse 0m 29s the patch passed -1 whitespace 0m 0s The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 4m 33s the patch passed +1 javadoc 2m 44s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 31s the patch passed with JDK v1.7.0_91 -1 unit 204m 5s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 177m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 58s Patch generated 56 ASF License warnings. 458m 22s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.tracing.TestTracing   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.tracing.TestTracing   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.TestHSync   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777325/h8578_20151212.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 42d14de649af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7fb212e findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13854/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13854/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13854/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 8m 9s trunk passed
          +1 compile 0m 44s trunk passed with JDK v1.8.0_66
          +1 compile 0m 44s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 3s trunk passed
          +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 49s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.8.0_66
          +1 javac 0m 43s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.7.0_91
          +1 javac 0m 43s the patch passed
          -1 checkstyle 0m 16s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 543).
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 15s the patch passed
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 52s the patch passed with JDK v1.7.0_91
          -1 unit 56m 3s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 53m 51s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 19s Patch generated 56 ASF License warnings.
          138m 15s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.qjournal.client.TestQuorumJournalManager
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
            hadoop.hdfs.server.namenode.TestCacheDirectives
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.shortcircuit.TestShortCircuitCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777372/h8578_20151213.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5d310977cfd7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7fb212e
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13855/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13855/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 8m 9s trunk passed +1 compile 0m 44s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 3s trunk passed +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 49s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 51s the patch passed +1 compile 0m 43s the patch passed with JDK v1.8.0_66 +1 javac 0m 43s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_91 +1 javac 0m 43s the patch passed -1 checkstyle 0m 16s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 543). +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 15s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 52s the patch passed with JDK v1.7.0_91 -1 unit 56m 3s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 53m 51s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 19s Patch generated 56 ASF License warnings. 138m 15s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.server.namenode.TestCacheDirectives JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777372/h8578_20151213.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5d310977cfd7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7fb212e findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13855/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13855/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13855/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8578_20151213.patch: Fixes an error message.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8578_20151213.patch: Fixes an error message.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 14m 42s trunk passed
          +1 compile 2m 2s trunk passed with JDK v1.8.0_66
          +1 compile 1m 30s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 45s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 34s trunk passed
          +1 javadoc 2m 20s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 49s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 51s the patch passed
          +1 compile 2m 10s the patch passed with JDK v1.8.0_66
          +1 javac 2m 10s the patch passed
          +1 compile 1m 34s the patch passed with JDK v1.7.0_91
          +1 javac 1m 34s the patch passed
          -1 checkstyle 0m 32s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 543).
          +1 mvnsite 2m 9s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          -1 whitespace 0m 0s The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 4m 58s the patch passed
          +1 javadoc 3m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 13s the patch passed with JDK v1.7.0_91
          -1 unit 175m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 152m 24s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 43s Patch generated 56 ASF License warnings.
          386m 17s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestBlockReaderLocal
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.web.TestWebHdfsTokens
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestReplication
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.TestSecureNameNode
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.web.TestWebHdfsFileSystemContract
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777325/h8578_20151212.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6bb83883ca7d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bf5295b
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13853/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13853/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 14m 42s trunk passed +1 compile 2m 2s trunk passed with JDK v1.8.0_66 +1 compile 1m 30s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 45s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 34s trunk passed +1 javadoc 2m 20s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 49s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 51s the patch passed +1 compile 2m 10s the patch passed with JDK v1.8.0_66 +1 javac 2m 10s the patch passed +1 compile 1m 34s the patch passed with JDK v1.7.0_91 +1 javac 1m 34s the patch passed -1 checkstyle 0m 32s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 543). +1 mvnsite 2m 9s the patch passed +1 mvneclipse 0m 36s the patch passed -1 whitespace 0m 0s The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 4m 58s the patch passed +1 javadoc 3m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 13s the patch passed with JDK v1.7.0_91 -1 unit 175m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 152m 24s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 43s Patch generated 56 ASF License warnings. 386m 17s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestReplication   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777325/h8578_20151212.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6bb83883ca7d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf5295b findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13853/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13853/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13853/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8578_20151212.patch: fixes the bug and cleans up the code.

          > ... thanks for taking this up and for simple way of parallel upgrade.

          No, thank you! It is based on your patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8578_20151212.patch: fixes the bug and cleans up the code. > ... thanks for taking this up and for simple way of parallel upgrade. No, thank you! It is based on your patch.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          It is a bug – it checks CTime too early. Also, upgrade does not need to check it. Will upload a new patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - It is a bug – it checks CTime too early. Also, upgrade does not need to check it. Will upload a new patch.
          Hide
          vinayrpet Vinayakumar B added a comment - - edited

          Tsz Wo Nicholas Sze, thanks for taking this up and for simple way of parallel upgrade.
          I have checked the latest patch.
          In case of normal "start-dfs.sh -upgrade", I observe below log.

          2015-12-12 22:34:09,126 WARN  common.Storage (BlockPoolSliceStorage.java:loadBpStorageDirectories(225)) - Failed to analyze storage directories for block pool BP-1461195821-127.0.0.1-1449939842938
          java.io.IOException: Data-node and name-node CTimes must be the same.
          	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:184)
          	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:221)
          	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:249)
          	at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:442)
          	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:526)
          	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1523)
          	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1484)
          	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:320)
          	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228)
          	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
          	at java.lang.Thread.run(Thread.java:745)

          Still I see that storagedir was added to successVolumes, that means, exceptions will be ignored..?

          Show
          vinayrpet Vinayakumar B added a comment - - edited Tsz Wo Nicholas Sze , thanks for taking this up and for simple way of parallel upgrade. I have checked the latest patch. In case of normal "start-dfs.sh -upgrade", I observe below log. 2015-12-12 22:34:09,126 WARN common.Storage (BlockPoolSliceStorage.java:loadBpStorageDirectories(225)) - Failed to analyze storage directories for block pool BP-1461195821-127.0.0.1-1449939842938 java.io.IOException: Data-node and name-node CTimes must be the same. at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:184) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:221) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:249) at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:442) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:526) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1523) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1484) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:320) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829) at java.lang.Thread.run(Thread.java:745) Still I see that storagedir was added to successVolumes, that means, exceptions will be ignored..?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 7m 28s trunk passed
          +1 compile 0m 37s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 51s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 46s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_66
          +1 javac 0m 39s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.7.0_91
          +1 javac 0m 41s the patch passed
          -1 checkstyle 0m 15s Patch generated 4 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 545).
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 1s the patch passed
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 45s the patch passed with JDK v1.7.0_91
          -1 unit 66m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 66m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 18s Patch generated 58 ASF License warnings.
          158m 59s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestDataStorage
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.server.datanode.TestBlockReplacement
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.TestDataStorage
            hadoop.hdfs.server.datanode.TestBlockReplacement



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777245/h8578_20151211b.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 391613856c21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f5a9114
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13849/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13849/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 7m 28s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 48s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_66 +1 javac 0m 39s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_91 +1 javac 0m 41s the patch passed -1 checkstyle 0m 15s Patch generated 4 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 558, now 545). +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 14s the patch passed -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 1s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_91 -1 unit 66m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 66m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 18s Patch generated 58 ASF License warnings. 158m 59s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestDataStorage   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.datanode.TestBlockReplacement JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.TestDataStorage   hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777245/h8578_20151211b.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 391613856c21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5a9114 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13849/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13849/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13849/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 10m 20s trunk passed
          +1 compile 1m 10s trunk passed with JDK v1.8.0_66
          +1 compile 0m 58s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 11s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 29s trunk passed
          +1 javadoc 1m 38s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 34s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 4s the patch passed
          +1 compile 1m 14s the patch passed with JDK v1.8.0_66
          +1 javac 1m 14s the patch passed
          +1 compile 0m 57s the patch passed with JDK v1.7.0_91
          +1 javac 0m 57s the patch passed
          -1 checkstyle 0m 21s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 684, now 678).
          +1 mvnsite 1m 10s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 41s the patch passed
          +1 javadoc 1m 37s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 33s the patch passed with JDK v1.7.0_91
          -1 unit 78m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 74m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 23s Patch generated 56 ASF License warnings.
          189m 39s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.TestDFSRollback
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.server.namenode.TestCheckpoint
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestDFSRollback
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestLeaseRecovery2
          JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777241/HDFS-8578-17.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 81364181f279 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f5a9114
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13848/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13848/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 10m 20s trunk passed +1 compile 1m 10s trunk passed with JDK v1.8.0_66 +1 compile 0m 58s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 11s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 29s trunk passed +1 javadoc 1m 38s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 34s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 4s the patch passed +1 compile 1m 14s the patch passed with JDK v1.8.0_66 +1 javac 1m 14s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_91 +1 javac 0m 57s the patch passed -1 checkstyle 0m 21s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 684, now 678). +1 mvnsite 1m 10s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 41s the patch passed +1 javadoc 1m 37s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 33s the patch passed with JDK v1.7.0_91 -1 unit 78m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 74m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 23s Patch generated 56 ASF License warnings. 189m 39s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestFileAppend   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSRollback   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestDFSRollback   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestLeaseRecovery2 JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777241/HDFS-8578-17.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 81364181f279 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5a9114 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13848/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13848/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13848/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment - - edited

          > Storage.storegeDirs was made synchronized lists, so this was not a problem.

          Synchronized lists does not synchronize list iteration.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - - edited > Storage.storegeDirs was made synchronized lists, so this was not a problem. Synchronized lists does not synchronize list iteration.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8578_20151211b.patch: fixes unit tests.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8578_20151211b.patch: fixes unit tests.
          Hide
          vinayrpet Vinayakumar B added a comment -

          One possible synchronization bug in the patch:

          Storage.storegeDirs was made synchronized lists, so this was not a problem.

          Another potential dead lock problem:

          Ooops.. Yup, this is a problem. Actually createStorageID doesn't need synchronization.

          Checked other parts as well for synchronization issues in DataStorage. I dont think any more exists.
          Uploading updated patch.

          Show
          vinayrpet Vinayakumar B added a comment - One possible synchronization bug in the patch: Storage.storegeDirs was made synchronized lists, so this was not a problem. Another potential dead lock problem: Ooops.. Yup, this is a problem. Actually createStorageID doesn't need synchronization. Checked other parts as well for synchronization issues in DataStorage. I dont think any more exists. Uploading updated patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 1s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 7m 59s trunk passed
          +1 compile 0m 41s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_66
          +1 javac 0m 39s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.7.0_91
          +1 javac 0m 41s the patch passed
          -1 checkstyle 0m 16s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 557, now 543).
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 0s The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 50s the patch passed with JDK v1.7.0_91
          -1 unit 52m 59s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 50m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 20s Patch generated 58 ASF License warnings.
          130m 46s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger
            hadoop.hdfs.server.datanode.TestDataStorage
            hadoop.hdfs.TestDatanodeLayoutUpgrade
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestDFSStartupVersions
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestDataStorage
            hadoop.hdfs.TestDatanodeLayoutUpgrade
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestDFSStartupVersions



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777186/h8578_20151211.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux eeed32562d3f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 576b569
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13841/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13841/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 7m 59s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 49s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_66 +1 javac 0m 39s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_91 +1 javac 0m 41s the patch passed -1 checkstyle 0m 16s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 557, now 543). +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_91 -1 unit 52m 59s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 50m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 20s Patch generated 58 ASF License warnings. 130m 46s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger   hadoop.hdfs.server.datanode.TestDataStorage   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSStartupVersions JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestDataStorage   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSStartupVersions Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777186/h8578_20151211.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux eeed32562d3f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 576b569 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13841/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13841/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13841/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Because, during bpStorage.recoverTransitionRead, linkBlocks will be made async, but while returning, immediately, rename of current-previous happens, also it logs as upgrade completed, even before all links are created and actual upgrade is over.

          It seem that we can move rename in call(). Here is a patch.

          h8578_20151211.patch

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Because, during bpStorage.recoverTransitionRead, linkBlocks will be made async, but while returning, immediately, rename of current-previous happens, also it logs as upgrade completed, even before all links are created and actual upgrade is over. It seem that we can move rename in call(). Here is a patch. h8578_20151211.patch
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          One possible synchronization bug in the patch:

          • call -> containsStorageDir -> iterate storageDirs
          • call -> addStorageDir -> storageDirs.add(sd)

          Another potential dead lock problem:

          • call -> loadStorageDirectory -> doTransition -> cannot enter createStorageID since it is synchronized.
          • synchronized addStorageLocations but waiting for call.

          I believe there are many similar problems which need to be studied carefully.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - One possible synchronization bug in the patch: call -> containsStorageDir -> iterate storageDirs call -> addStorageDir -> storageDirs.add(sd) Another potential dead lock problem: call -> loadStorageDirectory -> doTransition -> cannot enter createStorageID since it is synchronized. synchronized addStorageLocations but waiting for call. I believe there are many similar problems which need to be studied carefully.
          Hide
          vinayrpet Vinayakumar B added a comment - - edited

          Synchronization on DataStorage will affect only for this part of the code, which basically does the formatting/loading version file in datanode level. Since data loaded from version file will be same accross all VERSION file, it should not matter.

                    File root = dataDir.getFile();
                    try {
                      if (!containsStorageDir(root)) {
                        // It first ensures the datanode level format is completed.
                        StorageDirectory sd = loadStorageDirectory(datanode, nsInfo,
                            root, startOpt);
                        addStorageDir(sd);
                      } else {
                        LOG.info("Storage directory " + dataDir
                            + " has already been used.");
                      }
                    } catch (IOException e) {
                      LOG.warn("Failed to add Storage directory " + dataDir, e);
                      return null;
                    }

          Earlier, getDatanodeUuid() was called, which was getting blocked, so only I had removed synchronization on addStorageLocations().
          Still I believe there should not be any problem with synchronization.

          Only thing is, since I have restored synchronizarion on addStorageLocations(), synchronization on DataStorage#setFieldsFromProperties(..) is removed in latest patch. Still since all values are same values, there should not be any problem with this.

          Show
          vinayrpet Vinayakumar B added a comment - - edited Synchronization on DataStorage will affect only for this part of the code, which basically does the formatting/loading version file in datanode level. Since data loaded from version file will be same accross all VERSION file, it should not matter. File root = dataDir.getFile(); try { if (!containsStorageDir(root)) { // It first ensures the datanode level format is completed. StorageDirectory sd = loadStorageDirectory(datanode, nsInfo, root, startOpt); addStorageDir(sd); } else { LOG.info( "Storage directory " + dataDir + " has already been used." ); } } catch (IOException e) { LOG.warn( "Failed to add Storage directory " + dataDir, e); return null ; } Earlier, getDatanodeUuid() was called, which was getting blocked, so only I had removed synchronization on addStorageLocations() . Still I believe there should not be any problem with synchronization. Only thing is, since I have restored synchronizarion on addStorageLocations(), synchronization on DataStorage#setFieldsFromProperties(..) is removed in latest patch. Still since all values are same values, there should not be any problem with this.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Even if addStorageLocations is synchronized, the code inside call() does not have the lock. I think it is hard to make sure the code moved into call() is synchronized correctly.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Even if addStorageLocations is synchronized, the code inside call() does not have the lock. I think it is hard to make sure the code moved into call() is synchronized correctly.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Good catch! It needs more refactoring to make it work. Will wait for your patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Good catch! It needs more refactoring to make it work. Will wait for your patch.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attaching the updated patch,
          restoring the synchronization on addStorageLocations(), but made datanodeUuid as volatile and removed synchronizarion on get/setters.

          Show
          vinayrpet Vinayakumar B added a comment - Attaching the updated patch, restoring the synchronization on addStorageLocations() , but made datanodeUuid as volatile and removed synchronizarion on get/setters.
          Hide
          vinayrpet Vinayakumar B added a comment - - edited

          I think its a good idea to run all linkBlocks in parallel.
          But I do believe patch will not work.

          Because, during bpStorage.recoverTransitionRead, linkBlocks will be made async, but while returning, immediately, rename of current-previous happens, also it logs as upgrade completed, even before all links are created and actual upgrade is over.

          In my v15 patch, synchronization was removed on addStorageLocations because, getDatanodeUuid() was also synchronized, and threads were getting blocked because of this.
          But now, since datanodeUuId is updated only once, we can remove synchronization on this. and can make it volatile.
          Will update the patch shortly.

          Show
          vinayrpet Vinayakumar B added a comment - - edited I think its a good idea to run all linkBlocks in parallel. But I do believe patch will not work. Because, during bpStorage.recoverTransitionRead , linkBlocks will be made async, but while returning, immediately, rename of current-previous happens, also it logs as upgrade completed, even before all links are created and actual upgrade is over. In my v15 patch, synchronization was removed on addStorageLocations because, getDatanodeUuid() was also synchronized, and threads were getting blocked because of this. But now, since datanodeUuId is updated only once, we can remove synchronization on this. and can make it volatile. Will update the patch shortly.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8578_20151210.patch: execute hardlink tasks in parallel.

          The patch is big due to a lot of method header changes. The change actually is simple and safe since the hard link code are almost static. There is no synchronization issues.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8578_20151210.patch: execute hardlink tasks in parallel. The patch is big due to a lot of method header changes. The change actually is simple and safe since the hard link code are almost static. There is no synchronization issues.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Vinayakumar B for the new patch.

          I think we have to keep addStorageLocations synchronized since addStorageLocations involves a lot of code. It is very hard to keep synchronization correct if addStorageLocations is not synchronized. In case that there is a synchronization bug, the datanode memory state may be inconsistent and result in data loss. So we should be very careful here.

          It would be great if we can process (including load, upgrade, recover) all the directories in parallel. However, it is hard to get everything correct. Let's focus on only upgrade since it is our problem today.

          I played around the code. It seems that it is relatively easy to run upgrade in parallel since the hardlink related code are mostly static. I will base on Vinay's patch to work on a patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Vinayakumar B for the new patch. I think we have to keep addStorageLocations synchronized since addStorageLocations involves a lot of code. It is very hard to keep synchronization correct if addStorageLocations is not synchronized. In case that there is a synchronization bug, the datanode memory state may be inconsistent and result in data loss. So we should be very careful here. It would be great if we can process (including load, upgrade, recover) all the directories in parallel. However, it is hard to get everything correct. Let's focus on only upgrade since it is our problem today. I played around the code. It seems that it is relatively easy to run upgrade in parallel since the hardlink related code are mostly static. I will base on Vinay's patch to work on a patch.
          Hide
          ctrezzo Chris Trezzo added a comment -

          I agree with Tsz Wo Nicholas Sze let's fix the OOM part in a follow up jira. It wound up being more complicated then I originally thought. I also agree that we should make the number of parallel storage dir threads configurable so that users have some way to control the memory footprint of the upgrade.

          Show
          ctrezzo Chris Trezzo added a comment - I agree with Tsz Wo Nicholas Sze let's fix the OOM part in a follow up jira. It wound up being more complicated then I originally thought. I also agree that we should make the number of parallel storage dir threads configurable so that users have some way to control the memory footprint of the upgrade.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 9m 26s trunk passed
          +1 compile 0m 54s trunk passed with JDK v1.8.0_66
          +1 compile 0m 49s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 24s trunk passed
          +1 javadoc 1m 20s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 3s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 56s the patch passed
          +1 compile 0m 52s the patch passed with JDK v1.8.0_66
          +1 javac 0m 52s the patch passed
          +1 compile 0m 51s the patch passed with JDK v1.7.0_91
          +1 javac 0m 51s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 27s the patch passed
          +1 javadoc 1m 18s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 7s the patch passed with JDK v1.7.0_91
          -1 unit 63m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 60m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 27s Patch generated 58 ASF License warnings.
          155m 50s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestQuota
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestEncryptionZones
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776731/HDFS-8578-15.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1050fe52e08d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 132478e
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13821/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13821/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 9m 26s trunk passed +1 compile 0m 54s trunk passed with JDK v1.8.0_66 +1 compile 0m 49s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 24s trunk passed +1 javadoc 1m 20s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 3s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 56s the patch passed +1 compile 0m 52s the patch passed with JDK v1.8.0_66 +1 javac 0m 52s the patch passed +1 compile 0m 51s the patch passed with JDK v1.7.0_91 +1 javac 0m 51s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 27s the patch passed +1 javadoc 1m 18s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 7s the patch passed with JDK v1.7.0_91 -1 unit 63m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 60m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 27s Patch generated 58 ASF License warnings. 155m 50s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestQuota   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestEncryptionZones JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776731/HDFS-8578-15.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1050fe52e08d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 132478e findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13821/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13821/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13821/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          HDFS-9536 has been created to follow-up #2, OOM issue during parallel upgrade.
          Lets take further discussion about that OOM there.

          Show
          vinayrpet Vinayakumar B added a comment - HDFS-9536 has been created to follow-up #2, OOM issue during parallel upgrade. Lets take further discussion about that OOM there.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Updated patch as suggested by Tsz Wo Nicholas Sze.

          1. Introduced config dfs.datanode.parallel.volumes.load.threads.num which defaults to # of data dirs configured.
          Show
          vinayrpet Vinayakumar B added a comment - Updated patch as suggested by Tsz Wo Nicholas Sze . Introduced config dfs.datanode.parallel.volumes.load.threads.num which defaults to # of data dirs configured.
          Hide
          vinayrpet Vinayakumar B added a comment -

          I suggest that we fix #1 first and then fix #2 in a separated JIRA in order to keep the patch simple. Also, #2 need more discussion. We should also have numParallelThreads configurable for the moment due to #2. At least, users could set it to numParallelThreads to 1 to avoid #2. What do you think?

          Yes, we can take solving OOM problem in separate jira, just not to hurry.
          I will update the patch for #1, along with configuration for numParallelThreads.

          Show
          vinayrpet Vinayakumar B added a comment - I suggest that we fix #1 first and then fix #2 in a separated JIRA in order to keep the patch simple. Also, #2 need more discussion. We should also have numParallelThreads configurable for the moment due to #2. At least, users could set it to numParallelThreads to 1 to avoid #2. What do you think? Yes, we can take solving OOM problem in separate jira, just not to hurry. I will update the patch for #1, along with configuration for numParallelThreads.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > ... Also, #2 need more discussion. ...

          Datanode's memory should be large enough to hold all the blocks, otherwise, it won't run. So I think using BlockingQueue with a fixed capacity may not be the best way to solve the problem. We should be able to reduce the memory footprint in the data structures.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > ... Also, #2 need more discussion. ... Datanode's memory should be large enough to hold all the blocks, otherwise, it won't run. So I think using BlockingQueue with a fixed capacity may not be the best way to solve the problem. We should be able to reduce the memory footprint in the data structures.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          There are two different problems:

          1. data dirs are processed sequentially;
          2. OOM if all data dirs are processed in parallel.

          I suggest that we fix #1 first and then fix #2 in a separated JIRA in order to keep the patch simple. Also, #2 need more discussion. We should also have numParallelThreads configurable for the moment due to #2. At least, users could set it to numParallelThreads to 1 to avoid #2. What do you think?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - There are two different problems: data dirs are processed sequentially; OOM if all data dirs are processed in parallel. I suggest that we fix #1 first and then fix #2 in a separated JIRA in order to keep the patch simple. Also, #2 need more discussion. We should also have numParallelThreads configurable for the moment due to #2. At least, users could set it to numParallelThreads to 1 to avoid #2. What do you think?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 8m 53s trunk passed
          +1 compile 0m 50s trunk passed with JDK v1.8.0_66
          +1 compile 0m 46s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 17s trunk passed
          +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 1s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 56s the patch passed
          +1 compile 0m 51s the patch passed with JDK v1.8.0_66
          +1 javac 0m 51s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.7.0_91
          +1 javac 0m 47s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 22s the patch passed
          +1 javadoc 1m 15s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 2s the patch passed with JDK v1.7.0_91
          -1 unit 61m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 55m 3s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 20s Patch generated 56 ASF License warnings.
          158m 12s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestBlockReaderLocal
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.hdfs.server.namenode.TestCacheDirectives



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776547/HDFS-8578-14.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2c7fd65361b4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 50edcb9
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13814/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13814/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 8m 53s trunk passed +1 compile 0m 50s trunk passed with JDK v1.8.0_66 +1 compile 0m 46s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 17s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 1s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 56s the patch passed +1 compile 0m 51s the patch passed with JDK v1.8.0_66 +1 javac 0m 51s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_91 +1 javac 0m 47s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 22s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 2s the patch passed with JDK v1.7.0_91 -1 unit 61m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 55m 3s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 20s Patch generated 56 ASF License warnings. 158m 12s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.security.TestDelegationTokenForProxyUser JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.namenode.TestCacheDirectives Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776547/HDFS-8578-14.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2c7fd65361b4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 50edcb9 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13814/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13814/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13814/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          You're misinterpreting the output entirely.

          a. The first message means the value of JAVA_HOME that was inherited from pre-Docker doesn't work in Docker. Since Yetus is running under Docker and wasn't told to use anything different, it's going to find a new one to use instead.

          b. Oh, --jenkins was passed, so let's turn on all the specific bits that are enabled when running under Jenkins regardless of whether this is Docker or not.

          You'll note:

          apache-yetus-201a378/shelldocs/
          apache-yetus-201a378/shelldocs/shelldocs.py
          apache-yetus-201a378/yetus-project/
          apache-yetus-201a378/yetus-project/pom.xml
          Running in Jenkins mode
          Processing: HDFS-8578
          HDFS-8578 patch is being downloaded at Tue Dec  8 10:29:37 UTC 2015 from
          https://issues.apache.org/jira/secure/attachment/12776283/HDFS-8578-13.patch
          

          ... that Yetus tell you it is running in Jenkins mode immediately after untarring too. Jenkins and Docker are not exclusive modes to each other.

          Show
          aw Allen Wittenauer added a comment - You're misinterpreting the output entirely. a. The first message means the value of JAVA_HOME that was inherited from pre-Docker doesn't work in Docker. Since Yetus is running under Docker and wasn't told to use anything different, it's going to find a new one to use instead. b. Oh, --jenkins was passed, so let's turn on all the specific bits that are enabled when running under Jenkins regardless of whether this is Docker or not. You'll note: apache-yetus-201a378/shelldocs/ apache-yetus-201a378/shelldocs/shelldocs.py apache-yetus-201a378/yetus-project/ apache-yetus-201a378/yetus-project/pom.xml Running in Jenkins mode Processing: HDFS-8578 HDFS-8578 patch is being downloaded at Tue Dec 8 10:29:37 UTC 2015 from https: //issues.apache.org/jira/secure/attachment/12776283/HDFS-8578-13.patch ... that Yetus tell you it is running in Jenkins mode immediately after untarring too. Jenkins and Docker are not exclusive modes to each other.
          Hide
          vinayrpet Vinayakumar B added a comment -

          I mean. Its falling back to jenkins mode due to Java missing in image.

          Show
          vinayrpet Vinayakumar B added a comment - I mean. Its falling back to jenkins mode due to Java missing in image.
          Hide
          aw Allen Wittenauer added a comment -

          ... except that message only shows up when running under Docker.

          Show
          aw Allen Wittenauer added a comment - ... except that message only shows up when running under Docker.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Allen Wittenauer, yetus is not actually running in docker.

          JAVA_HOME: /home/jenkins/tools/java/jdk1.7.0_79 does not exist. Dockermode: attempting to switch to another.
          Running in Jenkins mode
          Processing: HDFS-8578
          Show
          vinayrpet Vinayakumar B added a comment - Allen Wittenauer , yetus is not actually running in docker. JAVA_HOME: /home/jenkins/tools/java/jdk1.7.0_79 does not exist. Dockermode: attempting to switch to another. Running in Jenkins mode Processing: HDFS-8578
          Hide
          vinayrpet Vinayakumar B added a comment -

          Fixed findbugs and checkstyles

          Show
          vinayrpet Vinayakumar B added a comment - Fixed findbugs and checkstyles
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 13m 13s trunk passed
          +1 compile 1m 58s trunk passed with JDK v1.8.0_66
          +1 compile 1m 21s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 20s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          +1 findbugs 2m 41s trunk passed
          +1 javadoc 1m 41s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 47s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 14s the patch passed
          +1 compile 1m 18s the patch passed with JDK v1.8.0_66
          +1 javac 1m 18s the patch passed
          +1 compile 1m 5s the patch passed with JDK v1.7.0_91
          +1 javac 1m 5s the patch passed
          -1 checkstyle 0m 22s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 173, now 167).
          +1 mvnsite 1m 15s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 54s hadoop-hdfs-project/hadoop-hdfs introduced 2 new FindBugs issues.
          +1 javadoc 1m 36s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 38s the patch passed with JDK v1.7.0_91
          -1 unit 103m 36s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 95m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 30s Patch generated 56 ASF License warnings.
          242m 37s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Exceptional return value of java.io.File.delete() ignored in org.apache.hadoop.hdfs.server.datanode.DataStorage.handleDuplicates(ArrayList) At DataStorage.java:ignored in org.apache.hadoop.hdfs.server.datanode.DataStorage.handleDuplicates(ArrayList) At DataStorage.java:[line 1177]
            new java.io.InterruptedIOException(String) not thrown in org.apache.hadoop.hdfs.server.datanode.DataStorage$3.call() At DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage$3.call() At DataStorage.java:[line 1120]
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestDatanodeLayoutUpgrade
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
            org.apache.hadoop.tracing.TestTracing
            org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestDatanodeLayoutUpgrade
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776283/HDFS-8578-13.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 88b98e96f2d0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fc47084
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13802/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13802/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 13m 13s trunk passed +1 compile 1m 58s trunk passed with JDK v1.8.0_66 +1 compile 1m 21s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 2m 41s trunk passed +1 javadoc 1m 41s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 47s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 14s the patch passed +1 compile 1m 18s the patch passed with JDK v1.8.0_66 +1 javac 1m 18s the patch passed +1 compile 1m 5s the patch passed with JDK v1.7.0_91 +1 javac 1m 5s the patch passed -1 checkstyle 0m 22s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 173, now 167). +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 54s hadoop-hdfs-project/hadoop-hdfs introduced 2 new FindBugs issues. +1 javadoc 1m 36s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 38s the patch passed with JDK v1.7.0_91 -1 unit 103m 36s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 95m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 30s Patch generated 56 ASF License warnings. 242m 37s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Exceptional return value of java.io.File.delete() ignored in org.apache.hadoop.hdfs.server.datanode.DataStorage.handleDuplicates(ArrayList) At DataStorage.java:ignored in org.apache.hadoop.hdfs.server.datanode.DataStorage.handleDuplicates(ArrayList) At DataStorage.java: [line 1177]   new java.io.InterruptedIOException(String) not thrown in org.apache.hadoop.hdfs.server.datanode.DataStorage$3.call() At DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage$3.call() At DataStorage.java: [line 1120] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS   org.apache.hadoop.tracing.TestTracing   org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776283/HDFS-8578-13.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 88b98e96f2d0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fc47084 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13802/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13802/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13802/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attaching the patches for trunk and branch-2.7 for the BlockingQueue approach to solve OOM error.

          Had to change the way duplicate entries are handled.
          Earlier, duplicates were processed on the list itself.

          Now after creating all links, FileAlreadyExistException will be thrown during link creation. based on this, duplicates list will be formed.

          Once all link workers are finished, duplicate entries will be processed against the already linked files.

          Please review.

          Show
          vinayrpet Vinayakumar B added a comment - Attaching the patches for trunk and branch-2.7 for the BlockingQueue approach to solve OOM error. Had to change the way duplicate entries are handled. Earlier, duplicates were processed on the list itself. Now after creating all links, FileAlreadyExistException will be thrown during link creation. based on this, duplicates list will be formed. Once all link workers are finished, duplicate entries will be processed against the already linked files. Please review.
          Hide
          ctrezzo Chris Trezzo added a comment -

          One possible solution to limit memory growth is to rewrite DataStorage#linkBlocks and DataStorage#linkBlocksHelper to use a producer/consumer type model with a bounded queue. For example, you could use a LinkedBlockingQueue with a fixed capacity. The logic roughly in the DataStorage#linkBlocksHelper method would be the producer that adds LinkArgs objects to the queue. The logic in the linkWorkers ExecutorService would simply do what it does now, except it would pull LinkArgs objects out of the queue.

          Vinayakumar B Do you want to take a crack at this? If you don't have time in the next day or so let me know and I will take a look.

          Show
          ctrezzo Chris Trezzo added a comment - One possible solution to limit memory growth is to rewrite DataStorage#linkBlocks and DataStorage#linkBlocksHelper to use a producer/consumer type model with a bounded queue. For example, you could use a LinkedBlockingQueue with a fixed capacity. The logic roughly in the DataStorage#linkBlocksHelper method would be the producer that adds LinkArgs objects to the queue. The logic in the linkWorkers ExecutorService would simply do what it does now, except it would pull LinkArgs objects out of the queue. Vinayakumar B Do you want to take a crack at this? If you don't have time in the next day or so let me know and I will take a look.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 8m 22s trunk passed
          +1 compile 0m 43s trunk passed with JDK v1.8.0_66
          +1 compile 0m 44s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 1m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 55s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 53s the patch passed
          +1 compile 0m 44s the patch passed with JDK v1.8.0_66
          +1 javac 0m 44s the patch passed
          +1 compile 0m 48s the patch passed with JDK v1.7.0_85
          +1 javac 0m 48s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 51s the patch passed with JDK v1.7.0_85
          -1 unit 59m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 58m 44s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 30s Patch generated 56 ASF License warnings.
          146m 41s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775818/HDFS-8578-12.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ad6e5a28e9ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0ac8fb4
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13771/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13771/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13771/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 22s trunk passed +1 compile 0m 43s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 53s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_66 +1 javac 0m 44s the patch passed +1 compile 0m 48s the patch passed with JDK v1.7.0_85 +1 javac 0m 48s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_85 -1 unit 59m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 58m 44s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 30s Patch generated 56 ASF License warnings. 146m 41s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775818/HDFS-8578-12.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ad6e5a28e9ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0ac8fb4 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13771/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13771/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13771/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          One big spot where I see the memory footprint potentially increasing is in DataStorage#linkBlocksHelper

          I think you are right. Each list contains LinkArgs entry for all block files and meta files inside each volume. this will be precreated even before starting any hardlink workers for each volume. Each LinkArgs have 2 File objects.
          In the above one of the test results, for 1.326278 blocks, There will be 1.326278*2 links required.
          So total 1.326278(blocks)*2(blockfile+metafile) LinkArgs instances will be created at same time.

          Show
          vinayrpet Vinayakumar B added a comment - One big spot where I see the memory footprint potentially increasing is in DataStorage#linkBlocksHelper I think you are right. Each list contains LinkArgs entry for all block files and meta files inside each volume. this will be precreated even before starting any hardlink workers for each volume. Each LinkArgs have 2 File objects. In the above one of the test results, for 1.326278 blocks, There will be 1.326278*2 links required. So total 1.326278(blocks)*2(blockfile+metafile) LinkArgs instances will be created at same time.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Vinayakumar B Woops, sorry I missed the part about no parallelism across namespaces. So 12 of these lists in memory.

          Show
          ctrezzo Chris Trezzo added a comment - Vinayakumar B Woops, sorry I missed the part about no parallelism across namespaces. So 12 of these lists in memory.
          Hide
          ctrezzo Chris Trezzo added a comment -

          One big spot where I see the memory footprint potentially increasing is in DataStorage#linkBlocksHelper. We build a list of LinkArgs called idBasedLayoutSingleLinks. This has 1 LinkArgs object per hard-link that we will need to create. Now with the parallelism we will have potentially 12 * 3 = 36 of these lists in memory at the same time. In other words, 1 LinkArgs object in memory for every hard-link we have to create on the data node.

          Show
          ctrezzo Chris Trezzo added a comment - One big spot where I see the memory footprint potentially increasing is in DataStorage#linkBlocksHelper . We build a list of LinkArgs called idBasedLayoutSingleLinks . This has 1 LinkArgs object per hard-link that we will need to create. Now with the parallelism we will have potentially 12 * 3 = 36 of these lists in memory at the same time. In other words, 1 LinkArgs object in memory for every hard-link we have to create on the data node.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Hard-linking is also parallelized within each of these threads (default is 12 threads). So the maximum number of threads you would potentially see is 12 disks (really it is storage directories, but let's assume there are 1 storage dir per disk) * 3 namespaces * 12 (default but configurable) hard-link worker threads = 432 threads.

          I missed hard-link workers in my earlier comment. So effective threads would be 12*12=144. As said no parallelism across namespace.

          Show
          vinayrpet Vinayakumar B added a comment - Hard-linking is also parallelized within each of these threads (default is 12 threads). So the maximum number of threads you would potentially see is 12 disks (really it is storage directories, but let's assume there are 1 storage dir per disk) * 3 namespaces * 12 (default but configurable) hard-link worker threads = 432 threads. I missed hard-link workers in my earlier comment. So effective threads would be 12*12=144. As said no parallelism across namespace.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Joep Rottinghuis

          With 12 disks and 3 namespaces that would mean 36 parallel threads right?

          Hard-linking is also parallelized within each of these threads (default is 12 threads). So the maximum number of threads you would potentially see is 12 disks (really it is storage directories, but let's assume there are 1 storage dir per disk) * 3 namespaces * 12 (default but configurable) hard-link worker threads = 432 threads.

          I had seen OOM errors with 2.6 release when it processed 6 disks in parallel.

          Raju Bairishetti Vinayakumar B Do you have a better sense of how much the memory footprint of the data node increases due to this parallelism?

          The only other case that I see where we do a full scan of the storage directories for hard-linking is during the DataStorage#prepareVolume code path. This has already been parallelized in the DataNode#refreshVolumes method. The number of parallel threads in this case is 1 per changed storage location * 12 (default but configurable) hard-link worker threads.

          I am attempting to revert one of our test clusters back to a 256x256 layout and will test this patch out on this cluster. Without parallelism datanodes on this cluster were each taking 1.5hrs to upgrade.

          Show
          ctrezzo Chris Trezzo added a comment - Joep Rottinghuis With 12 disks and 3 namespaces that would mean 36 parallel threads right? Hard-linking is also parallelized within each of these threads (default is 12 threads). So the maximum number of threads you would potentially see is 12 disks (really it is storage directories, but let's assume there are 1 storage dir per disk) * 3 namespaces * 12 (default but configurable) hard-link worker threads = 432 threads. I had seen OOM errors with 2.6 release when it processed 6 disks in parallel. Raju Bairishetti Vinayakumar B Do you have a better sense of how much the memory footprint of the data node increases due to this parallelism? The only other case that I see where we do a full scan of the storage directories for hard-linking is during the DataStorage#prepareVolume code path. This has already been parallelized in the DataNode#refreshVolumes method. The number of parallel threads in this case is 1 per changed storage location * 12 (default but configurable) hard-link worker threads. I am attempting to revert one of our test clusters back to a 256x256 layout and will test this patch out on this cluster. Without parallelism datanodes on this cluster were each taking 1.5hrs to upgrade.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Patch with re-interrupt.
          Please review.

          Show
          vinayrpet Vinayakumar B added a comment - Patch with re-interrupt. Please review.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Joep Rottinghuis for review.

          With 12 disks and 3 namespaces that would mean 36 parallel threads right?

          Parallelism is within blockpool/namespace. Not all blockpools will be in parallel. Datanode#initStorage() have the synchronization.

                synchronized (this) {
                  storage.recoverTransitionRead(this, nsInfo, dataDirs, startOpt);
                }

          Perhaps it would be better to make this configurable. It can default to 0, meaning as parallel as possible,

          Earlier versions patch of patch had configurable value. But it was changed to # of storage dirs on suggestion of Haohui Mai.

          I can imagine that 12 parallel threads (1 per disk in the above example) might turn out to be a reasonable compromise for some use cases.

          Yes, this is what happens.

          Wrt. catching InterruptedException, even though the newer patches do distinguish between ExecutionException | CancellationException and others, it is still good form to leave the interrupt status intact by re-interupting the current thread before returning:
          Or in this case perhaps yield and break out of the for loop, because the interrupt indicated that the thread should clean up and wrap up asap.

          Yes, Will upload updated patch to re-interrupt thread.
          Its would be good to wait till all the threads finish instead of leaving fs in bad state.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Joep Rottinghuis for review. With 12 disks and 3 namespaces that would mean 36 parallel threads right? Parallelism is within blockpool/namespace. Not all blockpools will be in parallel. Datanode#initStorage() have the synchronization. synchronized ( this ) { storage.recoverTransitionRead( this , nsInfo, dataDirs, startOpt); } Perhaps it would be better to make this configurable. It can default to 0, meaning as parallel as possible, Earlier versions patch of patch had configurable value. But it was changed to # of storage dirs on suggestion of Haohui Mai . I can imagine that 12 parallel threads (1 per disk in the above example) might turn out to be a reasonable compromise for some use cases. Yes, this is what happens. Wrt. catching InterruptedException, even though the newer patches do distinguish between ExecutionException | CancellationException and others, it is still good form to leave the interrupt status intact by re-interupting the current thread before returning: Or in this case perhaps yield and break out of the for loop, because the interrupt indicated that the thread should clean up and wrap up asap. Yes, Will upload updated patch to re-interrupt thread. Its would be good to wait till all the threads finish instead of leaving fs in bad state.
          Hide
          jrottinghuis Joep Rottinghuis added a comment -

          Or in this case perhaps yield and break out of the for loop, because the interrupt indicated that the thread should clean up and wrap up asap.

          Show
          jrottinghuis Joep Rottinghuis added a comment - Or in this case perhaps yield and break out of the for loop, because the interrupt indicated that the thread should clean up and wrap up asap.
          Hide
          jrottinghuis Joep Rottinghuis added a comment -

          Wrt. catching InterruptedException, even though the newer patches do distinguish between ExecutionException | CancellationException and others, it is still good form to leave the interrupt status intact by re-interupting the current thread before returning:

          Thread.currentThread().interrupt();
          
          Show
          jrottinghuis Joep Rottinghuis added a comment - Wrt. catching InterruptedException, even though the newer patches do distinguish between ExecutionException | CancellationException and others, it is still good form to leave the interrupt status intact by re-interupting the current thread before returning: Thread .currentThread().interrupt();
          Hide
          jrottinghuis Joep Rottinghuis added a comment -

          If I read the patch correctly the parallelism will be the total number of storage directories:

          int numParallelThreads = dataDirs.size();
          

          With 12 disks and 3 namespaces that would mean 36 parallel threads right?

          Perhaps it would be better to make this configurable. It can default to 0, meaning as parallel as possible, or any other explicit value set (up to # storage directories, although the newFixedThreadPool with more threads than running would simply not run more in parallel anyway). That way cluster admins can choose to either dial this all the way up, hammer the disks and pagecache and get it over with as soon as possible, or perhaps tune it down a bit in case they choose to keep the NM up and executing tasks in the meantime.
          I can imagine that 12 parallel threads (1 per disk in the above example) might turn out to be a reasonable compromise for some use cases.

          Show
          jrottinghuis Joep Rottinghuis added a comment - If I read the patch correctly the parallelism will be the total number of storage directories: int numParallelThreads = dataDirs.size(); With 12 disks and 3 namespaces that would mean 36 parallel threads right? Perhaps it would be better to make this configurable. It can default to 0, meaning as parallel as possible, or any other explicit value set (up to # storage directories, although the newFixedThreadPool with more threads than running would simply not run more in parallel anyway). That way cluster admins can choose to either dial this all the way up, hammer the disks and pagecache and get it over with as soon as possible, or perhaps tune it down a bit in case they choose to keep the NM up and executing tasks in the meantime. I can imagine that 12 parallel threads (1 per disk in the above example) might turn out to be a reasonable compromise for some use cases.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 9m 5s trunk passed
          +1 compile 0m 53s trunk passed with JDK v1.8.0_66
          +1 compile 0m 49s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 17s trunk passed
          +1 javadoc 1m 20s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 12s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 0s the patch passed
          +1 compile 0m 54s the patch passed with JDK v1.8.0_66
          +1 javac 0m 54s the patch passed
          +1 compile 0m 51s the patch passed with JDK v1.7.0_85
          +1 javac 0m 51s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 28s the patch passed
          +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 11s the patch passed with JDK v1.7.0_85
          -1 unit 71m 26s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 69m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 24s Patch generated 58 ASF License warnings.
          173m 35s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.TestBlockScanner
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
            org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.net.TestNetworkTopology
            org.apache.hadoop.hdfs.TestHFlush
            org.apache.hadoop.fs.TestWebHdfsFileContextMainOperations
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775573/HDFS-8578-11.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 014c5b8b384e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 62e9348
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13753/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13753/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 9m 5s trunk passed +1 compile 0m 53s trunk passed with JDK v1.8.0_66 +1 compile 0m 49s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 17s trunk passed +1 javadoc 1m 20s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 12s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 0s the patch passed +1 compile 0m 54s the patch passed with JDK v1.8.0_66 +1 javac 0m 54s the patch passed +1 compile 0m 51s the patch passed with JDK v1.7.0_85 +1 javac 0m 51s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 28s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 11s the patch passed with JDK v1.7.0_85 -1 unit 71m 26s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 69m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 24s Patch generated 58 ASF License warnings. 173m 35s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.net.TestNetworkTopology   org.apache.hadoop.hdfs.TestHFlush   org.apache.hadoop.fs.TestWebHdfsFileContextMainOperations JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775573/HDFS-8578-11.patch JIRA Issue HDFS-8578 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 014c5b8b384e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 62e9348 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13753/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13753/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13753/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          could we assign these values earlier at the time you load properties from all the StorageDirectories for getting datanodeUuid? This could make things more clear.

          All properties are loaded while reading from VERSION file.

          // Load the VERSION file to know the datanodeUUID
           readProperties(sd);

          Use multiple threads to write a shared field without locking is not a good choice although we do not have any problem right now. It implies that the method is thread safe so later people may add some dangerous code in the method...

          Thats a good suggestion, thanks Duo Zhang. Made DataStorage#setFieldsFromProperties(..) synchronized.

          Show
          vinayrpet Vinayakumar B added a comment - could we assign these values earlier at the time you load properties from all the StorageDirectories for getting datanodeUuid? This could make things more clear. All properties are loaded while reading from VERSION file. // Load the VERSION file to know the datanodeUUID readProperties(sd); Use multiple threads to write a shared field without locking is not a good choice although we do not have any problem right now. It implies that the method is thread safe so later people may add some dangerous code in the method... Thats a good suggestion, thanks Duo Zhang . Made DataStorage#setFieldsFromProperties(..) synchronized.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attached updated patch

          Show
          vinayrpet Vinayakumar B added a comment - Attached updated patch
          Hide
          Apache9 Duo Zhang added a comment -

          Could we assign these values earlier at the time you load properties from all the StorageDirectories for getting datanodeUuid? This could make things more clear.
          Use multiple threads to write a shared field without locking is not a good choice although we do not have any problem right now. It implies that the method is thread safe so later people may add some dangerous code in the method...

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Could we assign these values earlier at the time you load properties from all the StorageDirectories for getting datanodeUuid? This could make things more clear. Use multiple threads to write a shared field without locking is not a good choice although we do not have any problem right now. It implies that the method is thread safe so later people may add some dangerous code in the method... Thanks.
          Hide
          vinayrpet Vinayakumar B added a comment -

          I skimmed patch v10, seems you do not modify the format method. So how do you deal with the concurrent modification to layoutVersion and other properties? And layoutVersion could be changed in other place. And also, seems other properties are always assigned with same values, so could we move this to another place that only execute once? The code is a little confusing right now...

          There will be only one DataStorage instance per DN. And Values assigned in DataStorage.format() will be same for all directories, because these are not read from disk. layoutVersion is from the code, and other values from NamespaceInfo from namenode. Out of DataStorage's properties in VERSION file below, except storageID(which is not a field of DataStorage), others will be same for all directories after format/upgrade. Below are the properties from Datastorage's VERSION file.

          storageID=DS-8a5170b7-a105-45cd-b9b2-9d01c160e11f
          clusterID=testClusterID
          cTime=0
          datanodeUuid=35e3d456-c507-47c8-aaf9-54e77ce49ce0
          storageType=DATA_NODE
          layoutVersion=-56
          

          So I dont think handling synchronization for properties of DataStorage is required.
          Only thing is, have to read datanodeUuid, which will be used later, before going in parallel.

          Hope I have cleared the confusion?

          -Thanks

          Show
          vinayrpet Vinayakumar B added a comment - I skimmed patch v10, seems you do not modify the format method. So how do you deal with the concurrent modification to layoutVersion and other properties? And layoutVersion could be changed in other place. And also, seems other properties are always assigned with same values, so could we move this to another place that only execute once? The code is a little confusing right now... There will be only one DataStorage instance per DN. And Values assigned in DataStorage.format() will be same for all directories, because these are not read from disk. layoutVersion is from the code, and other values from NamespaceInfo from namenode. Out of DataStorage's properties in VERSION file below, except storageID (which is not a field of DataStorage), others will be same for all directories after format/upgrade. Below are the properties from Datastorage's VERSION file. storageID=DS-8a5170b7-a105-45cd-b9b2-9d01c160e11f clusterID=testClusterID cTime=0 datanodeUuid=35e3d456-c507-47c8-aaf9-54e77ce49ce0 storageType=DATA_NODE layoutVersion=-56 So I dont think handling synchronization for properties of DataStorage is required. Only thing is, have to read datanodeUuid, which will be used later, before going in parallel. Hope I have cleared the confusion? -Thanks
          Hide
          Apache9 Duo Zhang added a comment -

          Oh, great, there is already an inprogress patch.

          I skimmed patch v10, seems you do not modify the format method. So how do you deal with the concurrent modification to layoutVersion and other properties? And layoutVersion could be changed in other place. And also, seems other properties are always assigned with same values, so could we move this to another place that only execute once? The code is a little confusing right now...

            private void format(StorageDirectory sd, NamespaceInfo nsInfo,
                        String datanodeUuid) throws IOException {
              sd.clearDirectory(); // create directory
              this.layoutVersion = HdfsServerConstants.DATANODE_LAYOUT_VERSION;
              this.clusterID = nsInfo.getClusterID();
              this.namespaceID = nsInfo.getNamespaceID();
              this.cTime = 0;
              this.datanodeUuid = datanodeUuid;
          
              if (sd.getStorageUuid() == null) {
                // Assign a new Storage UUID.
                sd.setStorageUuid(DatanodeStorage.generateUuid());
              }
          
              writeProperties(sd);
            }
          

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Oh, great, there is already an inprogress patch. I skimmed patch v10, seems you do not modify the format method. So how do you deal with the concurrent modification to layoutVersion and other properties? And layoutVersion could be changed in other place. And also, seems other properties are always assigned with same values, so could we move this to another place that only execute once? The code is a little confusing right now... private void format(StorageDirectory sd, NamespaceInfo nsInfo, String datanodeUuid) throws IOException { sd.clearDirectory(); // create directory this .layoutVersion = HdfsServerConstants.DATANODE_LAYOUT_VERSION; this .clusterID = nsInfo.getClusterID(); this .namespaceID = nsInfo.getNamespaceID(); this .cTime = 0; this .datanodeUuid = datanodeUuid; if (sd.getStorageUuid() == null ) { // Assign a new Storage UUID. sd.setStorageUuid(DatanodeStorage.generateUuid()); } writeProperties(sd); } Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 5s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 3m 15s trunk passed
          +1 compile 0m 33s trunk passed with JDK v1.8.0_60
          +1 compile 0m 31s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 15s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          -1 findbugs 1m 53s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 53s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_60
          +1 javac 0m 37s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_79
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 25s the patch passed
          +1 javadoc 1m 16s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 8s the patch passed with JDK v1.7.0_79
          -1 unit 65m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 55m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 25s Patch generated 58 ASF License warnings.
          142m 33s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestAbandonBlock
            hadoop.hdfs.server.datanode.TestBlockScanner
          JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestPersistBlocks
          JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-02
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12764964/HDFS-8578-10.patch
          JIRA Issue HDFS-8578
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 5f2ff10c3a54 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / 6e4f8a4
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13323/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 228MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13323/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 3m 15s trunk passed +1 compile 0m 33s trunk passed with JDK v1.8.0_60 +1 compile 0m 31s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 15s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 1m 53s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 6s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 53s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 45s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_60 +1 javac 0m 37s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_79 +1 javac 0m 38s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 25s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 8s the patch passed with JDK v1.7.0_79 -1 unit 65m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 55m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 25s Patch generated 58 ASF License warnings. 142m 33s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestAbandonBlock   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestPersistBlocks JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-02 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12764964/HDFS-8578-10.patch JIRA Issue HDFS-8578 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 5f2ff10c3a54 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / 6e4f8a4 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13323/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13323/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 228MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13323/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 19s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 49s There were no new javac warning messages.
          +1 javadoc 10m 10s There were no new javadoc warning messages.
          -1 release audit 0m 16s The applied patch generated 1 release audit warnings.
          +1 checkstyle 1m 23s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 8s Pre-build of native portion
          -1 hdfs tests 203m 5s Tests failed in hadoop-hdfs.
              248m 40s  



          Reason Tests
          Failed unit tests hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.web.TestWebHDFSOAuth2
            hadoop.hdfs.server.datanode.TestRefreshNamenodes
            hadoop.hdfs.web.TestWebHdfsContentLength
          Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
            org.apache.hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold
            org.apache.hadoop.hdfs.TestMiniDFSCluster



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764964/HDFS-8578-10.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 30e2f83
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12788/artifact/patchprocess/patchReleaseAuditProblems.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12788/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12788/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12788/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 49s There were no new javac warning messages. +1 javadoc 10m 10s There were no new javadoc warning messages. -1 release audit 0m 16s The applied patch generated 1 release audit warnings. +1 checkstyle 1m 23s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 8s Pre-build of native portion -1 hdfs tests 203m 5s Tests failed in hadoop-hdfs.     248m 40s   Reason Tests Failed unit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.web.TestWebHDFSOAuth2   hadoop.hdfs.server.datanode.TestRefreshNamenodes   hadoop.hdfs.web.TestWebHdfsContentLength Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool   org.apache.hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold   org.apache.hadoop.hdfs.TestMiniDFSCluster Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764964/HDFS-8578-10.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 30e2f83 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12788/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12788/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12788/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12788/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Updated the patch
          1. Removed the new Configuration for num of parallel threads, using no of disks.
          2. fixed TestDataNodeHotSwapVolumes#testReplicatingAfterRemoveVolume() failure. Though its not related to patch, it was failing frequently in jenkins runs for this patch.

          Show
          vinayrpet Vinayakumar B added a comment - Updated the patch 1. Removed the new Configuration for num of parallel threads, using no of disks. 2. fixed TestDataNodeHotSwapVolumes#testReplicatingAfterRemoveVolume() failure. Though its not related to patch, it was failing frequently in jenkins runs for this patch.
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Haohui Mai sure, I will give a try. I had seen OOM errors with 2.6 release when it processed 6 disks in parallel.

          Show
          raju.bairishetti Raju Bairishetti added a comment - Haohui Mai sure, I will give a try. I had seen OOM errors with 2.6 release when it processed 6 disks in parallel.
          Hide
          wheat9 Haohui Mai added a comment -

          Datanode may require more memory to process all volumes/disks in parallel.

          That doesn't sound right to me. Can you identify why it requires more memory if the tasks of each threads are to create hardlinks for blocks? Note that in the current implementation the lists of blocks have been pre-calculated.

          Show
          wheat9 Haohui Mai added a comment - Datanode may require more memory to process all volumes/disks in parallel. That doesn't sound right to me. Can you identify why it requires more memory if the tasks of each threads are to create hardlinks for blocks? Note that in the current implementation the lists of blocks have been pre-calculated.
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Haohui Mai Thanks for reviewing and comments.
          Haohui Mai Datanode may *require more memory* to process all volumes/disks in parallel. I feel, It would be nice to have a configuration in which we can tell how many disks a datanode can process in parallel. If we configure the value to 1 then it falls back to sequential mode(i.e. processing one disk at a time). We can control datanode heap memory by processing few number of disks in parallel.

          At INMOBI, We have deployed HDFS-8578-branch-2.6.0.patch in production. We had seen improvements in datanode upgrade time. We have tuned the memory &gc settings to avoid OOM errors.

          Show
          raju.bairishetti Raju Bairishetti added a comment - Haohui Mai Thanks for reviewing and comments. Haohui Mai Datanode may * require more memory * to process all volumes/disks in parallel. I feel, It would be nice to have a configuration in which we can tell how many disks a datanode can process in parallel. If we configure the value to 1 then it falls back to sequential mode(i.e. processing one disk at a time) . We can control datanode heap memory by processing few number of disks in parallel. At INMOBI, We have deployed HDFS-8578-branch-2.6.0.patch in production. We had seen improvements in datanode upgrade time. We have tuned the memory &gc settings to avoid OOM errors.
          Hide
          vinayrpet Vinayakumar B added a comment -

          thanks Haohui Mai
          I will update accordingly

          Show
          vinayrpet Vinayakumar B added a comment - thanks Haohui Mai I will update accordingly
          Hide
          wheat9 Haohui Mai added a comment -

          We've seen in production that the upgrades take ~3hours for 24 disks.

          I think it makes sense to simply run a thread for each disk instead of having a new configuration, unless it is possible to parallelize the hard links for individual disks.

          Show
          wheat9 Haohui Mai added a comment - We've seen in production that the upgrades take ~3hours for 24 disks. I think it makes sense to simply run a thread for each disk instead of having a new configuration, unless it is possible to parallelize the hard links for individual disks.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 28s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 8m 3s There were no new javac warning messages.
          +1 javadoc 10m 5s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 32s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 29s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 191m 15s Tests failed in hadoop-hdfs.
              234m 46s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764013/HDFS-8578-09.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 66dad85
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12708/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12708/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12708/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12708/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 28s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 3s There were no new javac warning messages. +1 javadoc 10m 5s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 32s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 29s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 191m 15s Tests failed in hadoop-hdfs.     234m 46s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764013/HDFS-8578-09.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 66dad85 Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12708/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12708/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12708/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12708/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Rebased the patch.

          All above tests are passing locally.

          Show
          vinayrpet Vinayakumar B added a comment - Rebased the patch. All above tests are passing locally.
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Vinayakumar B Seeing check style issue and test failures. Looks good to me otherwise. +1 (non-binding)

          Show
          raju.bairishetti Raju Bairishetti added a comment - Vinayakumar B Seeing check style issue and test failures. Looks good to me otherwise. +1 (non-binding)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 20s The applied patch generated 1 new checkstyle issues (total was 595, now 590).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 1s Pre-build of native portion
          -1 hdfs tests 161m 15s Tests failed in hadoop-hdfs.
              204m 51s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747740/HDFS-8578-08.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 2a1d656
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11865/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11865/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11865/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11865/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 20s The applied patch generated 1 new checkstyle issues (total was 595, now 590). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 1s Pre-build of native portion -1 hdfs tests 161m 15s Tests failed in hadoop-hdfs.     204m 51s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747740/HDFS-8578-08.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 2a1d656 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11865/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11865/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11865/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11865/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attaching a patch with one more additional change.
          finding datanodeUUID by checking all storages, before processing in parallel.
          This is required for formatting new dir added and datanode restarted.

          Show
          vinayrpet Vinayakumar B added a comment - Attaching a patch with one more additional change. finding datanodeUUID by checking all storages, before processing in parallel. This is required for formatting new dir added and datanode restarted.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 21s The applied patch generated 1 new checkstyle issues (total was 597, now 592).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 3s Pre-build of native portion
          -1 hdfs tests 161m 1s Tests failed in hadoop-hdfs.
              204m 22s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12745242/HDFS-8578-07.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4084eaf
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11699/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11699/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11699/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11699/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 21s The applied patch generated 1 new checkstyle issues (total was 597, now 592). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 3s Pre-build of native portion -1 hdfs tests 161m 1s Tests failed in hadoop-hdfs.     204m 22s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745242/HDFS-8578-07.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4084eaf checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11699/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11699/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11699/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11699/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Fixed TestReplication which fails intermittently, not exactly related to this Jira.

          TestHDFSCLI failed due to collision with hadoop-common precommit job.
          TestDataNodeRollingUpgrade passes locally.

          Show
          vinayrpet Vinayakumar B added a comment - Fixed TestReplication which fails intermittently, not exactly related to this Jira. TestHDFSCLI failed due to collision with hadoop-common precommit job. TestDataNodeRollingUpgrade passes locally.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 1s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 55s There were no new javac warning messages.
          +1 javadoc 10m 0s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 33s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 37s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 7s Pre-build of native portion
          -1 hdfs tests 158m 41s Tests failed in hadoop-hdfs.
              201m 29s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestReplication
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12745204/HDFS-8578-06.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a431ed9
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11695/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11695/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11695/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 1s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 0s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 33s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 37s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 7s Pre-build of native portion -1 hdfs tests 158m 41s Tests failed in hadoop-hdfs.     201m 29s   Reason Tests Failed unit tests hadoop.hdfs.TestReplication   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745204/HDFS-8578-06.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a431ed9 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11695/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11695/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11695/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attached patch with some more test fixes

          Show
          vinayrpet Vinayakumar B added a comment - Attached patch with some more test fixes
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 6s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 34s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 4s Pre-build of native portion
          -1 hdfs tests 160m 18s Tests failed in hadoop-hdfs.
              201m 13s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.datanode.TestDataStorage



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12745014/HDFS-8578-05.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 19295b3
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11682/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11682/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11682/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 6s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 34s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 4s Pre-build of native portion -1 hdfs tests 160m 18s Tests failed in hadoop-hdfs.     201m 13s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestDataStorage Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745014/HDFS-8578-05.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 19295b3 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11682/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11682/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11682/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment - - edited

          Attached the patch with checkstyle,whitespace and test failure fixes

          Show
          vinayrpet Vinayakumar B added a comment - - edited Attached the patch with checkstyle,whitespace and test failure fixes
          Hide
          vinayrpet Vinayakumar B added a comment -

          Fixed checkstyle and whitespace warnings.
          Please review.

          Show
          vinayrpet Vinayakumar B added a comment - Fixed checkstyle and whitespace warnings. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 21m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 8m 46s There were no new javac warning messages.
          +1 javadoc 9m 53s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 30s The applied patch generated 2 new checkstyle issues (total was 591, now 587).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 21s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 5s Pre-build of native portion
          -1 hdfs tests 93m 3s Tests failed in hadoop-hdfs.
              142m 16s  



          Reason Tests
          Timed out tests org.apache.hadoop.tools.TestJMXGet
            org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
            org.apache.hadoop.fs.viewfs.TestViewFsWithXAttrs
            org.apache.hadoop.fs.viewfs.TestViewFsHdfs
            org.apache.hadoop.fs.viewfs.TestViewFsDefaultValue
            org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744707/HDFS-8578-04.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b489080
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11662/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11662/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11662/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11662/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11662/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 46s There were no new javac warning messages. +1 javadoc 9m 53s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 30s The applied patch generated 2 new checkstyle issues (total was 591, now 587). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 5s Pre-build of native portion -1 hdfs tests 93m 3s Tests failed in hadoop-hdfs.     142m 16s   Reason Tests Timed out tests org.apache.hadoop.tools.TestJMXGet   org.apache.hadoop.fs.loadGenerator.TestLoadGenerator   org.apache.hadoop.fs.viewfs.TestViewFsWithXAttrs   org.apache.hadoop.fs.viewfs.TestViewFsHdfs   org.apache.hadoop.fs.viewfs.TestViewFsDefaultValue   org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744707/HDFS-8578-04.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b489080 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11662/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11662/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11662/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11662/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11662/console This message was automatically generated.
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          +1(non-binding) for the latest patch.

          Show
          raju.bairishetti Raju Bairishetti added a comment - +1(non-binding) for the latest patch.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attached the patch, for using the max dataDirs size threads and Min 1.

          Show
          vinayrpet Vinayakumar B added a comment - Attached the patch, for using the max dataDirs size threads and Min 1.
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Amareshwari Sriramadasu yes, we should go with the number of total volumes if provided value is more. Can we use -1 value to start the thread pool with number of volumes in the datanode?

          Show
          raju.bairishetti Raju Bairishetti added a comment - Amareshwari Sriramadasu yes, we should go with the number of total volumes if provided value is more. Can we use -1 value to start the thread pool with number of volumes in the datanode?
          Hide
          amareshwari Amareshwari Sriramadasu added a comment -

          Go with number of total volumes if provided value is more?

          Show
          amareshwari Amareshwari Sriramadasu added a comment - Go with number of total volumes if provided value is more?
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Thanks Vinayakumar B for the quick response and with the patch.

          Can we shutdown the ExecutorService once the work is done?

          Vinayakumar B Can we handle this case if you feel it is correct?

          One more minor ask: Do we need to validate the config value which is provided by user? Like the config value is valid or not.

          int numParallelThreads = datanode.getConf().getInt(
          +        DFSConfigKeys.DFS_DATANODE_PARALLEL_VOLUME_LOAD_THREADS_NUM_KEY,
          +        dataDirs.size());
          

          I feel we should not honor the config value if it is more than number of volumes on the datanode. Can we calculate the num of threads by checking the minimum number between user provided value and number of total volumes (i.e. min(userProvidedValue, dataDirs.size()))? Usually users keep single config value across all datanodes but number of disks on datanodes can be vary. In such cases we may end up spawning more threads than necessary.

          Show
          raju.bairishetti Raju Bairishetti added a comment - Thanks Vinayakumar B for the quick response and with the patch. Can we shutdown the ExecutorService once the work is done? Vinayakumar B Can we handle this case if you feel it is correct? One more minor ask: Do we need to validate the config value which is provided by user? Like the config value is valid or not. int numParallelThreads = datanode.getConf().getInt( + DFSConfigKeys.DFS_DATANODE_PARALLEL_VOLUME_LOAD_THREADS_NUM_KEY, + dataDirs.size()); I feel we should not honor the config value if it is more than number of volumes on the datanode. Can we calculate the num of threads by checking the minimum number between user provided value and number of total volumes (i.e. min(userProvidedValue, dataDirs.size()))? Usually users keep single config value across all datanodes but number of disks on datanodes can be vary. In such cases we may end up spawning more threads than necessary.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Updated the patch with configuration

          Please review

          Show
          vinayrpet Vinayakumar B added a comment - Updated the patch with configuration Please review
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Raju Bairishetti for posting the metrics.
          Good to see the improvements.
          Will post a updated patch soon with the configuration for the num of parallel threads

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Raju Bairishetti for posting the metrics. Good to see the improvements. Will post a updated patch soon with the configuration for the num of parallel threads
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Sequential processing of volumes in prod clusters
          |Cluster || number of blocks/links|| Upgrade time || Disk health||
          |testCluster1 |296472 |20 mins | good |
          ....

          Posted these metrics as we are seeing huge difference in processing times of volumes across clusters. These metrics are not related to patch as the above benchmarking was done without patch.

          performance metrics with the patch:
          ||Node ||Number of blocks or links ||Single volume time total time for processing all volumes ||Processing mechanism||
          |node1 |1.326278 million |75 secs |75 secs | parallel |
          |node2 |1.325818 million |60 secs |342 secs |sequential

          Datanode is expecting higher heap memory(Xmx) to process volumes in parallel.

          Vinayakumar B Can we introduce a new config which will tell how many number of volumes can be processed in parallel (i.e. thread pool size)? If the config value is one then it processes all the volumes in sequential.

          Show
          raju.bairishetti Raju Bairishetti added a comment - Sequential processing of volumes in prod clusters |Cluster || number of blocks/links|| Upgrade time || Disk health|| |testCluster1 |296472 |20 mins | good | .... Posted these metrics as we are seeing huge difference in processing times of volumes across clusters. These metrics are not related to patch as the above benchmarking was done without patch . performance metrics with the patch: ||Node ||Number of blocks or links ||Single volume time total time for processing all volumes ||Processing mechanism|| |node1 |1.326278 million |75 secs |75 secs | parallel | |node2 |1.325818 million |60 secs |342 secs |sequential Datanode is expecting higher heap memory(Xmx) to process volumes in parallel. Vinayakumar B Can we introduce a new config which will tell how many number of volumes can be processed in parallel (i.e. thread pool size)? If the config value is one then it processes all the volumes in sequential.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744407/HDFS-8578-branch-2.6.0.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6b7692c
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11641/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744407/HDFS-8578-branch-2.6.0.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6b7692c Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11641/console This message was automatically generated.
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Attaching a patch which works with 2.6.0 branch and adding results.

          We are seeing a a huge diff in upgrade times of datanode on various clusters.

          Sequential processing of volumes in prod clusters

          Cluster number of blocks/links Upgrade time Disk health
          testCluster1 296472 20 mins good
          testCluster1 295768 1hr 4 mins bad
          testCluster2 70793 55 secs good
          testCluster2 388245 8 mins good
          devCluster 160606 2 secs good
          stagingCluster1 1.325818 million 60 secs good

          Note: Assuming disk is good if we don't see any errors in dmesg output.

          performance metrics with the patch:

          Node Number of blocks or links Single volume time total time for processing all volumes Processing mechanism
          node1 1.326278 million 75 secs 75 secs parallel
          node2 1.325818 million 60 secs 342 secs sequential
          Show
          raju.bairishetti Raju Bairishetti added a comment - Attaching a patch which works with 2.6.0 branch and adding results. We are seeing a a huge diff in upgrade times of datanode on various clusters. Sequential processing of volumes in prod clusters Cluster number of blocks/links Upgrade time Disk health testCluster1 296472 20 mins good testCluster1 295768 1hr 4 mins bad testCluster2 70793 55 secs good testCluster2 388245 8 mins good devCluster 160606 2 secs good stagingCluster1 1.325818 million 60 secs good Note: Assuming disk is good if we don't see any errors in dmesg output. performance metrics with the patch: Node Number of blocks or links Single volume time total time for processing all volumes Processing mechanism node1 1.326278 million 75 secs 75 secs parallel node2 1.325818 million 60 secs 342 secs sequential
          Hide
          vinayrpet Vinayakumar B added a comment -

          I think overall functionalitywise, both does same. So, performance results against earlier patch also fine IMO.

          Show
          vinayrpet Vinayakumar B added a comment - I think overall functionalitywise, both does same. So, performance results against earlier patch also fine IMO.
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          We are using the 2.6 branch in our cluster. I am benchmarking against the 2.6 branch with a different patch which has changes in BlockPoolSliceStorage class. I don't think those performance metrics will be applicable for this patch.

          Show
          raju.bairishetti Raju Bairishetti added a comment - We are using the 2.6 branch in our cluster. I am benchmarking against the 2.6 branch with a different patch which has changes in BlockPoolSliceStorage class. I don't think those performance metrics will be applicable for this patch.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Raju Bairishetti, would you mind testing with this patch with your loads?

          Show
          vinayrpet Vinayakumar B added a comment - Raju Bairishetti , would you mind testing with this patch with your loads?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Vinayakumar B Have you done any performance benchmarking with this approach? If yes, Could you please post the results here?

          Nope. I dont have any results.

          Show
          vinayrpet Vinayakumar B added a comment - Vinayakumar B Have you done any performance benchmarking with this approach? If yes, Could you please post the results here? Nope. I dont have any results.
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Vinayakumar B Have you done any performance benchmarking with this approach? If yes, Could you please post the results here?

          Show
          raju.bairishetti Raju Bairishetti added a comment - Vinayakumar B Have you done any performance benchmarking with this approach? If yes, Could you please post the results here?
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Some minor comments:

          Typo mistake in the logs: exception instead of exeption
          LOG.error("Unexpected exeption while loading storage directories"
          LOG.error("InterruptedExeption while loading storage directories"

          Can we shutdown the ExecutorService once the work is done?

          Show
          raju.bairishetti Raju Bairishetti added a comment - Some minor comments: Typo mistake in the logs: exception instead of exeption LOG.error("Unexpected exeption while loading storage directories" LOG.error("InterruptedExeption while loading storage directories" Can we shutdown the ExecutorService once the work is done?
          Hide
          raju.bairishetti Raju Bairishetti added a comment -

          Vinayakumar B I Agree, Making changes in BlockPoolSliceStorage won't help us in improving the performance as HDFS-7035 makes the data dir/disk processing atomic.

          My bad, initially I was looking into the 2.6.0 branch. Changes in BlockPoolSliceStorage would have been helpful only 2.6.0 branch. Lots of refactoring happened as part of HDFS-7035. It was merged to 2.7.0 branch.

          Show
          raju.bairishetti Raju Bairishetti added a comment - Vinayakumar B I Agree, Making changes in BlockPoolSliceStorage won't help us in improving the performance as HDFS-7035 makes the data dir/disk processing atomic. My bad, initially I was looking into the 2.6.0 branch. Changes in BlockPoolSliceStorage would have been helpful only 2.6.0 branch. Lots of refactoring happened as part of HDFS-7035 . It was merged to 2.7.0 branch .
          Hide
          vinayrpet Vinayakumar B added a comment -

          OK, so your idea is that that will allow us to parallelize upgrades between different block pools. Fair enough.

          Not exactly. DataStorage#addStorageLocations(..) is called per blockpool only. And two different blockpools initialization is synchronized in Datanode.java itself.

          synchronized (this) {
                  storage.recoverTransitionRead(this, nsInfo, dataDirs, startOpt);
                }

          My change was because for the same blockpool, each directory is being loaded separately in . DataStorage#addStorageLocations(..).

          List<File> bpDataDirs = new ArrayList<File>();
                bpDataDirs.add(BlockPoolSliceStorage.getBpRoot(bpid, new File(root,
                        STORAGE_DIR_CURRENT)));
          ...
          ...
          bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs, startOpt);
          

          So in BlockPoolSliceStorage#loadBpStorageDirectories(... Collection<File> dataDirs, ...), dataDirs will have only one directory during startup. So adding parallelism there will not change anything.

          I don't think this is true. The DN will not start up if more than dfs.datanode.failed.volumes.tolerated volumes have failed, as per this code:

          Yes!! You are right. I just checked only DataStorage#addStorageLocations(..), had totally forgot about dfs.datanode.failed.volumes.tolerated.

          Show
          vinayrpet Vinayakumar B added a comment - OK, so your idea is that that will allow us to parallelize upgrades between different block pools. Fair enough. Not exactly. DataStorage#addStorageLocations(..) is called per blockpool only. And two different blockpools initialization is synchronized in Datanode.java itself. synchronized ( this ) { storage.recoverTransitionRead( this , nsInfo, dataDirs, startOpt); } My change was because for the same blockpool, each directory is being loaded separately in . DataStorage#addStorageLocations(..) . List<File> bpDataDirs = new ArrayList<File>(); bpDataDirs.add(BlockPoolSliceStorage.getBpRoot(bpid, new File(root, STORAGE_DIR_CURRENT))); ... ... bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs, startOpt); So in BlockPoolSliceStorage#loadBpStorageDirectories(... Collection<File> dataDirs, ...) , dataDirs will have only one directory during startup. So adding parallelism there will not change anything. I don't think this is true. The DN will not start up if more than dfs.datanode.failed.volumes.tolerated volumes have failed, as per this code: Yes!! You are right. I just checked only DataStorage#addStorageLocations(..) , had totally forgot about dfs.datanode.failed.volumes.tolerated .
          Hide
          cmccabe Colin P. McCabe added a comment -

          Just realized that, parallelism should be added at the datanode level in DataStorage#addStorageLocations(). Not in the BlockPoolSliceStorage.

          OK, so your idea is that that will allow us to parallelize upgrades between different block pools. Fair enough.

          And during startup, If any one of the blockpool storage directories failed to load/upgrade also, datanode continues to start with other directories available. Only if all directories are failed to load then only it will fail.

          I don't think this is true. The DN will not start up if more than dfs.datanode.failed.volumes.tolerated volumes have failed, as per this code:

            /**
             * An FSDataset has a directory where it loads its data files.
             */
            FsDatasetImpl(DataNode datanode, DataStorage storage, Configuration conf
                ) throws IOException {
          ...
              if (volsFailed > volFailuresTolerated) {
                throw new DiskErrorException("Too many failed volumes - "
                    + "current valid volumes: " + storage.getNumStorageDirs()
                    + ", volumes configured: " + volsConfigured
                    + ", volumes failed: " + volsFailed
                    + ", volume failures tolerated: " + volFailuresTolerated);
              }
          

          Updated the patch, please review.

          ok

          Show
          cmccabe Colin P. McCabe added a comment - Just realized that, parallelism should be added at the datanode level in DataStorage#addStorageLocations(). Not in the BlockPoolSliceStorage. OK, so your idea is that that will allow us to parallelize upgrades between different block pools. Fair enough. And during startup, If any one of the blockpool storage directories failed to load/upgrade also, datanode continues to start with other directories available. Only if all directories are failed to load then only it will fail. I don't think this is true. The DN will not start up if more than dfs.datanode.failed.volumes.tolerated volumes have failed, as per this code: /** * An FSDataset has a directory where it loads its data files. */ FsDatasetImpl(DataNode datanode, DataStorage storage, Configuration conf ) throws IOException { ... if (volsFailed > volFailuresTolerated) { throw new DiskErrorException( "Too many failed volumes - " + "current valid volumes: " + storage.getNumStorageDirs() + ", volumes configured: " + volsConfigured + ", volumes failed: " + volsFailed + ", volume failures tolerated: " + volFailuresTolerated); } Updated the patch, please review. ok
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          Any concerns about overloading the controller?

          In general, the upgrade workload is creating a bunch of hardlinks, often one per block file. This is not a large amount of I/O in terms of bandwidth. Normally it completes in a second or two. The only cases we have seen problems are where write caching is turned off on the hard disks, forcing a lot of non-sequential I/O to update the inode entries. I would also argue that it is Linux's responsibility to manage sending commands to the disk controller and backing off (putting the user mode process to sleep) if there are too many in flight. So I don't see any concerns here about overloading the disk controller.

          Show
          cmccabe Colin P. McCabe added a comment - -