Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10966

Enhance Dispatcher logic on deciding when to give up a source DataNode

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When a Dispatcher thread works on a source DataNode, in each iteration it tries to execute a PendingMove. If no block is moved after 5 iterations, this source (over-utlized) DataNode is given up for this Balancer iteration (20 mins). This is problematic if the source DataNode was heavily loaded in the beginning of the iteration. It will quickly encounter 5 unsuccessful moves and be abandoned.

      We should enhance this logic by e.g. using elapsed time instead of number of iterations.

      // Check if the previous move was successful
              } else {
                // source node cannot find a pending block to move, iteration +1
                noPendingMoveIteration++;
                // in case no blocks can be moved for source node's task,
                // jump out of while-loop after 5 iterations.
                if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
                  LOG.info("Failed to find a pending move "  + noPendingMoveIteration
                      + " times.  Skipping " + this);
                  resetScheduledSize();
                }
              }
      
      1. HDFS-10966.00.patch
        9 kB
        Zhe Zhang
      2. HDFS-10966.01.patch
        11 kB
        Zhe Zhang
      3. HDFS-10966-branch-2.7.00.patch
        12 kB
        Zhe Zhang

        Issue Links

          Activity

          Hide
          zhz Zhe Zhang added a comment -

          Thanks much Kihwal!

          Show
          zhz Zhe Zhang added a comment - Thanks much Kihwal!
          Hide
          kihwal Kihwal Lee added a comment -

          Committed to branch-2.7. Thanks for working on this Mark Wagner and finishing it up, Zhe Zhang.

          Show
          kihwal Kihwal Lee added a comment - Committed to branch-2.7. Thanks for working on this Mark Wagner and finishing it up, Zhe Zhang .
          Hide
          kihwal Kihwal Lee added a comment -

          The 2.7 patch looks good.

          Show
          kihwal Kihwal Lee added a comment - The 2.7 patch looks good.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 8s branch-2.7 passed
          +1 compile 1m 2s branch-2.7 passed with JDK v1.8.0_111
          +1 compile 1m 2s branch-2.7 passed with JDK v1.7.0_121
          +1 checkstyle 0m 34s branch-2.7 passed
          +1 mvnsite 1m 5s branch-2.7 passed
          +1 mvneclipse 0m 19s branch-2.7 passed
          +1 findbugs 2m 59s branch-2.7 passed
          +1 javadoc 1m 0s branch-2.7 passed with JDK v1.8.0_111
          +1 javadoc 1m 47s branch-2.7 passed with JDK v1.7.0_121
          +1 mvninstall 0m 55s the patch passed
          +1 compile 1m 3s the patch passed with JDK v1.8.0_111
          +1 javac 1m 3s the patch passed
          +1 compile 1m 0s the patch passed with JDK v1.7.0_121
          +1 javac 1m 0s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 597 unchanged - 3 fixed = 604 total (was 600)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 1862 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 44s The patch 94 line(s) with tabs.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 1m 1s the patch passed with JDK v1.8.0_111
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_121
          -1 unit 57m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          -1 asflicense 0m 19s The patch generated 3 ASF License warnings.
          162m 10s



          Reason Tests
          JDK v1.8.0_111 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestDistributedFileSystem
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-10966
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839871/HDFS-10966-branch-2.7.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 04de11e5de60 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 0da0bf0
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17618/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17618/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 8s branch-2.7 passed +1 compile 1m 2s branch-2.7 passed with JDK v1.8.0_111 +1 compile 1m 2s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 34s branch-2.7 passed +1 mvnsite 1m 5s branch-2.7 passed +1 mvneclipse 0m 19s branch-2.7 passed +1 findbugs 2m 59s branch-2.7 passed +1 javadoc 1m 0s branch-2.7 passed with JDK v1.8.0_111 +1 javadoc 1m 47s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 55s the patch passed +1 compile 1m 3s the patch passed with JDK v1.8.0_111 +1 javac 1m 3s the patch passed +1 compile 1m 0s the patch passed with JDK v1.7.0_121 +1 javac 1m 0s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 597 unchanged - 3 fixed = 604 total (was 600) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 1862 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 44s The patch 94 line(s) with tabs. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_111 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_121 -1 unit 57m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_121. -1 asflicense 0m 19s The patch generated 3 ASF License warnings. 162m 10s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDistributedFileSystem JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-10966 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839871/HDFS-10966-branch-2.7.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 04de11e5de60 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 0da0bf0 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17618/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17618/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17618/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Kihwal Lee for the final review! Attaching branch-2.7 patch.

          Show
          zhz Zhe Zhang added a comment - Thanks Kihwal Lee for the final review! Attaching branch-2.7 patch.
          Hide
          kihwal Kihwal Lee added a comment -

          I've committed this to trunk through branch-2.8. Please provide a 2.7 version of the patch.

          Show
          kihwal Kihwal Lee added a comment - I've committed this to trunk through branch-2.8. Please provide a 2.7 version of the patch.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10868 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10868/)
          HDFS-10966. Enhance Dispatcher logic on deciding when to give up a (kihwal: rev 49a09179e3fadae090126261be0a7fe0aa48798e)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10868 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10868/ ) HDFS-10966 . Enhance Dispatcher logic on deciding when to give up a (kihwal: rev 49a09179e3fadae090126261be0a7fe0aa48798e) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Hide
          kihwal Kihwal Lee added a comment -

          +1

          Show
          kihwal Kihwal Lee added a comment - +1
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 52s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 693 unchanged - 3 fixed = 699 total (was 696)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 36s the patch passed
          +1 unit 61m 6s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          79m 42s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10966
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839654/HDFS-10966.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 8ba1d89a341b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7584fbf
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17613/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17613/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17613/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 52s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 693 unchanged - 3 fixed = 699 total (was 696) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 36s the patch passed +1 unit 61m 6s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 79m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10966 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839654/HDFS-10966.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8ba1d89a341b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7584fbf Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17613/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17613/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17613/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment - - edited

          Thanks Kihwal for the review! Uploading a new patch:

          1. This patch does change the Balancer behavior introduced by HDFS-4261, around the timeout logic. But I don't think there's a negative effect. By staying in the dispatchBlocks while loop longer, the only overhead is to check chooseNextMove, which only checks local states, without issuing NameNode workload. Even if we jump out of the while loop, the thread for that Source cannot be reused at another Source anyway. In TestBalancer I reset the config value to 5s, and the run time is normal.
          2. Added to hdfs-default.xml, thx for the catch.
          3. I think it is a good idea, added.
          Show
          zhz Zhe Zhang added a comment - - edited Thanks Kihwal for the review! Uploading a new patch: This patch does change the Balancer behavior introduced by HDFS-4261 , around the timeout logic. But I don't think there's a negative effect. By staying in the dispatchBlocks while loop longer, the only overhead is to check chooseNextMove , which only checks local states, without issuing NameNode workload. Even if we jump out of the while loop, the thread for that Source cannot be reused at another Source anyway. In TestBalancer I reset the config value to 5s, and the run time is normal. Added to hdfs-default.xml , thx for the catch. I think it is a good idea, added.
          Hide
          kihwal Kihwal Lee added a comment -

          the patch looks reasonable in general.

          • It is replacing the logic introduced by HDFS-4261, so please double check there is no regression. Does the run-time of related test cases change?
          • Please document the new config variable in hdfs-default.xml.
          • If chooseNextMove() keeps failing, the while loop can spin for up to one minute. How about inserting a small delay, say, 100ms?
          Show
          kihwal Kihwal Lee added a comment - the patch looks reasonable in general. It is replacing the logic introduced by HDFS-4261 , so please double check there is no regression. Does the run-time of related test cases change? Please document the new config variable in hdfs-default.xml . If chooseNextMove() keeps failing, the while loop can spin for up to one minute. How about inserting a small delay, say, 100ms?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 4m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 18s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 38s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 702 unchanged - 3 fixed = 708 total (was 705)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 53m 59s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          77m 13s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.tools.TestDFSZKFailoverController
            hadoop.hdfs.tools.TestDelegationTokenFetcher
            hadoop.tools.TestHdfsConfigFields
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10966
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838640/HDFS-10966.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 24a5c6f025f4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fad9609
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17539/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17539/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17539/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17539/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 4m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 18s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 702 unchanged - 3 fixed = 708 total (was 705) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 53m 59s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 77m 13s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10966 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838640/HDFS-10966.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 24a5c6f025f4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fad9609 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17539/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17539/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17539/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17539/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Mark Wagner did an internal patch (based on 2.6). I'm uploading on his behalf.

          Kihwal Lee Could you take a look? Thanks!

          Show
          zhz Zhe Zhang added a comment - Mark Wagner did an internal patch (based on 2.6). I'm uploading on his behalf. Kihwal Lee Could you take a look? Thanks!

            People

            • Assignee:
              mwagner Mark Wagner
              Reporter:
              zhz Zhe Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development