Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: balancer & mover
    • Labels:
      None

      Description

      Disk balancer follows/reuses one rule designed by HDFS balancer, that is, only one instance is allowed to run at the same time. This is correct in production system to avoid any inconsistencies, but it's not ideal to write and run unit tests. For example, it should be allowed run plan, execute, scan commands under one setup of disk balancer. One instance rule will throw exception by complaining 'Another instance is running'. In such a case, there's no way to do a full life cycle tests which involves a sequence of commands.

      1. HDFS-11038.000.patch
        15 kB
        Xiaobing Zhou
      2. HDFS-11038.001.patch
        15 kB
        Xiaobing Zhou

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10693 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10693/)
          HDFS-11038. DiskBalancer: support running multiple commands in single (aengineer: rev 9f32364d283dec47dd07490e253d477a0d14ac71)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancerCLI.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/PlanCommand.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/DiskBalancerTestUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10693 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10693/ ) HDFS-11038 . DiskBalancer: support running multiple commands in single (aengineer: rev 9f32364d283dec47dd07490e253d477a0d14ac71) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancerCLI.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/PlanCommand.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/DiskBalancerTestUtil.java
          Hide
          anu Anu Engineer added a comment -

          Xiaobing Zhou Thank you for the contribution. I have committed this to trunk.

          Show
          anu Anu Engineer added a comment - Xiaobing Zhou Thank you for the contribution. I have committed this to trunk.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          The test failure is irrelevant to the patch. I marked it in HDFS-10406.

          Show
          xiaobingo Xiaobing Zhou added a comment - The test failure is irrelevant to the patch. I marked it in HDFS-10406 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 19s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 42s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 39s the patch passed
          -1 unit 58m 35s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          77m 54s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
          Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
            org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            org.apache.hadoop.fs.TestEnhancedByteBufferAccess
            org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
            org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            org.apache.hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA
            org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
            org.apache.hadoop.hdfs.server.namenode.TestAuditLogs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-11038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835426/HDFS-11038.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0590dcfda5a2 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 22ff0ef
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17301/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17301/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17301/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 19s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 58m 35s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 77m 54s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestReconstructStripedBlocks Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement   org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   org.apache.hadoop.fs.TestEnhancedByteBufferAccess   org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM   org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer   org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   org.apache.hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA   org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA   org.apache.hadoop.hdfs.server.namenode.TestAuditLogs Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835426/HDFS-11038.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0590dcfda5a2 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22ff0ef Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17301/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17301/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17301/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          anu Anu Engineer added a comment -

          +1, pending Jenkins. LGTM. Thanks for updating the patch.

          Show
          anu Anu Engineer added a comment - +1, pending Jenkins. LGTM. Thanks for updating the patch.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Posted a patch v001 to cover the changes aforementioned.

          Show
          xiaobingo Xiaobing Zhou added a comment - Posted a patch v001 to cover the changes aforementioned.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Sure, we can merge these two, for example
          Writing plan to: /system/diskbalancer/2016-Oct-26-15-29-58
          /system/diskbalancer/2016-Oct-26-15-29-58/dbe1178e-a4fc-4cb7-9419-5fbf6e0f67a3.plan.json

          will be like
          Writing plan to:
          /system/diskbalancer/2016-Oct-26-15-29-58/dbe1178e-a4fc-4cb7-9419-5fbf6e0f67a3.plan.json

          Thanks Anu Engineer

          Show
          xiaobingo Xiaobing Zhou added a comment - Sure, we can merge these two, for example Writing plan to: /system/diskbalancer/2016-Oct-26-15-29-58 /system/diskbalancer/2016-Oct-26-15-29-58/dbe1178e-a4fc-4cb7-9419-5fbf6e0f67a3.plan.json will be like Writing plan to: /system/diskbalancer/2016-Oct-26-15-29-58/dbe1178e-a4fc-4cb7-9419-5fbf6e0f67a3.plan.json Thanks Anu Engineer
          Hide
          anu Anu Engineer added a comment -

          Just wondering if it makes sense to print both of this to console ? Should we modify the writing file info trace to have the full path if needed ..

          Show
          anu Anu Engineer added a comment - Just wondering if it makes sense to print both of this to console ? Should we modify the writing file info trace to have the full path if needed ..
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Anu Engineer, planFileFullName is prefix path plus planFileName, for example,
          planFileName: 1b7b7765-2555-4c6b-ace1-f13321dae758.plan.json
          planFileFullName: /system/diskbalancer/2016-Oct-26-14-57-10/1b7b7765-2555-4c6b-ace1-f13321dae758.plan.json

          Show
          xiaobingo Xiaobing Zhou added a comment - Anu Engineer , planFileFullName is prefix path plus planFileName, for example, planFileName: 1b7b7765-2555-4c6b-ace1-f13321dae758.plan.json planFileFullName: /system/diskbalancer/2016-Oct-26-14-57-10/1b7b7765-2555-4c6b-ace1-f13321dae758.plan.json
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 7s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 55s the patch passed
          +1 compile 0m 50s the patch passed
          +1 javac 0m 50s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 5s the patch passed
          +1 javadoc 0m 41s the patch passed
          -1 unit 49m 37s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          70m 50s



          Reason Tests
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            org.apache.hadoop.hdfs.TestFileCreation
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
            org.apache.hadoop.hdfs.TestFileAppend3
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            org.apache.hadoop.hdfs.TestLeaseRecovery
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache
            org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-11038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835415/HDFS-11038.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 74f3221344b9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f511cc8
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17298/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17298/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 7s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 55s the patch passed +1 compile 0m 50s the patch passed +1 javac 0m 50s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 5s the patch passed +1 javadoc 0m 41s the patch passed -1 unit 49m 37s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 70m 50s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   org.apache.hadoop.hdfs.TestFileCreation   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   org.apache.hadoop.hdfs.TestFileAppend3   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   org.apache.hadoop.hdfs.TestLeaseRecovery   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835415/HDFS-11038.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 74f3221344b9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f511cc8 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17298/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17298/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          anu Anu Engineer added a comment -

          Thank you for the patch, just a quick question. In PlanCommand.java#execute are you sure that planFileName and planFileFullName are different ?

          Show
          anu Anu Engineer added a comment - Thank you for the patch, just a quick question. In PlanCommand.java#execute are you sure that planFileName and planFileFullName are different ?
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          I posted initial patch v000 for reviews.

          Show
          xiaobingo Xiaobing Zhou added a comment - I posted initial patch v000 for reviews.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Reopened this ticket since HDFS-9462 will be deprecated in that some of work overlaps with Query command, however, the work supporting running multiple commands sequentially in one setup is quite useful, and will be tracked here.

          Show
          xiaobingo Xiaobing Zhou added a comment - Reopened this ticket since HDFS-9462 will be deprecated in that some of work overlaps with Query command, however, the work supporting running multiple commands sequentially in one setup is quite useful, and will be tracked here.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          As explained, it's already implemented in HDFS-9462 patch. Closed this ticket.

          Show
          xiaobingo Xiaobing Zhou added a comment - As explained, it's already implemented in HDFS-9462 patch. Closed this ticket.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          This can be fixed by closing FileSystem instance maintaine by org.apache.hadoop.hdfs.server.diskbalancer.command. See also HDFS-9462.002.patch.

          110	  /**
          111	   * Cleans any resources held by this command.
          112	   * <p>
          113	   * The main goal is to delete id file created in
          114	   * {@link org.apache.hadoop.hdfs.server.balancer.checkAndMarkRunning},
          115	   * otherwise, it's not allowed to run multiple commands in a row.
          116	   * </p>
          117	   */
          118	  @Override
          119	  public void close() throws IOException {
          120	    if (fs != null) {
          121	      fs.close();
          122	    }
          123	  }
          124	
          
          Show
          xiaobingo Xiaobing Zhou added a comment - This can be fixed by closing FileSystem instance maintaine by org.apache.hadoop.hdfs.server.diskbalancer.command. See also HDFS-9462.002.patch . 110 /** 111 * Cleans any resources held by this command. 112 * <p> 113 * The main goal is to delete id file created in 114 * {@link org.apache.hadoop.hdfs.server.balancer.checkAndMarkRunning}, 115 * otherwise, it's not allowed to run multiple commands in a row. 116 * </p> 117 */ 118 @Override 119 public void close() throws IOException { 120 if (fs != null ) { 121 fs.close(); 122 } 123 } 124

            People

            • Assignee:
              xiaobingo Xiaobing Zhou
              Reporter:
              xiaobingo Xiaobing Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development