Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12072

Provide fairness between EC and non-EC recovery tasks.

    Details

      Description

      In DatanodeManager#handleHeartbeat, it takes up to maxTransfer reconstruction tasks for non-EC, then if the request can not be full filled, it takes more tasks from EC reconstruction tasks.

          List<BlockTargetPair> pendingList = nodeinfo.getReplicationCommand(
              maxTransfers);
          if (pendingList != null) {
            cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId,
                pendingList));
            maxTransfers -= pendingList.size();
          }
          // check pending erasure coding tasks
          List<BlockECReconstructionInfo> pendingECList = nodeinfo
              .getErasureCodeCommand(maxTransfers);
          if (pendingECList != null) {
            cmds.add(new BlockECReconstructionCommand(
                DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList));
          }
      

      So on a large cluster, if there are large number of constantly non-EC reconstruction tasks, EC reconstruction tasks do not have a chance to run.

      1. HDFS-12072.00.patch
        8 kB
        Lei (Eddy) Xu
      2. HDFS-12072.01.patch
        9 kB
        Lei (Eddy) Xu

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12206 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12206/)
        HDFS-12072. Provide fairness between EC and non-EC recovery tasks. (wang: rev b29894889742dda654cd88a7ce72a4e51fccb328)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12206 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12206/ ) HDFS-12072 . Provide fairness between EC and non-EC recovery tasks. (wang: rev b29894889742dda654cd88a7ce72a4e51fccb328) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Thanks Andrew Wang!

        Show
        eddyxu Lei (Eddy) Xu added a comment - Thanks Andrew Wang !
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for the contribution Eddy, committed this to trunk!

        Show
        andrew.wang Andrew Wang added a comment - Thanks for the contribution Eddy, committed this to trunk!
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM +1, thanks for working on this Eddy, will commit shortly.

        Show
        andrew.wang Andrew Wang added a comment - LGTM +1, thanks for working on this Eddy, will commit shortly.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 16m 14s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 39s trunk passed
        +1 mvnsite 1m 3s trunk passed
        -1 findbugs 2m 0s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings.
        +1 javadoc 0m 44s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 54s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 34s the patch passed
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 37s the patch passed
              Other Tests
        -1 unit 65m 4s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        93m 44s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12072
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880494/HDFS-12072.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3cfb5affa7f9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 024c3ec
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20570/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20570/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20570/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20570/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 14s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 3s trunk passed -1 findbugs 2m 0s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. +1 javadoc 0m 44s trunk passed       Patch Compile Tests +1 mvninstall 0m 54s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 65m 4s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 93m 44s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12072 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880494/HDFS-12072.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3cfb5affa7f9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 024c3ec Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20570/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20570/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20570/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20570/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Thanks for the review, Andrew Wang

        Updated the patch to address the comments.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Thanks for the review, Andrew Wang Updated the patch to address the comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 16s trunk passed
        +1 compile 0m 49s trunk passed
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 0m 53s trunk passed
        -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 74 unchanged - 0 fixed = 77 total (was 74)
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 38s the patch passed
              Other Tests
        -1 unit 64m 51s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        89m 46s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestDecommissionWithStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12072
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879737/HDFS-12072.00.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 74b6334de6ab 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2be9412
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20506/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20506/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 16s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 53s trunk passed -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed       Patch Compile Tests +1 mvninstall 0m 48s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 74 unchanged - 0 fixed = 77 total (was 74) +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 38s the patch passed       Other Tests -1 unit 64m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 89m 46s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12072 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879737/HDFS-12072.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 74b6334de6ab 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2be9412 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20506/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20506/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for working on this Eddy! A few comments from looking at the patch:

        • Prefer if we rename instances of "ReplicateTask" to "ReplicationTask"
        • One idea to avoid polling twice, since we know the size of each queue (getNumberOfBlocksToBeReplicated and getNumberOfBlocksToBeErasureCoded), we can compute a ratio and then multiply.
        • Adding a slightly longer explanatory comment would also be good
        • Nit: There's also an unused import in the test which will probably be flagged in checkstyle.
        Show
        andrew.wang Andrew Wang added a comment - Thanks for working on this Eddy! A few comments from looking at the patch: Prefer if we rename instances of "ReplicateTask" to "ReplicationTask" One idea to avoid polling twice, since we know the size of each queue (getNumberOfBlocksToBeReplicated and getNumberOfBlocksToBeErasureCoded), we can compute a ratio and then multiply. Adding a slightly longer explanatory comment would also be good Nit: There's also an unused import in the test which will probably be flagged in checkstyle.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Update the patch to get half of maxTransfers tasks from each of 3x and EC pending recovery queue.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Update the patch to get half of maxTransfers tasks from each of 3x and EC pending recovery queue.

          People

          • Assignee:
            eddyxu Lei (Eddy) Xu
            Reporter:
            eddyxu Lei (Eddy) Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development