HBase
  1. HBase
  2. HBASE-11083

ExportSnapshot should provide capability to limit bandwidth consumption

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0
    • Component/s: snapshots
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This capability was first brought up in this thread:
      http://search-hadoop.com/m/DHED4Td8Xb1

      The rewritten distcp already provides this capability.
      See MAPREDUCE-2765

      distcp implementation utilizes ThrottledInputStream which provides bandwidth throttling on a specified InputStream.

      As a first step, we can

      • add an option to ExportSnapshot which expresses bandwidth per map in MB
      • utilize ThrottledInputStream in ExportSnapshot#ExportMapper#copyFile().

        Activity

        Ted Yu created issue -
        Ted Yu made changes -
        Field Original Value New Value
        Assignee Ted Yu [ yuzhihong@gmail.com ]
        Ted Yu made changes -
        Attachment 11083-v1.txt [ 12642060 ]
        Hide
        Matteo Bertozzi added a comment -

        assuming that the distcp/hadoop2 dependency is ok for the project
        the patch looks good to me

        Show
        Matteo Bertozzi added a comment - assuming that the distcp/hadoop2 dependency is ok for the project the patch looks good to me
        Ted Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12642060/11083-v1.txt
        against trunk revision .
        ATTACHMENT ID: 12642060

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 8 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:

        -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.mapreduce.TestTableMapReduceBase.testMultiRegionTable(TestTableMapReduceBase.java:96)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642060/11083-v1.txt against trunk revision . ATTACHMENT ID: 12642060 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc . The javadoc tool appears to have generated 8 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.mapreduce.TestTableMapReduceBase.testMultiRegionTable(TestTableMapReduceBase.java:96) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9406//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Patch v2 adds optimization where ThrottledInputStream is only created when bandwidth parameter is non-default value.

        Show
        Ted Yu added a comment - Patch v2 adds optimization where ThrottledInputStream is only created when bandwidth parameter is non-default value.
        Ted Yu made changes -
        Attachment 11083-v2.txt [ 12642081 ]
        Hide
        Matteo Bertozzi added a comment -

        +1, is this trunk only since it depends on hadoop2?

        Show
        Matteo Bertozzi added a comment - +1, is this trunk only since it depends on hadoop2?
        Hide
        Ted Yu added a comment -

        is this trunk only since it depends on hadoop2?

        Yes.

        Show
        Ted Yu added a comment - is this trunk only since it depends on hadoop2? Yes.
        Ted Yu made changes -
        Attachment 11083-v2.txt [ 12642081 ]
        Ted Yu made changes -
        Attachment 11083-v2.txt [ 12642082 ]
        Ted Yu made changes -
        Attachment 11083-v1.txt [ 12642060 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12642082/11083-v2.txt
        against trunk revision .
        ATTACHMENT ID: 12642082

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 8 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642082/11083-v2.txt against trunk revision . ATTACHMENT ID: 12642082 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc . The javadoc tool appears to have generated 8 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9408//console This message is automatically generated.
        Ted Yu made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.99.0 [ 12325675 ]
        Hide
        Ted Yu added a comment -

        Thanks for the review, Matteo.

        Show
        Ted Yu added a comment - Thanks for the review, Matteo.
        Hide
        Tianying Chang added a comment -

        Ted Yu We use 0.94.7 and Hadoop 2.0.0-cdh4.2.0. Can this patch work with this combination?

        Show
        Tianying Chang added a comment - Ted Yu We use 0.94.7 and Hadoop 2.0.0-cdh4.2.0. Can this patch work with this combination?
        Hide
        Ted Yu added a comment -

        You can check whether MAPREDUCE-2765 is in the distro you're using - I assume it is.
        If so, it should work after minor modification.

        Show
        Ted Yu added a comment - You can check whether MAPREDUCE-2765 is in the distro you're using - I assume it is. If so, it should work after minor modification.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #5121 (See https://builds.apache.org/job/HBase-TRUNK/5121/)
        HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1590291)

        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5121 (See https://builds.apache.org/job/HBase-TRUNK/5121/ ) HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1590291) /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Ted Yu made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #309 (See https://builds.apache.org/job/HBase-0.98/309/)
        HBASE-11090 Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1595946)

        • /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport
        • /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #309 (See https://builds.apache.org/job/HBase-0.98/309/ ) HBASE-11090 Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1595946) /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #5135 (See https://builds.apache.org/job/HBase-TRUNK/5135/)
        HBASE-11090 Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1595947)

        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport
        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java
        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5135 (See https://builds.apache.org/job/HBase-TRUNK/5135/ ) HBASE-11090 Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1595947) /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Hide
        Enis Soztutar added a comment -

        Closing this issue after 0.99.0 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 0.99.0 release.
        Enis Soztutar made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        4h 48m 1 Ted Yu 26/Apr/14 06:36
        Patch Available Patch Available Resolved Resolved
        23h 22m 1 Ted Yu 27/Apr/14 05:59
        Resolved Resolved Closed Closed
        300d 18h 30m 1 Enis Soztutar 21/Feb/15 23:29

          People

          • Assignee:
            Ted Yu
            Reporter:
            Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development