HBase
  1. HBase
  2. HBASE-11090

Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 0.98.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBASE-11083 allows ExportSnapshot to limit bandwidth usage.
      Here is one approach for backporting:

      Create the following classes (class name is tentative):
      hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/util/ThrottledInputStream.java
      hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/util/ThrottledInputStream.java

      each of which extends the corresponding ThrottledInputStream in hadoop-1 / hadoop-2

      ExportSnapshot would reference util.ThrottledInputStream, depending on which compatibility module gets bundled.

      ThrottledInputStream.java in hadoop-1 branch was backported through MAPREDUCE-5081 which went into 1.2.0 release.
      We need to decide how hadoop releases earlier than 1.2.0 should be supported.

      Second approach for backporting is to make a copy of ThrottledInputStream and include it in hbase codebase.

      1. 11090-trunk-v2.txt
        6 kB
        Ted Yu
      2. 11090-trunk.txt
        6 kB
        Ted Yu
      3. 11090-98.txt
        9 kB
        Ted Yu
      4. 11090-0.98-v1.txt
        9 kB
        Ted Yu

        Activity

        Hide
        Enis Soztutar added a comment -

        Closing this issue after 0.99.0 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 0.99.0 release.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #5135 (See https://builds.apache.org/job/HBase-TRUNK/5135/)
        HBASE-11090 Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1595947)

        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport
        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java
        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5135 (See https://builds.apache.org/job/HBase-TRUNK/5135/ ) HBASE-11090 Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1595947) /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #309 (See https://builds.apache.org/job/HBase-0.98/309/)
        HBASE-11090 Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1595946)

        • /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport
        • /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #309 (See https://builds.apache.org/job/HBase-0.98/309/ ) HBASE-11090 Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption (tedyu: rev 1595946) /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        Hide
        Ted Yu added a comment -

        Thanks for the review, Andy.

        I have integrated to 0.98 and trunk.

        Show
        Ted Yu added a comment - Thanks for the review, Andy. I have integrated to 0.98 and trunk.
        Hide
        Andrew Purtell added a comment -

        +1 on patch v2 for trunk, sorry for the delay Ted.

        +1 on a backport / commit to 0.98 of HBASE-11083 if you wish.

        Show
        Andrew Purtell added a comment - +1 on patch v2 for trunk, sorry for the delay Ted. +1 on a backport / commit to 0.98 of HBASE-11083 if you wish.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12643324/11090-trunk-v2.txt
        against trunk revision .
        ATTACHMENT ID: 12643324

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12643324/11090-trunk-v2.txt against trunk revision . ATTACHMENT ID: 12643324 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9464//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Andrew Purtell:
        If you think patch v2 is good, I will provide patch for 0.98 as well.

        Show
        Ted Yu added a comment - Andrew Purtell : If you think patch v2 is good, I will provide patch for 0.98 as well.
        Hide
        Ted Yu added a comment -

        Patch v2 moves the new class to hbase-common module

        Show
        Ted Yu added a comment - Patch v2 moves the new class to hbase-common module
        Hide
        Ted Yu added a comment -

        Hbase-common would be better for the backport.

        The change in Pom was intentional - the dependency was solely for this class.

        Show
        Ted Yu added a comment - Hbase-common would be better for the backport. The change in Pom was intentional - the dependency was solely for this class.
        Hide
        Andrew Purtell added a comment -

        The port of ThrottledInputStream looks fine. Should it go into hbase-common? Is the POM change intentional?

        Show
        Andrew Purtell added a comment - The port of ThrottledInputStream looks fine. Should it go into hbase-common? Is the POM change intentional?
        Hide
        Ted Yu added a comment -

        Andrew Purtell:
        Is the trunk patch good to go ?

        Show
        Ted Yu added a comment - Andrew Purtell : Is the trunk patch good to go ?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12643172/11090-trunk.txt
        against trunk revision .
        ATTACHMENT ID: 12643172

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestMultiParallel

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12643172/11090-trunk.txt against trunk revision . ATTACHMENT ID: 12643172 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestMultiParallel Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9454//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Patch for trunk which adds the backport.
        ExportSnapshot.java refers to the backported class.

        Show
        Ted Yu added a comment - Patch for trunk which adds the backport. ExportSnapshot.java refers to the backported class.
        Hide
        Andrew Purtell added a comment -

        If you commit ThrottledInputStream not into a Hadoop compat module, which I think is overkill, then it should go into trunk first, then 0.98, as customary. Once we have it in our codebase it doesn't matter what happens with Hadoop.

        Show
        Andrew Purtell added a comment - If you commit ThrottledInputStream not into a Hadoop compat module, which I think is overkill, then it should go into trunk first, then 0.98, as customary. Once we have it in our codebase it doesn't matter what happens with Hadoop.
        Hide
        Ted Yu added a comment -

        w.r.t. whether to move ThrottledInputStream into trunk, my thinking is that there would be one or more hadoop 2.x releases before HBase 1.0 goes out.
        Meaning, hadoop-2 dependency would be > 2.4.0
        If there is improvement to ThrottledInputStream in this timeframe, we would pick that up by upgrading to newer 2.x hadoop release.

        What do you think ?

        Show
        Ted Yu added a comment - w.r.t. whether to move ThrottledInputStream into trunk, my thinking is that there would be one or more hadoop 2.x releases before HBase 1.0 goes out. Meaning, hadoop-2 dependency would be > 2.4.0 If there is improvement to ThrottledInputStream in this timeframe, we would pick that up by upgrading to newer 2.x hadoop release. What do you think ?
        Hide
        Andrew Purtell added a comment - - edited

        We don't support Hadoop 1 on 0.98 but still must build against it, and we munge -1 and -2 maven artifacts like 0.96.

        The patch looks fine Ted, although if moving ThrottledInputStream into our codebase, then it should go in on trunk as well, right?

        Show
        Andrew Purtell added a comment - - edited We don't support Hadoop 1 on 0.98 but still must build against it, and we munge -1 and -2 maven artifacts like 0.96. The patch looks fine Ted, although if moving ThrottledInputStream into our codebase, then it should go in on trunk as well, right?
        Hide
        Ted Yu added a comment -

        HBase-0.98-on-Hadoop-1.1 would be broken if I use the patch from HBASE-11083 directly.

        Show
        Ted Yu added a comment - HBase-0.98-on-Hadoop-1.1 would be broken if I use the patch from HBASE-11083 directly.
        Hide
        Matteo Bertozzi added a comment -

        doesn't this page says that only hadoop 2 is supported in 98?
        https://hbase.apache.org/book/configuration.html#hadoop

        Show
        Matteo Bertozzi added a comment - doesn't this page says that only hadoop 2 is supported in 98? https://hbase.apache.org/book/configuration.html#hadoop
        Hide
        Ted Yu added a comment -

        Matteo Bertozzi,Andrew Purtell:
        What do you think of the patch ?

        Thanks

        Show
        Ted Yu added a comment - Matteo Bertozzi , Andrew Purtell : What do you think of the patch ? Thanks
        Hide
        Ted Yu added a comment -

        Patch v1 backports ThrottledInputStream.java from hadoop.

        This is in consideration of supporting hadoop releases earlier than 1.2.0

        Comments are welcome.

        Show
        Ted Yu added a comment - Patch v1 backports ThrottledInputStream.java from hadoop. This is in consideration of supporting hadoop releases earlier than 1.2.0 Comments are welcome.

          People

          • Assignee:
            Ted Yu
            Reporter:
            Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development