Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.99.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Import with this fix supports

      a) Filtering of the row using the Filter#filterRowKey(byte[] buffer, int offset, int length).

      b) Accepts durability parameter (Ex: -Dimport.wal.durability=SKIP_WAL ) while importing the data into HBase. If the data doesn't need to be replicated to the DR cluster or if the same import job would be run on the DR cluster, consider using SKIP_WAL durability for performance.
      Show
      Import with this fix supports a) Filtering of the row using the Filter#filterRowKey(byte[] buffer, int offset, int length). b) Accepts durability parameter (Ex: -Dimport.wal.durability=SKIP_WAL ) while importing the data into HBase. If the data doesn't need to be replicated to the DR cluster or if the same import job would be run on the DR cluster, consider using SKIP_WAL durability for performance.

      Description

      Following improvements can be made to the Import logic

      a) Make the import extensible (i.e., remove the filter from being a static member of Import and make it an instance variable of the mapper, make the mappers or variables of interest protected. )

      b) Make sure that the Import calls filterRowKey method of the filter (Useful if we want to filter the data of an organization based on the row key or using filters like PrefixFilter which filter the data in filterRowKey method rather than the filterKeyValue method). The existing test case in TestImportExport#testWithFilter works with this assumption but is so far successful because there is only one row inserted into the table.

      c) Provide an option to specify the durability during the import (Specifying the Durability as SKIP_WAL would improve the performance of restore considerably.) Lars Hofhansl suggested that this should be a parameter to the import.

      d) Some minor refactoring to avoid building a comma separated string for the filter args.

      1. HBASE-10416-rev1.patch
        20 kB
        Vasu Mariyala
      2. HBASE-10416.patch
        19 kB
        Vasu Mariyala

        Activity

        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #69 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/69/)
        HBASE-10416 Improvements to the import flow (Vasu Mariyala) (tedyu: rev 1562343)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #69 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/69/ ) HBASE-10416 Improvements to the import flow (Vasu Mariyala) (tedyu: rev 1562343) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Hide
        Nick Dimiduk added a comment -

        Vasu Mariyala let's take up the conversation of the other tickets elsewhere.

        Show
        Nick Dimiduk added a comment - Vasu Mariyala let's take up the conversation of the other tickets elsewhere.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #113 (See https://builds.apache.org/job/HBase-0.98/113/)
        HBASE-10416 Improvements to the import flow (Vasu Mariyala) (tedyu: rev 1562342)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #113 (See https://builds.apache.org/job/HBase-0.98/113/ ) HBASE-10416 Improvements to the import flow (Vasu Mariyala) (tedyu: rev 1562342) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #104 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/104/)
        HBASE-10416 Improvements to the import flow (Vasu Mariyala) (tedyu: rev 1562342)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #104 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/104/ ) HBASE-10416 Improvements to the import flow (Vasu Mariyala) (tedyu: rev 1562342) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4863 (See https://builds.apache.org/job/HBase-TRUNK/4863/)
        HBASE-10416 Improvements to the import flow (tedyu: rev 1562343)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4863 (See https://builds.apache.org/job/HBase-TRUNK/4863/ ) HBASE-10416 Improvements to the import flow (tedyu: rev 1562343) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Hide
        Andrew Purtell added a comment -

        +1 for 0.98, thanks Ted.

        Show
        Andrew Purtell added a comment - +1 for 0.98, thanks Ted.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12625635/HBASE-10416-rev1.patch
        against trunk revision .
        ATTACHMENT ID: 12625635

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.util.TestHBaseFsck

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12625635/HBASE-10416-rev1.patch against trunk revision . ATTACHMENT ID: 12625635 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.util.TestHBaseFsck Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8544//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        lgtm

        Andrew Purtell:
        Do you want this in 0.98 ?

        Show
        Ted Yu added a comment - lgtm Andrew Purtell : Do you want this in 0.98 ?
        Hide
        Vasu Mariyala added a comment -

        Sorry for the delay and thanks for the review comments

        Ted Yu

        1. Felt that constructing a filter object from filter class and filter args would be utility method and would be useful when extending the import utility for specific customizations.
        2. Fixed the long line & the java doc warnings.
        3. Updated the release note description in the jira.

        Nick Dimiduk

        Saw your other issues related to making things like mapper or reducer configurable and reuse the code. Would you mind discussing on these issues when you are free. You can ping me in gmail.

        Show
        Vasu Mariyala added a comment - Sorry for the delay and thanks for the review comments Ted Yu 1. Felt that constructing a filter object from filter class and filter args would be utility method and would be useful when extending the import utility for specific customizations. 2. Fixed the long line & the java doc warnings. 3. Updated the release note description in the jira. Nick Dimiduk Saw your other issues related to making things like mapper or reducer configurable and reuse the code. Would you mind discussing on these issues when you are free. You can ping me in gmail.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12625121/HBASE-10416.patch
        against trunk revision .
        ATTACHMENT ID: 12625121

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12625121/HBASE-10416.patch against trunk revision . ATTACHMENT ID: 12625121 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8504//console This message is automatically generated.
        Hide
        Nick Dimiduk added a comment -

        Good work Vasu Mariyala. Have you seen my notes on HBASE-8074 and HBASE-8115 ?

        Show
        Nick Dimiduk added a comment - Good work Vasu Mariyala . Have you seen my notes on HBASE-8074 and HBASE-8115 ?
        Hide
        Ted Yu added a comment -

        Nit:

        -  private static Filter instantiateFilter(Configuration conf) {
        -    // get the filter, if it was configured
        +  public static Filter instantiateFilter(Configuration conf) {
        

        Does instantiateFilter() need to be public ?

        +  public static void flushRegionsIfNecessary(Configuration conf) throws IOException, InterruptedException {
        

        Long line above.

        Mind adding release notes for the new parameters ?

        Show
        Ted Yu added a comment - Nit: - private static Filter instantiateFilter(Configuration conf) { - // get the filter, if it was configured + public static Filter instantiateFilter(Configuration conf) { Does instantiateFilter() need to be public ? + public static void flushRegionsIfNecessary(Configuration conf) throws IOException, InterruptedException { Long line above. Mind adding release notes for the new parameters ?
        Hide
        Lars Hofhansl added a comment -

        looks good to me. Thanks Vasu.

        Show
        Lars Hofhansl added a comment - looks good to me. Thanks Vasu.
        Hide
        Vasu Mariyala added a comment -

        Attaching the patch for the above mentioned issues.

        Show
        Vasu Mariyala added a comment - Attaching the patch for the above mentioned issues.

          People

          • Assignee:
            Vasu Mariyala
            Reporter:
            Vasu Mariyala
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development