HBase
  1. HBase
  2. HBASE-3271

Allow .META. table to be exported

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.6
    • Fix Version/s: 0.94.2
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      mapreduce.Export is able to export .META. table

      Description

      I tried to export .META. table in 0.20.6 and got:

      [hadoop@us01-ciqps1-name01 hbase]$ bin/hbase org.apache.hadoop.hbase.mapreduce.Export .META. h-meta 1 0 0
      10/11/23 20:59:05 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
      2010-11-23 20:59:05.255::INFO: Logging to STDERR via org.mortbay.log.StdErrLog
      2010-11-23 20:59:05.255::INFO: verisons=1, starttime=0, endtime=9223372036854775807
      10/11/23 20:59:05 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
      10/11/23 20:59:05 INFO zookeeper.ZooKeeper: Client environment:host.name=us01-ciqps1-name01.carrieriq.com
      10/11/23 20:59:05 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_21
      10/11/23 20:59:05 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
      ...
      10/11/23 20:59:05 INFO zookeeper.ClientCnxn: Server connection successful
      10/11/23 20:59:05 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.202.50.112:60020
      10/11/23 20:59:05 DEBUG client.HConnectionManager$TableServers: Found ROOT at 10.202.50.112:60020
      10/11/23 20:59:05 DEBUG client.HConnectionManager$TableServers: Cached location for .META.,,1 is us01-ciqps1-grid02.carrieriq.com:60020
      Exception in thread "main" java.io.IOException: Expecting at least one region.
      at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:281)
      at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
      at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
      at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
      at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
      at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:146)

      Related code is:
      if (keys == null || keys.getFirst() == null ||
      keys.getFirst().length == 0)

      { throw new IOException("Expecting at least one region."); }

      My intention was to save the dangling rows in .META. (for future investigation) which prevented a table from being created.

      1. 3271.94
        3 kB
        Ted Yu
      2. HBASE-3271.patch
        4 kB
        Liang Xie
      3. HBASE-3271-v2.patch
        4 kB
        Liang Xie

        Activity

        stack made changes -
        Fix Version/s 0.94.2 [ 12321884 ]
        Fix Version/s 0.95.0 [ 12324094 ]
        Hide
        stack added a comment -

        Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl)

        Show
        stack added a comment - Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl)
        stack made changes -
        Fix Version/s 0.98.0 [ 12323143 ]
        stack made changes -
        Fix Version/s 0.98.0 [ 12323143 ]
        Fix Version/s 0.94.2 [ 12321884 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.2 [ 12321884 ]
        stack made changes -
        Fix Version/s 0.95.0 [ 12324094 ]
        Fix Version/s 0.96.0 [ 12320040 ]
        Fix Version/s 0.94.2 [ 12321884 ]
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/)
        HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376807)

        Result = FAILURE
        Tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/ ) HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376807) Result = FAILURE Tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #50 (See https://builds.apache.org/job/HBase-0.94-security/50/)
        HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376807)

        Result = FAILURE
        Tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #50 (See https://builds.apache.org/job/HBase-0.94-security/50/ ) HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376807) Result = FAILURE Tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Ted Yu made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Assignee liang xie [ xieliang007 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #425 (See https://builds.apache.org/job/HBase-0.94/425/)
        HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376807)

        Result = FAILURE
        Tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #425 (See https://builds.apache.org/job/HBase-0.94/425/ ) HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376807) Result = FAILURE Tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Ted Yu made changes -
        Release Note mapreduce.Export is able to export .META. table
        Fix Version/s 0.94.2 [ 12321884 ]
        Hide
        Ted Yu added a comment -

        Integrated to 0.94 branch as well.

        Show
        Ted Yu added a comment - Integrated to 0.94 branch as well.
        Ted Yu made changes -
        Attachment 3271.94 [ 12542232 ]
        Hide
        Ted Yu added a comment -

        Patch for 0.94 branch.

        TestImportExport passes.

        Show
        Ted Yu added a comment - Patch for 0.94 branch. TestImportExport passes.
        Ted Yu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #144 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/144/)
        HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376487)

        Result = FAILURE
        Tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #144 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/144/ ) HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376487) Result = FAILURE Tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3257 (See https://builds.apache.org/job/HBase-TRUNK/3257/)
        HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376487)

        Result = SUCCESS
        Tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3257 (See https://builds.apache.org/job/HBase-TRUNK/3257/ ) HBASE-3271 Allow .META. table to be exported (Liang Xie) (Revision 1376487) Result = SUCCESS Tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
        Hide
        Ted Yu added a comment -
        Running org.apache.hadoop.hbase.mapreduce.TestImportExport
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.517 sec
        

        Failed test is not related to this patch.

        Patch integrated to trunk.

        Thanks for the patch, Liang.

        Show
        Ted Yu added a comment - Running org.apache.hadoop.hbase.mapreduce.TestImportExport Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.517 sec Failed test is not related to this patch. Patch integrated to trunk. Thanks for the patch, Liang.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542086/HBASE-3271-v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542086/HBASE-3271-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2657//console This message is automatically generated.
        Liang Xie made changes -
        Attachment HBASE-3271-v2.patch [ 12542086 ]
        Hide
        Liang Xie added a comment -

        Modified per Ted's comments

        Show
        Liang Xie added a comment - Modified per Ted's comments
        Hide
        Ted Yu added a comment -

        Please address my comment @ 22/Aug/12 18:03 and attach new patch.

        Thanks

        Show
        Ted Yu added a comment - Please address my comment @ 22/Aug/12 18:03 and attach new patch. Thanks
        Hide
        Liang Xie added a comment -

        @Ted, what should i do for "QA run was aborted" ? or could you kindly just kick another QA run instead?

        Show
        Liang Xie added a comment - @Ted, what should i do for "QA run was aborted" ? or could you kindly just kick another QA run instead?
        Hide
        Ted Yu added a comment -
        Show
        Ted Yu added a comment - According to https://builds.apache.org/job/PreCommit-HBASE-Build/2649/console , QA run was aborted.
        Ted Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.96.0 [ 12320040 ]
        Hide
        Ted Yu added a comment - - edited

        nit: the following two lines are too long:

        +    GenericOptionsParser opts = new GenericOptionsParser(new Configuration(cluster.getConfiguration()), args);
        ...
        +      InputSplit split = new TableSplit(table.getTableName(), HConstants.EMPTY_BYTE_ARRAY, HConstants.EMPTY_BYTE_ARRAY,
        
        Show
        Ted Yu added a comment - - edited nit: the following two lines are too long: + GenericOptionsParser opts = new GenericOptionsParser( new Configuration(cluster.getConfiguration()), args); ... + InputSplit split = new TableSplit(table.getTableName(), HConstants.EMPTY_BYTE_ARRAY, HConstants.EMPTY_BYTE_ARRAY,
        Hide
        stack added a comment -

        Patch looks fine to me. Will integrate in next day or so unless objection.

        Show
        stack added a comment - Patch looks fine to me. Will integrate in next day or so unless objection.
        Hide
        Liang Xie added a comment -

        @Stack @Ted or some other guys, could you kindly help to review? thanks in advance

        Show
        Liang Xie added a comment - @Stack @Ted or some other guys, could you kindly help to review? thanks in advance
        Hide
        Liang Xie added a comment -

        I tested as following steps:
        1) ran new test case : mvn test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestImportExport

        Failed as expected:

        Running org.apache.hadoop.hbase.mapreduce.TestImportExport
        2012-08-22 13:18:03.351 java[2329:1903] Unable to load realm info from SCDynamicStore
        Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 124.011 sec <<< FAILURE!

        Results :

        Tests in error:
        testMetaExport(org.apache.hadoop.hbase.mapreduce.TestImportExport): Expecting at least one region.

        Tests run: 3, Failures: 0, Errors: 1, Skipped: 0

        2) After applied patch, the test case passed as expected

        3) Finally, i verified sucessfully at a real cluster manually as well

        Show
        Liang Xie added a comment - I tested as following steps: 1) ran new test case : mvn test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestImportExport Failed as expected: Running org.apache.hadoop.hbase.mapreduce.TestImportExport 2012-08-22 13:18:03.351 java [2329:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 124.011 sec <<< FAILURE! Results : Tests in error: testMetaExport(org.apache.hadoop.hbase.mapreduce.TestImportExport): Expecting at least one region. Tests run: 3, Failures: 0, Errors: 1, Skipped: 0 2) After applied patch, the test case passed as expected 3) Finally, i verified sucessfully at a real cluster manually as well
        Liang Xie made changes -
        Field Original Value New Value
        Attachment HBASE-3271.patch [ 12541871 ]
        Hide
        Liang Xie added a comment -

        Seems this issue is still existing, at least for TRUNK. I just tidied Ted's old code, changed a DEPRECATED method call, and created a new test case for this corner case

        Show
        Liang Xie added a comment - Seems this issue is still existing, at least for TRUNK. I just tidied Ted's old code, changed a DEPRECATED method call, and created a new test case for this corner case
        Hide
        stack added a comment -

        Any chance of your making the code above into a patch and attaching it to this issue so we can review it Ted? (This might be of help: http://www.apache.org/dev/contributors.html#patches). Thanks.

        Show
        stack added a comment - Any chance of your making the code above into a patch and attaching it to this issue so we can review it Ted? (This might be of help: http://www.apache.org/dev/contributors.html#patches ). Thanks.
        Hide
        Ted Yu added a comment - - edited

        I used this code:

            if (keys == null || keys.getFirst() == null || 
                keys.getFirst().length == 0) {
            	HRegionLocation regLoc = table.getRegionLocation(HConstants.EMPTY_BYTE_ARRAY);
            	if (null == regLoc)
            		throw new IOException("Expecting at least one region.");
            	List<InputSplit> splits = new ArrayList<InputSplit>(1); 
            	InputSplit split = new TableSplit(table.getTableName(),
            			HConstants.EMPTY_BYTE_ARRAY, HConstants.EMPTY_BYTE_ARRAY,
            			regLoc.getServerAddress().getHostname());
            	splits.add(split);
            	return splits;
            }
        

        The following command only exports rows in .META. which have 'packageindex' (refer to HBASE-3255):
        bin/hbase org.apache.hadoop.hbase.mapreduce.Export .META. h-meta 1 0 0 packageindex

        -rwxrwxrwx 1 hadoop users 90700 Nov 24 03:31 h-meta/part-m-00000

        Show
        Ted Yu added a comment - - edited I used this code: if (keys == null || keys.getFirst() == null || keys.getFirst().length == 0) { HRegionLocation regLoc = table.getRegionLocation(HConstants.EMPTY_BYTE_ARRAY); if ( null == regLoc) throw new IOException( "Expecting at least one region." ); List<InputSplit> splits = new ArrayList<InputSplit>(1); InputSplit split = new TableSplit(table.getTableName(), HConstants.EMPTY_BYTE_ARRAY, HConstants.EMPTY_BYTE_ARRAY, regLoc.getServerAddress().getHostname()); splits.add(split); return splits; } The following command only exports rows in .META. which have 'packageindex' (refer to HBASE-3255 ): bin/hbase org.apache.hadoop.hbase.mapreduce.Export .META. h-meta 1 0 0 packageindex -rwxrwxrwx 1 hadoop users 90700 Nov 24 03:31 h-meta/part-m-00000
        Ted Yu created issue -

          People

          • Assignee:
            Liang Xie
            Reporter:
            Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development