HBase
  1. HBase
  2. HBASE-5076

HBase shell hangs when creating some 'illegal' tables.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.92.0, 0.94.1, 0.94.2, 0.95.2
    • Fix Version/s: 0.95.0
    • Component/s: shell
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      In hbase shell. These commands hang:

      create 'hbase.version','foo'
      create 'splitlog','foo'
      

      Interestingly

      create 'hbase.id','foo'
      create existingtablename, 'foo'
      create '.META.','foo'
      create '-ROOT-','foo'
      

      are properly rejected.

      We should probably either rename to make the files illegal table names (hbase.version to .hbase.version and splitlog to .splitlog) or we could add more special cases.

        Activity

        Hide
        stack added a comment -

        Marking closed.

        Show
        stack added a comment - Marking closed.
        Hide
        Jonathan Hsieh added a comment -

        Liang Xie no worries and thanks for the consideration. I generally require a unit test or test output because it makes intent or changed output clear (not just for me but for others as well).

        Show
        Jonathan Hsieh added a comment - Liang Xie no worries and thanks for the consideration. I generally require a unit test or test output because it makes intent or changed output clear (not just for me but for others as well).
        Hide
        Liang Xie added a comment -

        Really thanks Jon for review. And blushing due to forget uploading my manual test output, lead to waste Jon's extra time

        Show
        Liang Xie added a comment - Really thanks Jon for review. And blushing due to forget uploading my manual test output, lead to waste Jon's extra time
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3296 (See https://builds.apache.org/job/HBase-TRUNK/3296/)
        HBASE-5076 HBase shell hangs when creating some 'illegal' tables (liang xie) (Revision 1379638)

        Result = FAILURE
        jmhsieh :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3296 (See https://builds.apache.org/job/HBase-TRUNK/3296/ ) HBASE-5076 HBase shell hangs when creating some 'illegal' tables (liang xie) (Revision 1379638) Result = FAILURE jmhsieh : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
        Hide
        Jonathan Hsieh added a comment -

        Committed to trunk. Thanks liang xie!

        Show
        Jonathan Hsieh added a comment - Committed to trunk. Thanks liang xie!
        Hide
        Jonathan Hsieh added a comment -

        nice,

        tested and it gave me this:

        hbase(main):003:0> create 'hbase.version', 'foo'
        
        ERROR: java.lang.IllegalArgumentException: hbase.version conflicted with system reserved words
        
        Here is some help for this command:
        Create table; pass table name, a dictionary of specifications per
        column family, and optionally a dictionary of table configuration.
        Dictionaries are described below in the GENERAL NOTES section.
        Examples:
        
          hbase> create 't1', {NAME => 'f1', VERSIONS => 5}
          hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
          hbase> # The above in shorthand would be the following:
          hbase> create 't1', 'f1', 'f2', 'f3'
          hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true}
          hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']}
          hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'}
          hbase> # Optionally pre-split the table into NUMREGIONS, using
          hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)
          hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}
        
          You can also keep around a reference to the created table:
        
          hbase> t1 = create 't1', 'f1'
        
          Which gives you a reference to the table named 't1', on which you can then
          call methods.
        
        
        hbase(main):004:0> create 'splitlog', 'foo'
        
        ERROR: java.lang.IllegalArgumentException: splitlog conflicted with system reserved words
        
        Here is some help for this command:
        Create table; pass table name, a dictionary of specifications per
        column family, and optionally a dictionary of table configuration.
        Dictionaries are described below in the GENERAL NOTES section.
        Examples:
        
          hbase> create 't1', {NAME => 'f1', VERSIONS => 5}
          hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
          hbase> # The above in shorthand would be the following:
          hbase> create 't1', 'f1', 'f2', 'f3'
          hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true}
          hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']}
          hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'}
          hbase> # Optionally pre-split the table into NUMREGIONS, using
          hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)
          hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}
        
          You can also keep around a reference to the created table:
        
          hbase> t1 = create 't1', 'f1'
        
          Which gives you a reference to the table named 't1', on which you can then
          call methods.
        
        Show
        Jonathan Hsieh added a comment - nice, tested and it gave me this: hbase(main):003:0> create 'hbase.version', 'foo' ERROR: java.lang.IllegalArgumentException: hbase.version conflicted with system reserved words Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true } hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} hbase> # Optionally pre-split the table into NUMREGIONS, using hbase> # SPLITALGO ( "HexStringSplit" , "UniformSplit" or classname) hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'} You can also keep around a reference to the created table: hbase> t1 = create 't1', 'f1' Which gives you a reference to the table named 't1', on which you can then call methods. hbase(main):004:0> create 'splitlog', 'foo' ERROR: java.lang.IllegalArgumentException: splitlog conflicted with system reserved words Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true } hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} hbase> # Optionally pre-split the table into NUMREGIONS, using hbase> # SPLITALGO ( "HexStringSplit" , "UniformSplit" or classname) hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'} You can also keep around a reference to the created table: hbase> t1 = create 't1', 'f1' Which gives you a reference to the table named 't1', on which you can then call methods.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12543097/HBASE-5076.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 110 warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.replication.TestReplication

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543097/HBASE-5076.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 110 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2738//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        Attached patch just do the filter job. Please kindly have a review, thanks!

        will file a new jira later per Jon's comments

        Show
        Liang Xie added a comment - Attached patch just do the filter job. Please kindly have a review, thanks! will file a new jira later per Jon's comments
        Hide
        Jonathan Hsieh added a comment -

        My suggestion is to do the filter first (so we can compatibly back port this to sustaining releases) and then file a jira to fix the problem by renaming the special files to the illegal table names in trunk. The trunk fix is significantly more involved because it requires migration checks for major version upgrades.

        Show
        Jonathan Hsieh added a comment - My suggestion is to do the filter first (so we can compatibly back port this to sustaining releases) and then file a jira to fix the problem by renaming the special files to the illegal table names in trunk. The trunk fix is significantly more involved because it requires migration checks for major version upgrades.
        Hide
        Liang Xie added a comment -

        BTW,this hang issue will introduce INCONSISTENT status :

        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/ROOT
        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/.META.
        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/.logs
        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/.oldlogs
        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/hbase.id
        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/splitlog
        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region info from hdfs:hdfs://10.235.2.121:9000/hbase/ROOT/70236052
        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region info from hdfs:hdfs://10.235.2.121:9000/hbase/.META./1028785192
        12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region info from hdfs:hdfs://10.235.2.121:9000/hbase/splitlog/6c35fbf061ec3d56c6c120fdb502f35e
        12/08/28 20:48:51 DEBUG util.HBaseFsck: HRegionInfo read:

        {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}

        12/08/28 20:48:51 DEBUG util.HBaseFsck: HRegionInfo read:

        {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}

        12/08/28 20:48:51 DEBUG util.HBaseFsck: HRegionInfo read:

        {NAME => 'splitlog,,1346156147925.6c35fbf061ec3d56c6c120fdb502f35e.', STARTKEY => '', ENDKEY => '', ENCODED => 6c35fbf061ec3d56c6c120fdb502f35e,}

        Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
        ERROR: Region

        { meta => splitlog,,1346156147925.6c35fbf061ec3d56c6c120fdb502f35e., hdfs => hdfs://10.235.2.121:9000/hbase/splitlog/6c35fbf061ec3d56c6c120fdb502f35e, deployed => }

        not deployed on any region server.
        12/08/28 20:48:51 DEBUG util.HBaseFsck: There are 3 region info entries
        Summary:
        ROOT is okay.
        Number of regions: 1
        Deployed on: ha2,60020,1346158063508
        .META. is okay.
        Number of regions: 1
        Deployed on: ha2,60020,1346158063508
        1 inconsistencies detected.
        Status: INCONSISTENT
        [root@xiaomi hbase]#

        Show
        Liang Xie added a comment - BTW,this hang issue will introduce INCONSISTENT status : 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/ ROOT 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/.META. 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/.logs 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/.oldlogs 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/hbase.id 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region dirs from hdfs://10.235.2.121:9000/hbase/splitlog 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region info from hdfs:hdfs://10.235.2.121:9000/hbase/ ROOT /70236052 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region info from hdfs:hdfs://10.235.2.121:9000/hbase/.META./1028785192 12/08/28 20:48:50 DEBUG util.HBaseFsck: Loading region info from hdfs:hdfs://10.235.2.121:9000/hbase/splitlog/6c35fbf061ec3d56c6c120fdb502f35e 12/08/28 20:48:51 DEBUG util.HBaseFsck: HRegionInfo read: {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,} 12/08/28 20:48:51 DEBUG util.HBaseFsck: HRegionInfo read: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 12/08/28 20:48:51 DEBUG util.HBaseFsck: HRegionInfo read: {NAME => 'splitlog,,1346156147925.6c35fbf061ec3d56c6c120fdb502f35e.', STARTKEY => '', ENDKEY => '', ENCODED => 6c35fbf061ec3d56c6c120fdb502f35e,} Number of empty REGIONINFO_QUALIFIER rows in .META.: 0 ERROR: Region { meta => splitlog,,1346156147925.6c35fbf061ec3d56c6c120fdb502f35e., hdfs => hdfs://10.235.2.121:9000/hbase/splitlog/6c35fbf061ec3d56c6c120fdb502f35e, deployed => } not deployed on any region server. 12/08/28 20:48:51 DEBUG util.HBaseFsck: There are 3 region info entries Summary: ROOT is okay. Number of regions: 1 Deployed on: ha2,60020,1346158063508 .META. is okay. Number of regions: 1 Deployed on: ha2,60020,1346158063508 1 inconsistencies detected. Status: INCONSISTENT [root@xiaomi hbase] #
        Hide
        Liang Xie added a comment -

        After enable hbase-shell debug log, found there's an infinite loop in metaScan method for some special tablenames:

        do {
        ...
                  if (rrs == null || rrs.length == 0 || rrs[0].size() == 0) {
                    break; //exit completely
                  }
        ...
        startRow = callable.getHRegionInfo().getEndKey();
        ...
        } while (Bytes.compareTo(startRow, HConstants.LAST_ROW) != 0);
        

        Maybe a feasible fix is to filter some system-reserved tablenames in HTableDescriptor.isLegalTableName.
        I can try to give a trivial patch if you or other experts agree

        Show
        Liang Xie added a comment - After enable hbase-shell debug log, found there's an infinite loop in metaScan method for some special tablenames: do { ... if (rrs == null || rrs.length == 0 || rrs[0].size() == 0) { break ; //exit completely } ... startRow = callable.getHRegionInfo().getEndKey(); ... } while (Bytes.compareTo(startRow, HConstants.LAST_ROW) != 0); Maybe a feasible fix is to filter some system-reserved tablenames in HTableDescriptor.isLegalTableName. I can try to give a trivial patch if you or other experts agree

          People

          • Assignee:
            Liang Xie
            Reporter:
            Jonathan Hsieh
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development