Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-10349

Table became unusable when master balanced its region after table was dropped

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.0
    • Fix Version/s: 0.98.0, 0.96.2, 0.99.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      0.98 was used.
      This was sequence of events:

      create 'tablethree_mod'
      snapshot 'tablethree_mod', 'snapshot_tablethree_mod'
      disable 'tablethree_mod'
      2014-01-15 09:34:51,749 restore_snapshot 'snapshot_tablethree_mod'
      2014-01-15 09:35:07,210 enable 'tablethree_mod'
      2014-01-15 09:35:46,134 delete_snapshot 'snapshot_tablethree_mod'
      2014-01-15 09:41:42,210 disable 'tablethree_mod'
      2014-01-15 09:41:43,610 drop 'tablethree_mod'
      create 'tablethree_mod'

      For the last table creation request:

       
      2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'create 'tablethree_mod',
      {NAME => 'f1', VERSIONS => 3}
      
      ,
      {NAME => 'f2', VERSIONS => 3}
      
      ,
      {NAME => 'f3', VERSIONS => 3}
      
      '
      2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'exists 'tablethree_mod''
      2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod', '0', 'f1:q1', 'value-0', 10'
      2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod', '1', 'f1:q1', 'value-1', 20'
      2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '2', 'f2:q2', 'value-2', 30'
      2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '3', 'f3:q3', 'value-3', 40'
      2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '4', 'f3:q3', 'value-4', 50'
      2014-01-15 10:03:53,000|beaver.component.hbase|INFO|Done writing commands to file. Will execute them now.
      2014-01-15 10:03:53,000|beaver.machine|INFO|RUNNING: /usr/lib/hbase/bin/hbase shell /grid/0/tmp/hwqe/artifacts/tmp-471142
      2014-01-15 10:03:55,878|beaver.machine|INFO|2014-01-15 10:03:55,878 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
      2014-01-15 10:03:57,283|beaver.machine|INFO|2014-01-15 10:03:57,283 WARN [main] conf.Configuration: hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
      2014-01-15 10:03:57,669|beaver.machine|INFO|2014-01-15 10:03:57,669 WARN [main] conf.Configuration: hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
      2014-01-15 10:03:57,720|beaver.machine|INFO|2014-01-15 10:03:57,720 WARN [main] conf.Configuration: hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
      2014-01-15 10:03:57,997|beaver.machine|INFO|
      2014-01-15 10:03:57,997|beaver.machine|INFO|ERROR: Table already exists: tablethree_mod!
      2014-01-15 10:03:57,997|beaver.machine|INFO|
      

      This was an intermittent issue after using Snapshots, a table is not properly dropped / and not able to properly re-create with the same name. And a HRegion is empty or null Error occurs. (When you try to drop the table it says it does not exist, and when you try to create the table it says that it does already exist).

      2014-01-15 10:04:02,462|beaver.machine|INFO|ERROR: HRegionInfo was null or empty in hbase:meta, row=keyvalues=
      {tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:seqnumDuringOpen/1389778905355/Put/vlen=8/mvcc=0, tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:server/1389778905355/Put/vlen=32/mvcc=0, tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:serverstartcode/1389778905355/Put/vlen=8/mvcc=0} 
      

      Thanks to Huned who discovered this issue.

        Attachments

        1. hbase-hbase-master-hor15n05.gq1.ygridcore.net.log.tar.gz
          2.23 MB
          Ted Yu
        2. HBASE-10349-meta-test-and-debug.patch
          2 kB
          Matteo Bertozzi
        3. hbase-10349.patch
          8 kB
          Jimmy Xiang
        4. hbase-10349_v2.patch
          9 kB
          Jimmy Xiang
        5. 10349-v2.txt
          3 kB
          Ted Yu
        6. 10349-v1.txt
          0.9 kB
          Ted Yu
        7. 10349-output.log
          4.72 MB
          Ted Yu
        8. 10349-hadoop-hdfs-namenode-hor11n14.gq1.ygridcore.net.zip
          800 kB
          Ted Yu

        Issue Links

          Activity

            People

            • Assignee:
              jxiang Jimmy Xiang
              Reporter:
              yuzhihong@gmail.com Ted Yu

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment