Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19331

Region start-key/end-key corruption in Hbase meta table

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 0.98.8
    • Fix Version/s: None
    • Component/s: Region Assignment
    • Labels:
      None
    • Environment:

      Reproduced on HBase 0.98.8 on hadoop-2

      Description

      when a region split happens on a key with trailing byte equals zero, the end key of the first resulting region and and start key of the second resulting region in meta table gets corrupted.

      Here is the link to code to reproduce this issue
      https://bitbucket.org/flytxt/hbase-meta-corruption-test

      Test Result

      [INFO] -------------------------------------------------------
      [INFO] T E S T S
      [INFO] -------------------------------------------------------
      [INFO] Running com.flytxt.HbaseRegionMetaTest
      log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
      log4j:WARN Please initialize the log4j system properly.
      log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      18:23:54.346 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table SAMPLE_TBL_1
      18:23:56.094 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table SAMPLE_TBL_2
      18:23:58.107 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table SAMPLE_TBL_1
      18:23:58.658 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table SAMPLE_TBL_1
      18:23:59.212 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to table SAMPLE_TBL_1
      18:24:00.046 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete .. lets split SAMPLE_TBL_1
      18:24:00.500 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to table SAMPLE_TBL_2
      18:24:02.073 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete .. lets split SAMPLE_TBL_2
      18:24:02.753 [main] INFO com.flytxt.HbaseRegionMetaTest - region split complete .. Lets verify region infos for table SAMPLE_TBL_1
      18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest - ===========================================================
      18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : SAMPLE_TBL_1,,1511355240515.56c8fd8e42228c3c1ec71f9a4da65f5f.
      18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id :1511355240515
      18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key :[] , Key length :0
      18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : [0, 0, 0, 0, 0, 19, -76] , Key length :7
      18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - -----------------------------------------------------------
      18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - ===========================================================
      18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : SAMPLE_TBL_1,\x00\x00\x00\x00\x00\x13\xB4,1511355240515.c06afed17b2a5c4fb54bacf704dd8a9e.
      18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id :1511355240515
      18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key :[0, 0, 0, 0, 0, 19, -76] , Key length :7
      18:24:02.763 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : [] , Key length :0
      18:24:02.763 [main] INFO com.flytxt.HbaseRegionMetaTest - -----------------------------------------------------------
      18:24:03.005 [main] INFO com.flytxt.HbaseRegionMetaTest - region split complete .. Lets verify region infos for table SAMPLE_TBL_2
      18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - ===========================================================
      18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : SAMPLE_TBL_2,,1511355242363.0679851100e16aad005c743af618452e.
      18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id :1511355242363
      18:24:03.007 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key :[] , Key length :0
      18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : [0, 0, 0, 0, 0, 0, 19, -57] , Key length :8
      18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - -----------------------------------------------------------
      18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - ===========================================================
      18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : SAMPLE_TBL_2,\x00\x00\x00\x00\x00\x00\x13\xC7,1511355242363.326b5175036efdeae89f9493b436e114.
      18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id :1511355242363
      18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key :[0, 0, 0, 0, 0, 0, 19, -57] , Key length :8
      18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : [] , Key length :0
      18:24:03.009 [main] INFO com.flytxt.HbaseRegionMetaTest - -----------------------------------------------------------
      [ERROR] Tests run: 3, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 10.777 s <<< FAILURE! - in com.flytxt.HbaseRegionMetaTest
      [ERROR] testTableRegionStartAndEndKey[0](com.flytxt.HbaseRegionMetaTest) Time elapsed: 0.228 s <<< FAILURE!
      java.lang.AssertionError:
      Region end key error for SAMPLE_TBL_1,,1511355240515.56c8fd8e42228c3c1ec71f9a4da65f5f.
      Expected: is <8>
      but: was <7>
      at com.flytxt.HbaseRegionMetaTest.testTableRegionStartAndEndKey(HbaseRegionMetaTest.java:170)

      [ERROR] testTableRegionStartAndEndKey[0](com.flytxt.HbaseRegionMetaTest) Time elapsed: 0.23 s <<< FAILURE!
      java.lang.AssertionError:
      Region start key error for SAMPLE_TBL_1,\x00\x00\x00\x00\x00\x13\xB4,1511355240515.c06afed17b2a5c4fb54bacf704dd8a9e.
      Expected: is <8>
      but: was <7>
      at com.flytxt.HbaseRegionMetaTest.testTableRegionStartAndEndKey(HbaseRegionMetaTest.java:167)

      [INFO]
      [INFO] Results:
      [INFO]
      [ERROR] Failures:
      [ERROR] com.flytxt.HbaseRegionMetaTest.testTableRegionStartAndEndKey[0](com.flytxt.HbaseRegionMetaTest)
      [ERROR] Run 1: HbaseRegionMetaTest.testTableRegionStartAndEndKey:170 Region end key error for SAMPLE_TBL_1,,1511355240515.56c8fd8e42228c3c1ec71f9a4da65f5f.
      Expected: is <8>
      but: was <7>
      [ERROR] Run 2: HbaseRegionMetaTest.testTableRegionStartAndEndKey:167 Region start key error for SAMPLE_TBL_1,\x00\x00\x00\x00\x00\x13\xB4,1511355240515.c06afed17b2a5c4fb54bacf704dd8a9e.
      Expected: is <8>
      but: was <7>
      [INFO]
      [INFO]
      [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0

        Attachments

        1. TestSplit.java
          8 kB
          Ted Yu

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              shamithmc Shamith kumar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: