HBase
  1. HBase
  2. HBASE-7287

HBase inconsistencies when merge failing on "Files have same sequenceid"

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When regions merge is failing because the files have the same sequenceID, the expected region is still created even if it's not used, which leaves the system with inconsistencies. The new region creation should be moved after the sequenceID test to avoid this issue, until we find a way to merge regions with the same sequenceID.

      1. HBASE-7287.patch
        0.6 kB
        Jean-Marc Spaggiari
      2. HBASE-7287-v2.patch
        0.7 kB
        Jean-Marc Spaggiari
      3. HBASE-7287-v3-trunk.patch
        0.8 kB
        Jean-Marc Spaggiari
      4. HBASE-7287-v4-trunk.patch
        0.8 kB
        Jean-Marc Spaggiari

        Activity

        Hide
        Jean-Marc Spaggiari added a comment -

        Since HBASE-1212 will remove this part of the code, there is no more need to have this fixed.

        Show
        Jean-Marc Spaggiari added a comment - Since HBASE-1212 will remove this part of the code, there is no more need to have this fixed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12563089/HBASE-7287-v4-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.hfile.TestHFileBlock
        org.apache.hadoop.hbase.coprocessor.TestWALObserver
        org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
        org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
        org.apache.hadoop.hbase.io.TestFileLink
        org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
        org.apache.hadoop.hbase.TestLocalHBaseCluster
        org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
        org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk
        org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers
        org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove
        org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
        org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
        org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
        org.apache.hadoop.hbase.coprocessor.TestMasterObserver
        org.apache.hadoop.hbase.replication.TestMasterReplication
        org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass
        org.apache.hadoop.hbase.coprocessor.TestClassLoading
        org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove
        org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
        org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
        org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
        org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
        org.apache.hadoop.hbase.TestGlobalMemStoreSize
        org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader
        org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite

        -1 core zombie tests. There are 12 zombie test(s): at org.apache.hadoop.hbase.master.TestOpenedRegionHandler.testOpenedRegionHandlerOnMasterRestart(TestOpenedRegionHandler.java:104)
        at org.apache.hadoop.hbase.mapreduce.TestWALPlayer.testWALPlayer(TestWALPlayer.java:94)
        at org.apache.hadoop.hbase.trace.TestHTraceHooks.testTraceCreateTable(TestHTraceHooks.java:66)
        at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeAllFromMinorCompaction(TestHFileOutputFormat.java:749)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563089/HBASE-7287-v4-trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.coprocessor.TestWALObserver org.apache.hadoop.hbase.replication.TestMultiSlaveReplication org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery org.apache.hadoop.hbase.io.TestFileLink org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort org.apache.hadoop.hbase.TestLocalHBaseCluster org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk org.apache.hadoop.hbase.mapreduce.TestTableMapReduce org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.replication.TestMasterReplication org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass org.apache.hadoop.hbase.coprocessor.TestClassLoading org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed org.apache.hadoop.hbase.io.encoding.TestChangingEncoding org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint org.apache.hadoop.hbase.TestGlobalMemStoreSize org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite -1 core zombie tests . There are 12 zombie test(s): at org.apache.hadoop.hbase.master.TestOpenedRegionHandler.testOpenedRegionHandlerOnMasterRestart(TestOpenedRegionHandler.java:104) at org.apache.hadoop.hbase.mapreduce.TestWALPlayer.testWALPlayer(TestWALPlayer.java:94) at org.apache.hadoop.hbase.trace.TestHTraceHooks.testTraceCreateTable(TestHTraceHooks.java:66) at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeAllFromMinorCompaction(TestHFileOutputFormat.java:749) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3825//console This message is automatically generated.
        Hide
        Jean-Marc Spaggiari added a comment -

        Re-submitting.

        Show
        Jean-Marc Spaggiari added a comment - Re-submitting.
        Hide
        Jean-Marc Spaggiari added a comment -

        Need to re-commit for trunk.

        Show
        Jean-Marc Spaggiari added a comment - Need to re-commit for trunk.
        Hide
        Jean-Marc Spaggiari added a comment -

        Attaching patch for trunk too.

        Show
        Jean-Marc Spaggiari added a comment - Attaching patch for trunk too.
        Hide
        Jean-Marc Spaggiari added a comment -

        Figured the patch format was not correct...

        Is it possible to push that until we get HBASE-1212 confirmed and pushed too? This will avoid inconsistencies on the system when merges are failing...

        Show
        Jean-Marc Spaggiari added a comment - Figured the patch format was not correct... Is it possible to push that until we get HBASE-1212 confirmed and pushed too? This will avoid inconsistencies on the system when merges are failing...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12556182/HBASE-7287.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3448//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12556182/HBASE-7287.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3448//console This message is automatically generated.
        Hide
        Jean-Marc Spaggiari added a comment -

        If HBASE-1212 is accepted, then this patch is no more required since HBASE-1212 is totally removing the sequenceID check this patch is fixing.

        Show
        Jean-Marc Spaggiari added a comment - If HBASE-1212 is accepted, then this patch is no more required since HBASE-1212 is totally removing the sequenceID check this patch is fixing.
        Hide
        Jean-Marc Spaggiari added a comment -

        Patch attached.

        Show
        Jean-Marc Spaggiari added a comment - Patch attached.
        Hide
        Jean-Marc Spaggiari added a comment -

        Finally, it's not possible to move the newRegionDir creation after the sequenceID check since some sequenceID for some Familly files might be correct and will need to create the new column family dirs (makeColumnFamilyDirs). So instead of moving the directory creation, here is a patch to remove the created directory if the merge is failing, to avoid system inconsistencies.

        Show
        Jean-Marc Spaggiari added a comment - Finally, it's not possible to move the newRegionDir creation after the sequenceID check since some sequenceID for some Familly files might be correct and will need to create the new column family dirs (makeColumnFamilyDirs). So instead of moving the directory creation, here is a patch to remove the created directory if the merge is failing, to avoid system inconsistencies.

          People

          • Assignee:
            Jean-Marc Spaggiari
            Reporter:
            Jean-Marc Spaggiari
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 1h
              1h
              Remaining:
              Remaining Estimate - 1h
              1h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development