HBase
  1. HBase
  2. HBASE-1212

merge tool expects regions all have different sequence ids

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently merging two regions, the merge tool will compare their sequence ids. If same, it will decrement one. It needs to do this because on region open, files are keyed by their sequenceid; if two the same, one will erase the other.

      Well, with the move to the aggregating hfile format, the sequenceid is written when the file is created and its no longer written into an aside file but as metadata on to the end of the file. Changing the sequenceid is no longer an option.

      This issue is about figuring a solution for the rare case where two store files have same sequence id AND we want to merge the two regions.

      1. HBASE-1212-v3-trunk.patch
        1 kB
        Jean-Marc Spaggiari
      2. HBASE-1212-v2.patch
        1 kB
        Jean-Marc Spaggiari
      3. HBASE-1212.patch
        1 kB
        Jean-Marc Spaggiari
      4. failure.log
        14 kB
        Jean-Marc Spaggiari

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94 #1165 (See https://builds.apache.org/job/HBase-0.94/1165/)
          HBASE-9504 Backport HBASE-1212 to 0.94 - merge tool expects regions all have different sequence ids (Jean-Marc Spaggiari) (larsh: rev 1528280)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94 #1165 (See https://builds.apache.org/job/HBase-0.94/1165/ ) HBASE-9504 Backport HBASE-1212 to 0.94 - merge tool expects regions all have different sequence ids (Jean-Marc Spaggiari) (larsh: rev 1528280) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94-security #306 (See https://builds.apache.org/job/HBase-0.94-security/306/)
          HBASE-9504 Backport HBASE-1212 to 0.94 - merge tool expects regions all have different sequence ids (Jean-Marc Spaggiari) (larsh: rev 1528280)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94-security #306 (See https://builds.apache.org/job/HBase-0.94-security/306/ ) HBASE-9504 Backport HBASE-1212 to 0.94 - merge tool expects regions all have different sequence ids (Jean-Marc Spaggiari) (larsh: rev 1528280) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          stack added a comment -

          Was integrated a while back. Resolving.

          Show
          stack added a comment - Was integrated a while back. Resolving.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #381 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/381/)
          HBASE-1212 merge tool expects regions all have different sequence ids (Jean-Marc) (Revision 1439495)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #381 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/381/ ) HBASE-1212 merge tool expects regions all have different sequence ids (Jean-Marc) (Revision 1439495) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3814 (See https://builds.apache.org/job/HBase-TRUNK/3814/)
          HBASE-1212 merge tool expects regions all have different sequence ids (Jean-Marc) (Revision 1439495)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3814 (See https://builds.apache.org/job/HBase-TRUNK/3814/ ) HBASE-1212 merge tool expects regions all have different sequence ids (Jean-Marc) (Revision 1439495) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Ted Yu added a comment -

          Integrated to trunk.

          Thanks for the patch, Jean-Marc.

          Thanks for the review, Chunhui.

          Show
          Ted Yu added a comment - Integrated to trunk. Thanks for the patch, Jean-Marc. Thanks for the review, Chunhui.
          Hide
          Jean-Marc Spaggiari added a comment -

          Should I upload one for 0.94?

          Show
          Jean-Marc Spaggiari added a comment - Should I upload one for 0.94?
          Hide
          Ted Yu added a comment -

          +1 from me.

          Show
          Ted Yu added a comment - +1 from me.
          Hide
          chunhui shen added a comment -

          We will use sequenceId to sort HFiles, like in compaction.
          When merging, there is no overlap between these two files from different regions, I think it's ok even if two files have the same sequence ID since we don't use it to distingush the files in Store any more

          Patch looks good for me

          Show
          chunhui shen added a comment - We will use sequenceId to sort HFiles, like in compaction. When merging, there is no overlap between these two files from different regions, I think it's ok even if two files have the same sequence ID since we don't use it to distingush the files in Store any more Patch looks good for me
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12566598/HBASE-1212-v3-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566598/HBASE-1212-v3-trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4191//console This message is automatically generated.
          Hide
          Jean-Marc Spaggiari added a comment -

          Re-submitting the refreshed patch.

          Show
          Jean-Marc Spaggiari added a comment - Re-submitting the refreshed patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12556386/HBASE-1212-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 104 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 23 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12556386/HBASE-1212-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 104 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 23 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3526//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          Change looks good to me. Taking a cursory look at the code we're indeed not using the sequenceId solely to distinguish HFiles.
          Can somebody reconfirm that.

          Show
          Lars Hofhansl added a comment - Change looks good to me. Taking a cursory look at the code we're indeed not using the sequenceId solely to distinguish HFiles. Can somebody reconfirm that.
          Hide
          Jean-Marc Spaggiari added a comment -

          Oh, sorry about that! Did not figured I missed that.

          Debian come with VSN < 1.7 so this command line was not working for me. I installed 1.7, run it and got the patch for all the modifications I made. So attached is the extract for HBASE-1212. Can I mark HBASE-7287 as resolved by this one?

          Show
          Jean-Marc Spaggiari added a comment - Oh, sorry about that! Did not figured I missed that. Debian come with VSN < 1.7 so this command line was not working for me. I installed 1.7, run it and got the patch for all the modifications I made. So attached is the extract for HBASE-1212 . Can I mark HBASE-7287 as resolved by this one?
          Hide
          Ted Yu added a comment -

          From your patch:

          Index: src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          

          In trunk, the path would be hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java

          Can you use 'svn diff' command to produce next patch under the root of your workspace ?

          Show
          Ted Yu added a comment - From your patch: Index: src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java In trunk, the path would be hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Can you use 'svn diff' command to produce next patch under the root of your workspace ?
          Hide
          Jean-Marc Spaggiari added a comment -

          Hi Ted,

          I have already attached the patch. I did it on my eclipse which is configured with the trunk. So it should be correct? Or have I done anything wrong?

          Thanks.

          Show
          Jean-Marc Spaggiari added a comment - Hi Ted, I have already attached the patch. I did it on my eclipse which is configured with the trunk. So it should be correct? Or have I done anything wrong? Thanks.
          Hide
          Ted Yu added a comment -

          Can you attach patch for trunk ?

          Thanks

          Show
          Ted Yu added a comment - Can you attach patch for trunk ? Thanks
          Hide
          Jean-Marc Spaggiari added a comment -

          Patch to remove the sequenceID check.

          Show
          Jean-Marc Spaggiari added a comment - Patch to remove the sequenceID check.
          Hide
          Jean-Marc Spaggiari added a comment -

          Based on Ryan's comment on March 3rd 2011 I did some more testing.

          Here is how I proceed:
          Creation of a test table.
          Truncate the test table if already there and not empty.
          Insert 100000 records
          hbck => ok.
          Split into 256 regions using HTML interface
          Balancer + major_compact
          Try to merge all the regions 2 by 2 until only 1 region remaining or until error.
          If no error, restart from split. (Usually need to do it twice)

          Now, at this stage, I have 2 regions with the same sequence ID that I'm not able to merge with the current un-patched HRegion code.

          Displayed:starting merge of regions: testtable,?\xEC\x80C^\xB2Q\xE8,1354799289826.eb1b2213a6b09a9d989c39abe3d6d18b. and testtable,?\xEC\xF9\x88\xBE{Sb,1354799292172.2d81e98f11bdda12dbf06c11eefefb39. into new region

          {NAME => 'testtable,?\xEC\x80C^\xB2Q\xE8,1354799434888.e4e403c2bf3b3b2561f26bfb70ef48d1.', STARTKEY => '?\xEC\x80C^\xB2Q\xE8', ENDKEY => '?\xEDw\xF5\x16\xC4\x81\xB7', ENCODED => e4e403c2bf3b3b2561f26bfb70ef48d1,}

          with start key <?\xEC\x80C^\xB2Q\xE8> and end key <?\xEDw\xF5\x16\xC4\x81\xB7>

          Start-hbase, hbck => One issue on the newly created region (HBASE-7287 e4e403c2bf3b3b2561f26bfb70ef48d)
          rowcount => 12/12/06 08:15:14 INFO mapred.JobClient: ROWS=100000

          Apply HBASE-1212 patch, stop HBase and restart the merge.
          Merge went well up to the end (all regions merged 2 by 2 until only 1 region remaining)
          Start-hbase, hbck => One issue on the newly created region (HBASE-7287 e4e403c2bf3b3b2561f26bfb70ef48d) Same as before, so no additionnal issue because of HBASE-1212.
          rowcount => 12/12/06 08:21:54 INFO mapred.JobClient: ROWS=100000

          The issue created by the merge when not patched (HBASE-7287 or HBASE-1212) is just an empty directory created which can be manually removed.

          If we apply HBASE-1212, then there is no more needs to apply HBASE-7287. If we want to make more tests for HBASE-1212 then it might be good to apply HBASE-7287 to avoid system inconsistencies.

          Based on my tests I can confirm that we can remove the sequenceID check and the merge will still be working fine even if 2 files got the same sequenceID.

          Show
          Jean-Marc Spaggiari added a comment - Based on Ryan's comment on March 3rd 2011 I did some more testing. Here is how I proceed: Creation of a test table. Truncate the test table if already there and not empty. Insert 100000 records hbck => ok. Split into 256 regions using HTML interface Balancer + major_compact Try to merge all the regions 2 by 2 until only 1 region remaining or until error. If no error, restart from split. (Usually need to do it twice) Now, at this stage, I have 2 regions with the same sequence ID that I'm not able to merge with the current un-patched HRegion code. Displayed:starting merge of regions: testtable,?\xEC\x80C^\xB2Q\xE8,1354799289826.eb1b2213a6b09a9d989c39abe3d6d18b. and testtable,?\xEC\xF9\x88\xBE{Sb,1354799292172.2d81e98f11bdda12dbf06c11eefefb39. into new region {NAME => 'testtable,?\xEC\x80C^\xB2Q\xE8,1354799434888.e4e403c2bf3b3b2561f26bfb70ef48d1.', STARTKEY => '?\xEC\x80C^\xB2Q\xE8', ENDKEY => '?\xEDw\xF5\x16\xC4\x81\xB7', ENCODED => e4e403c2bf3b3b2561f26bfb70ef48d1,} with start key <?\xEC\x80C^\xB2Q\xE8> and end key <?\xEDw\xF5\x16\xC4\x81\xB7> Start-hbase, hbck => One issue on the newly created region ( HBASE-7287 e4e403c2bf3b3b2561f26bfb70ef48d) rowcount => 12/12/06 08:15:14 INFO mapred.JobClient: ROWS=100000 Apply HBASE-1212 patch, stop HBase and restart the merge. Merge went well up to the end (all regions merged 2 by 2 until only 1 region remaining) Start-hbase, hbck => One issue on the newly created region ( HBASE-7287 e4e403c2bf3b3b2561f26bfb70ef48d) Same as before, so no additionnal issue because of HBASE-1212 . rowcount => 12/12/06 08:21:54 INFO mapred.JobClient: ROWS=100000 The issue created by the merge when not patched ( HBASE-7287 or HBASE-1212 ) is just an empty directory created which can be manually removed. If we apply HBASE-1212 , then there is no more needs to apply HBASE-7287 . If we want to make more tests for HBASE-1212 then it might be good to apply HBASE-7287 to avoid system inconsistencies. Based on my tests I can confirm that we can remove the sequenceID check and the merge will still be working fine even if 2 files got the same sequenceID.
          Hide
          Jean-Marc Spaggiari added a comment -

          Attached logs from a merge failure.

          Show
          Jean-Marc Spaggiari added a comment - Attached logs from a merge failure.
          Hide
          Jean-Marc Spaggiari added a comment -

          FYI, I'm facing this issue today while trying to merge all the regions of a table. I have created a 54 regions table, and while trying to merge all of them back into a single one I got this error. I'm able to reproduce it.

          Show
          Jean-Marc Spaggiari added a comment - FYI, I'm facing this issue today while trying to merge all the regions of a table. I have created a 54 regions table, and while trying to merge all of them back into a single one I got this error. I'm able to reproduce it.
          Hide
          Lukas added a comment -

          Can someone comment, if this is still an issue?

          Show
          Lukas added a comment - Can someone comment, if this is still an issue?
          Hide
          ryan rawson added a comment -

          now that we dont key files by their sequence id in Store.java, this is simpler: just allow duplicate sequence ids.

          Show
          ryan rawson added a comment - now that we dont key files by their sequence id in Store.java, this is simpler: just allow duplicate sequence ids.
          Hide
          stack added a comment -

          This is worse now that bulk upload is popular; all get low seqid so likelihood of clashing more likely.

          Made it so I had to do manual merge doing fixup of user cluster.

          Show
          stack added a comment - This is worse now that bulk upload is popular; all get low seqid so likelihood of clashing more likely. Made it so I had to do manual merge doing fixup of user cluster.
          Hide
          Lukas added a comment -

          Hi there,
          When using util.merge on a table (created under hbase 0.20.6, but now running under 0.90.1) I get the following exception very often (which I think is the result of this jira):

           
          FATAL util.Merge: Merge failed
          java.io.IOException: Files have same sequenceid: 1283640296746
          	at org.apache.hadoop.hbase.regionserver.HRegion.merge(HRegion.java:2785)
          	at org.apache.hadoop.hbase.util.Merge.merge(Merge.java:287)
          	at org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:238)
          	at org.apache.hadoop.hbase.util.Merge.run(Merge.java:110)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
          	at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
          

          Unfortunately at this point some families of the region were already moved to the new merged region which leaves the table in an inconsistent state.
          Is there any way to detect this failure case before running the merge tool (except for opening the store files in advance and calling getMaxSequenceId)?

          Same issue is also mentioned in the comments of HBASE-1621

          Show
          Lukas added a comment - Hi there, When using util.merge on a table (created under hbase 0.20.6, but now running under 0.90.1) I get the following exception very often (which I think is the result of this jira): FATAL util.Merge: Merge failed java.io.IOException: Files have same sequenceid: 1283640296746 at org.apache.hadoop.hbase.regionserver.HRegion.merge(HRegion.java:2785) at org.apache.hadoop.hbase.util.Merge.merge(Merge.java:287) at org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:238) at org.apache.hadoop.hbase.util.Merge.run(Merge.java:110) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379) Unfortunately at this point some families of the region were already moved to the new merged region which leaves the table in an inconsistent state. Is there any way to detect this failure case before running the merge tool (except for opening the store files in advance and calling getMaxSequenceId)? Same issue is also mentioned in the comments of HBASE-1621
          Hide
          stack added a comment -

          Thinking on it, this event should be extremely rare. Sequence ids are monotonically increasing in a running regionserver. Across a cluster, two files of the same family would have to end with same sequenceid. Then whats the likelihood that of all regions on cluster these are the two to merge (Merge is a little-used tool to date).

          To fix, would need to look at the content of the two files and make a judgement as to which should come before the other – which has the most recent edits. Maybe we could do something basic like let the file with the largest size prevail over the smaller. Once we'd figure which file to bring to the fore, we need to rewrite the hfile so we can change the sequence id. Since we're rewriting one of the files at least, might as well compact them.

          We could move to modification times. That should simplify this sequenceid story. It wouldn't remove this issue. We'd still have to figure which store file to favor if two happened to have same mod time.

          In bigtable, chubby owns the storefiles/sstables. Maybe thats where we should go so we don't have sequenceids anymore?

          Moving out of 0.20.0 because this issue rare and amount of work to address is large.

          Show
          stack added a comment - Thinking on it, this event should be extremely rare. Sequence ids are monotonically increasing in a running regionserver. Across a cluster, two files of the same family would have to end with same sequenceid. Then whats the likelihood that of all regions on cluster these are the two to merge (Merge is a little-used tool to date). To fix, would need to look at the content of the two files and make a judgement as to which should come before the other – which has the most recent edits. Maybe we could do something basic like let the file with the largest size prevail over the smaller. Once we'd figure which file to bring to the fore, we need to rewrite the hfile so we can change the sequence id. Since we're rewriting one of the files at least, might as well compact them. We could move to modification times. That should simplify this sequenceid story. It wouldn't remove this issue. We'd still have to figure which store file to favor if two happened to have same mod time. In bigtable, chubby owns the storefiles/sstables. Maybe thats where we should go so we don't have sequenceids anymore? Moving out of 0.20.0 because this issue rare and amount of work to address is large.
          Hide
          stack added a comment -

          I like the jgray idea that we use modtime instead of an edit number.

          Show
          stack added a comment - I like the jgray idea that we use modtime instead of an edit number.
          Hide
          stack added a comment -

          Moving to 0.20.0

          Show
          stack added a comment - Moving to 0.20.0
          Hide
          Nitay Joffe added a comment -

          As I noted on HBASE-1274, we can try using the file names to do the sequencing:

          What about using something that sorts lexicographically and sorting the dir.listFiles we use when grabbing the store files? Then when we move around two files with the same sequence ids to do a merge we can rename the actual files.

          Show
          Nitay Joffe added a comment - As I noted on HBASE-1274 , we can try using the file names to do the sequencing: What about using something that sorts lexicographically and sorting the dir.listFiles we use when grabbing the store files? Then when we move around two files with the same sequence ids to do a merge we can rename the actual files.

            People

            • Assignee:
              Jean-Marc Spaggiari
              Reporter:
              stack
            • Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development