HBase
  1. HBase
  2. HBASE-6160

META entries from daughters can be deleted before parent entries

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.2, 0.94.0, 0.95.2
    • Fix Version/s: 0.92.2, 0.94.1
    • Component/s: Client, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:

      Region A is split to -> B, C
      Region B is split to -> D, E

      After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.

      1. HBASE-6160v2092.txt
        8 kB
        stack
      2. HBASE-6160_v2.patch
        8 kB
        stack
      3. HBASE-6160_v2.patch
        8 kB
        Enis Soztutar
      4. HBASE-6160_v1.patch
        8 kB
        Enis Soztutar

        Activity

        Hide
        Enis Soztutar added a comment -

        One option for a fix is to ensure that META entry for the parent region is deleted before deleting the META entry, and do the META entry deletion recursively.

        Show
        Enis Soztutar added a comment - One option for a fix is to ensure that META entry for the parent region is deleted before deleting the META entry, and do the META entry deletion recursively.
        Hide
        Ted Yu added a comment - - edited

        @Enis:
        Can you attach log snippets to show the problem ?
        e.g. what was the interval between the two splits.

        Show
        Ted Yu added a comment - - edited @Enis: Can you attach log snippets to show the problem ? e.g. what was the interval between the two splits.
        Hide
        Enis Soztutar added a comment -

        The exception:

        12/06/04 06:50:41 ERROR security.UserGroupInformation: PriviledgedActionException as: cause:org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. cannot be found in META.
        Exception in thread "main" org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. cannot be found in META.
                at org.apache.hadoop.hbase.client.MetaScanner$BlockingMetaScannerVisitor.processRow(MetaScanner.java:433)
                at org.apache.hadoop.hbase.client.MetaScanner$TableMetaScannerVisitor.processRow(MetaScanner.java:490)
                at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:227)
                at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:57)
                at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:136)
                at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
                at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:361)
                at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133)
                at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:108)
                at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:86)
                at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:326)
                at org.apache.hadoop.hbase.client.HTable.getRegionLocations(HTable.java:499)
                at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:452)
                at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:132)
                at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
                at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
        
        

        So the region in question is

        50a4617eead34cad335a8dfa727d177d
        

        and from the logs we see that 25d9c4ff574a37bd95bf5e5be6d618dd is split into 1dc74065583c67b3916c4ed158cb53fa and 50a4617eead34cad335a8dfa727d177d

        ./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 04:56:02,855 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated, and report to master. Parent=TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd., new regions: TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa., TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.. Split took 4sec
        

        After some time, 50a4617eead34cad335a8dfa727d177d is further split into two:

        ./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 05:41:13,488 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated, and report to master. Parent=TestLoadAndVerify_1
        338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d., new regions: TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338802866393.16288
        65d7fa8e9eec3a7d8073465296e., TestLoadAndVerify_1338798130970,]y\x04\x00\x00\x00\x00\x00/000047_0,1338802866393.413cafe6c61426e26254c197e8c0a6ba.. Split took 7sec
        

        Further time passes, and CatalogJanitor deletes the META entry for that region:

        ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:16,688 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Deleting region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. because daughter splits no longer hold references
        ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughters references, qualifier=splitA and qualifier=splitB, from parent TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
        ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d
        ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,145 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. from META
        ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:52:12,891 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Daughter regiondir does not exist: hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d
        

        However, there is no log for the grantfather,25d9c4ff574a37bd95bf5e5be6d618dd, being deleted from META. In fact, META scan also confirms that the grandfather is in META, but not the father:

         TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:regioninfo, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.', STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl', ENCODED => 25d9c4ff574a37bd95bf5e5be6d618dd, OFFLINE => true, SPLIT => true,}
         TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:server, timestamp=1338799025521, value=ip-10-226-65-102.ec2.internal:60020
         TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:serverstartcode, timestamp=1338799025521, value=1338780117340
         TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:splitA, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa.', STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENCODED => 1dc74065583c67b3916c4ed158cb53fa,}
         TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:splitB, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.', STARTKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl', ENCODED => 50a4617eead34cad335a8dfa727d177d,}
        
        Show
        Enis Soztutar added a comment - The exception: 12/06/04 06:50:41 ERROR security.UserGroupInformation: PriviledgedActionException as: cause:org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. cannot be found in META. Exception in thread "main" org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. cannot be found in META. at org.apache.hadoop.hbase.client.MetaScanner$BlockingMetaScannerVisitor.processRow(MetaScanner.java:433) at org.apache.hadoop.hbase.client.MetaScanner$TableMetaScannerVisitor.processRow(MetaScanner.java:490) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:227) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:57) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:136) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:361) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:108) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:86) at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:326) at org.apache.hadoop.hbase.client.HTable.getRegionLocations(HTable.java:499) at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:452) at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:132) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979) So the region in question is 50a4617eead34cad335a8dfa727d177d and from the logs we see that 25d9c4ff574a37bd95bf5e5be6d618dd is split into 1dc74065583c67b3916c4ed158cb53fa and 50a4617eead34cad335a8dfa727d177d ./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 04:56:02,855 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated, and report to master. Parent=TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd., new regions: TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa., TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.. Split took 4sec After some time, 50a4617eead34cad335a8dfa727d177d is further split into two: ./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 05:41:13,488 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated, and report to master. Parent=TestLoadAndVerify_1 338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d., new regions: TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338802866393.16288 65d7fa8e9eec3a7d8073465296e., TestLoadAndVerify_1338798130970,]y\x04\x00\x00\x00\x00\x00/000047_0,1338802866393.413cafe6c61426e26254c197e8c0a6ba.. Split took 7sec Further time passes, and CatalogJanitor deletes the META entry for that region: ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:16,688 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Deleting region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. because daughter splits no longer hold references ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughters references, qualifier=splitA and qualifier=splitB, from parent TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs: //ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,145 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. from META ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:52:12,891 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Daughter regiondir does not exist: hdfs: //ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d However, there is no log for the grantfather,25d9c4ff574a37bd95bf5e5be6d618dd, being deleted from META. In fact, META scan also confirms that the grandfather is in META, but not the father: TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:regioninfo, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.', STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl', ENCODED => 25d9c4ff574a37bd95bf5e5be6d618dd, OFFLINE => true , SPLIT => true ,} TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:server, timestamp=1338799025521, value=ip-10-226-65-102.ec2.internal:60020 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:serverstartcode, timestamp=1338799025521, value=1338780117340 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:splitA, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa.', STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENCODED => 1dc74065583c67b3916c4ed158cb53fa,} TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:splitB, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.', STARTKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl', ENCODED => 50a4617eead34cad335a8dfa727d177d,}
        Hide
        Enis Soztutar added a comment -

        Attaching a patch for trunk.

        • Changes CatalogJanitor to not delete split parents, whose parents are still in META.
        • Adds a test case
        Show
        Enis Soztutar added a comment - Attaching a patch for trunk. Changes CatalogJanitor to not delete split parents, whose parents are still in META. Adds a test case
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12530889/HBASE-6160_v1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530889/HBASE-6160_v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//console This message is automatically generated.
        Hide
        stack added a comment -

        Weird. I thought this had been fixed previous. Patch looks good though Enis and nice test. Will commit unless objection in morning.

        Show
        stack added a comment - Weird. I thought this had been fixed previous. Patch looks good though Enis and nice test. Will commit unless objection in morning.
        Hide
        Ted Yu added a comment -

        Minor comments:

        +   * Scans META ane returns a pair of number of scanned rows, and
        

        'ane' -> 'and'
        I think 'a pair of' is not needed above.

        +        //we could not clean the parent, so it's daughters should not be cleaned as well (HBASE-6160)
        

        'as well' -> 'either'

        Show
        Ted Yu added a comment - Minor comments: + * Scans META ane returns a pair of number of scanned rows, and 'ane' -> 'and' I think 'a pair of' is not needed above. + //we could not clean the parent, so it's daughters should not be cleaned as well (HBASE-6160) 'as well' -> 'either'
        Hide
        ramkrishna.s.vasudevan added a comment - - edited

        @Enis
        The great grandfather should have ideally been compacted right, so that it does not have reference files, because he was the one who got splitted first. But the father's reference files got cleared/compacted thus leading to this problem. Correct me if am wrong?
        @Stack
        Do we still need to prioritize the reference files so that compaction of ref files surely happen? So i think after this patch still the reference file of the great grand father may not get compacted and we may never allow the CJ to remove the entry from META? Correct me if am wrong Stack?

        Show
        ramkrishna.s.vasudevan added a comment - - edited @Enis The great grandfather should have ideally been compacted right, so that it does not have reference files, because he was the one who got splitted first. But the father's reference files got cleared/compacted thus leading to this problem. Correct me if am wrong? @Stack Do we still need to prioritize the reference files so that compaction of ref files surely happen? So i think after this patch still the reference file of the great grand father may not get compacted and we may never allow the CJ to remove the entry from META? Correct me if am wrong Stack?
        Hide
        Enis Soztutar added a comment -

        @Ramkrishna
        yes, ideally that is the case. But we may end up with this, if for example, the regions are non-uniform. I think we still have to prioritize the ref files in compaction, since they also prevent splitting further.

        Show
        Enis Soztutar added a comment - @Ramkrishna yes, ideally that is the case. But we may end up with this, if for example, the regions are non-uniform. I think we still have to prioritize the ref files in compaction, since they also prevent splitting further.
        Hide
        ramkrishna.s.vasudevan added a comment -

        HBASE-5161 is already there for prioritizing reference files. I had reopened it already some time back.

        Show
        ramkrishna.s.vasudevan added a comment - HBASE-5161 is already there for prioritizing reference files. I had reopened it already some time back.
        Hide
        stack added a comment -

        @Ram Yes, the father was split and his children subsequently compacted and let go of references so he was removed but his no-good brother was still around... with references probably so the ancestor could not yet be cleared. The RegionOfflineException is because of the fix over in HBASE-5986.

        Yes, we still need to prioritize references files. I'd think that the no-good brother would eventually compact away his references making the great grandfather removable. Why would this patch change compactions or the rate at which regions are removed from .META. Ram? (I can see removals perhaps going a little slower, more lumpy, but otherwise, should be as it is now)?

        Show
        stack added a comment - @Ram Yes, the father was split and his children subsequently compacted and let go of references so he was removed but his no-good brother was still around... with references probably so the ancestor could not yet be cleared. The RegionOfflineException is because of the fix over in HBASE-5986 . Yes, we still need to prioritize references files. I'd think that the no-good brother would eventually compact away his references making the great grandfather removable. Why would this patch change compactions or the rate at which regions are removed from .META. Ram? (I can see removals perhaps going a little slower, more lumpy, but otherwise, should be as it is now)?
        Hide
        Enis Soztutar added a comment -

        v2 patch addressing Ted's comments.

        Show
        Enis Soztutar added a comment - v2 patch addressing Ted's comments.
        Hide
        stack added a comment -

        What I applied to trunk (has Ted's suggested fixes)

        Show
        stack added a comment - What I applied to trunk (has Ted's suggested fixes)
        Hide
        stack added a comment -

        Here is patch I applied to 0.92 and 0.94 branches.

        Show
        stack added a comment - Here is patch I applied to 0.92 and 0.94 branches.
        Hide
        stack added a comment -

        Committed to trunk, 0.92, and 0.94 branches. Thanks for the patch Enis.

        Show
        stack added a comment - Committed to trunk, 0.92, and 0.94 branches. Thanks for the patch Enis.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Thanks Enis and Stack for your reply.

        Yes, we still need to prioritize references files

        I just wanted to say we need to do the prioritizing(along with this). In some cases that we saw the region kept on growing.

        Show
        ramkrishna.s.vasudevan added a comment - Thanks Enis and Stack for your reply. Yes, we still need to prioritize references files I just wanted to say we need to do the prioritizing(along with this). In some cases that we saw the region kept on growing.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2990 (See https://builds.apache.org/job/HBase-TRUNK/2990/)
        HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346496)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2990 (See https://builds.apache.org/job/HBase-TRUNK/2990/ ) HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346496) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        stack added a comment -

        In some cases that we saw the region kept on growing.

        Yeah, that can happen. Usually though because of other issues; a write rate in excess of compaction rate, etc.

        Show
        stack added a comment - In some cases that we saw the region kept on growing. Yeah, that can happen. Usually though because of other issues; a write rate in excess of compaction rate, etc.
        Hide
        Enis Soztutar added a comment -

        Thanks Stack, you beat me to the 92, and 94 patches.

        Show
        Enis Soztutar added a comment - Thanks Stack, you beat me to the 92, and 94 patches.
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #249 (See https://builds.apache.org/job/HBase-0.94/249/)
        HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346503)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #249 (See https://builds.apache.org/job/HBase-0.94/249/ ) HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346503) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #450 (See https://builds.apache.org/job/HBase-0.92/450/)
        HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346506)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #450 (See https://builds.apache.org/job/HBase-0.92/450/ ) HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346506) Result = FAILURE stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #35 (See https://builds.apache.org/job/HBase-0.94-security/35/)
        HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346503)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #35 (See https://builds.apache.org/job/HBase-0.94-security/35/ ) HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346503) Result = SUCCESS stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #42 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/42/)
        HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346496)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #42 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/42/ ) HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346496) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3239 (See https://builds.apache.org/job/HBase-TRUNK/3239/)
        HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1374676)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3239 (See https://builds.apache.org/job/HBase-TRUNK/3239/ ) HBASE-6608 Fix for HBASE-6160 , META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1374676) Result = SUCCESS tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #136 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/136/)
        HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1374676)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #136 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/136/ ) HBASE-6608 Fix for HBASE-6160 , META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1374676) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #407 (See https://builds.apache.org/job/HBase-0.94/407/)
        HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #407 (See https://builds.apache.org/job/HBase-0.94/407/ ) HBASE-6608 Fix for HBASE-6160 , META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #508 (See https://builds.apache.org/job/HBase-0.92/508/)
        HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375159)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #508 (See https://builds.apache.org/job/HBase-0.92/508/ ) HBASE-6608 Fix for HBASE-6160 , META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375159) Result = SUCCESS tedyu : Files : /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #48 (See https://builds.apache.org/job/HBase-0.94-security/48/)
        HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #48 (See https://builds.apache.org/job/HBase-0.94-security/48/ ) HBASE-6608 Fix for HBASE-6160 , META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #116 (See https://builds.apache.org/job/HBase-0.92-security/116/)
        HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375159)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #116 (See https://builds.apache.org/job/HBase-0.92-security/116/ ) HBASE-6608 Fix for HBASE-6160 , META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375159) Result = FAILURE tedyu : Files : /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/)
        HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158)

        Result = FAILURE
        Tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/ ) HBASE-6608 Fix for HBASE-6160 , META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158) Result = FAILURE Tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java

          People

          • Assignee:
            Enis Soztutar
            Reporter:
            Enis Soztutar
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development