HBase
  1. HBase
  2. HBASE-5986

Clients can see holes in the META table when regions are being split

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.1, 0.94.1, 0.95.2
    • Fix Version/s: 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

      This is also related to HBASE-4335.

      1. HBASE-5986-test_v1.patch
        11 kB
        Enis Soztutar
      2. 5986-v2.txt
        37 kB
        Ted Yu
      3. HBASE-5986_v3.patch
        39 kB
        Ted Yu
      4. HBASE-5986-0.92.patch
        38 kB
        Enis Soztutar
      5. HBASE-5986-0.94.patch
        37 kB
        Enis Soztutar

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.92-security #109 (See https://builds.apache.org/job/HBase-0.92-security/109/)
          HBASE-5986 Clients can see holes in the META table when regions are being split (Enis) (Revision 1342866)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HTable.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Show
          Hudson added a comment - Integrated in HBase-0.92-security #109 (See https://builds.apache.org/job/HBase-0.92-security/109/ ) HBASE-5986 Clients can see holes in the META table when regions are being split (Enis) (Revision 1342866) Result = SUCCESS tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #32 (See https://builds.apache.org/job/HBase-0.94-security/32/)
          HBASE-5986 Clients can see holes in the META table when regions are being split (Enis) (Revision 1342865)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #32 (See https://builds.apache.org/job/HBase-0.94-security/32/ ) HBASE-5986 Clients can see holes in the META table when regions are being split (Enis) (Revision 1342865) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #422 (See https://builds.apache.org/job/HBase-0.92/422/)
          HBASE-5986 Clients can see holes in the META table when regions are being split (Enis) (Revision 1342866)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HTable.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #422 (See https://builds.apache.org/job/HBase-0.92/422/ ) HBASE-5986 Clients can see holes in the META table when regions are being split (Enis) (Revision 1342866) Result = FAILURE tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #218 (See https://builds.apache.org/job/HBase-0.94/218/)
          HBASE-5986 Clients can see holes in the META table when regions are being split (Enis) (Revision 1342865)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #218 (See https://builds.apache.org/job/HBase-0.94/218/ ) HBASE-5986 Clients can see holes in the META table when regions are being split (Enis) (Revision 1342865) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Hide
          Ted Yu added a comment -

          I verified the failed tests for 0.92 passed locally.

          Integrated to 0.92 branch.

          Thanks for the patches, Enis.

          Show
          Ted Yu added a comment - I verified the failed tests for 0.92 passed locally. Integrated to 0.92 branch. Thanks for the patches, Enis.
          Hide
          Ted Yu added a comment -

          I verified that the 3 failed tests for 0.94 passed.

          Integrated patch for 0.94 to 0.94 branch.

          Show
          Ted Yu added a comment - I verified that the 3 failed tests for 0.94 passed. Integrated patch for 0.94 to 0.94 branch.
          Hide
          Enis Soztutar added a comment -

          Here are the test results for 0.94:

          Tests run: 551, Failures: 0, Errors: 0, Skipped: 0
          ...
          Tests run: 932, Failures: 1, Errors: 2, Skipped: 9
          
          Failed tests:   testShutdownSimpleFixup(org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster): expected:<1> but was:<0>
          
          Tests in error: 
            testDelayedRpcImmediateReturnValue(org.apache.hadoop.hbase.ipc.TestDelayedRpc): Call to /127.0.0.1:53586 failed on socket timeout exception: java.net.SocketTimeoutException: 1000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/127.0.0.1:53623 remote=/127.0.0.1:53586]
            testLocalHBaseCluster(org.apache.hadoop.hbase.TestLocalHBaseCluster): Master not initialized after 200 seconds
          

          I rerun the tests locally with success, except TestLocalHBaseCluster. But it fails on 0.94 HEAD as well for me.

          For 0.92:

          
          Results :
          
          Failed tests:   testMultipleResubmits(org.apache.hadoop.hbase.master.TestSplitLogManager)
            testcomputeHDFSBlocksDistribution(org.apache.hadoop.hbase.util.TestFSUtils)
          
          Tests in error:
            testClusterRestart(org.apache.hadoop.hbase.master.TestRestartCluster): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
            testWholesomeSplit(org.apache.hadoop.hbase.regionserver.TestSplitTransaction): Failed delete of /homes/hortonde/enis/code/hbase-0.92/target/test-data/af023188-0b23-4f9d-a9bc-a074e94e57f8/org.apache.hadoop.hbase.regionserver.TestSplitTransaction/table/7c59b6677ad46bf3f652a83de1e62bcb
            testRollback(org.apache.hadoop.hbase.regionserver.TestSplitTransaction): Target HLog directory already exists: /homes/hortonde/enis/code/hbase-0.92/target/test-data/af023188-0b23-4f9d-a9bc-a074e94e57f8/org.apache.hadoop.hbase.regionserver.TestSplitTransaction/logs
            testRollback(org.apache.hadoop.hbase.regionserver.TestSplitTransaction)
            loadTest[0](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential): test timed out after 120000 milliseconds
            loadTest[0](org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel): test timed out after 120000 milliseconds
          
          Tests run: 1135, Failures: 2, Errors: 6, Skipped: 8
          

          Also run those failed tests locally with success. It seems we can go ahead with 0.92 and 0.94 if you don't have any concerns.

          Show
          Enis Soztutar added a comment - Here are the test results for 0.94: Tests run: 551, Failures: 0, Errors: 0, Skipped: 0 ... Tests run: 932, Failures: 1, Errors: 2, Skipped: 9 Failed tests: testShutdownSimpleFixup(org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster): expected:<1> but was:<0> Tests in error: testDelayedRpcImmediateReturnValue(org.apache.hadoop.hbase.ipc.TestDelayedRpc): Call to /127.0.0.1:53586 failed on socket timeout exception: java.net.SocketTimeoutException: 1000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/127.0.0.1:53623 remote=/127.0.0.1:53586] testLocalHBaseCluster(org.apache.hadoop.hbase.TestLocalHBaseCluster): Master not initialized after 200 seconds I rerun the tests locally with success, except TestLocalHBaseCluster. But it fails on 0.94 HEAD as well for me. For 0.92: Results : Failed tests: testMultipleResubmits(org.apache.hadoop.hbase.master.TestSplitLogManager) testcomputeHDFSBlocksDistribution(org.apache.hadoop.hbase.util.TestFSUtils) Tests in error: testClusterRestart(org.apache.hadoop.hbase.master.TestRestartCluster): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing testWholesomeSplit(org.apache.hadoop.hbase.regionserver.TestSplitTransaction): Failed delete of /homes/hortonde/enis/code/hbase-0.92/target/test-data/af023188-0b23-4f9d-a9bc-a074e94e57f8/org.apache.hadoop.hbase.regionserver.TestSplitTransaction/table/7c59b6677ad46bf3f652a83de1e62bcb testRollback(org.apache.hadoop.hbase.regionserver.TestSplitTransaction): Target HLog directory already exists: /homes/hortonde/enis/code/hbase-0.92/target/test-data/af023188-0b23-4f9d-a9bc-a074e94e57f8/org.apache.hadoop.hbase.regionserver.TestSplitTransaction/logs testRollback(org.apache.hadoop.hbase.regionserver.TestSplitTransaction) loadTest[0](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential): test timed out after 120000 milliseconds loadTest[0](org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel): test timed out after 120000 milliseconds Tests run: 1135, Failures: 2, Errors: 6, Skipped: 8 Also run those failed tests locally with success. It seems we can go ahead with 0.92 and 0.94 if you don't have any concerns.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #16 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/16/)
          HBASE-5986 Clients can see holes in the META table when regions are being split (Revision 1342099)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicStatistics.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #16 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/16/ ) HBASE-5986 Clients can see holes in the META table when regions are being split (Revision 1342099) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicStatistics.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Hide
          Enis Soztutar added a comment -

          @Ted, I did run TestEndToEndSplitTransaction, but not the whole suite. Let me do that.

          Show
          Enis Soztutar added a comment - @Ted, I did run TestEndToEndSplitTransaction, but not the whole suite. Let me do that.
          Hide
          Ted Yu added a comment -

          @Enis:
          Did you have a chance to run the backports through respective test suite ?

          Thanks

          Show
          Ted Yu added a comment - @Enis: Did you have a chance to run the backports through respective test suite ? Thanks
          Hide
          Enis Soztutar added a comment -

          Attaching patches for 0.92 and 0.94 branches. They are direct ports of the v3 patch, but 0.92 patch also includes HRegionServer.getOnlineRegions(byte[] tableName) function directly copied from 0.94, since we need it. I have discovered this when testing with 0.92, so I would like it to make into it.

          One minor mishap from my part is that the v3 patch which went into trunk includes an unrelated change in RegionServerDynamicStatistics. Related issue is HBASE-6025. Although the change is trivial ,changing RegionServerDynamicStatistics to extend hbase-specific MetricsMBeanBase rather than hadoop-specific MetricsDynamicMBeanBase, we may want to note this, or revert that part. Backport patches does not include this change.
          Sorry for the trouble guys.

          Show
          Enis Soztutar added a comment - Attaching patches for 0.92 and 0.94 branches. They are direct ports of the v3 patch, but 0.92 patch also includes HRegionServer.getOnlineRegions(byte[] tableName) function directly copied from 0.94, since we need it. I have discovered this when testing with 0.92, so I would like it to make into it. One minor mishap from my part is that the v3 patch which went into trunk includes an unrelated change in RegionServerDynamicStatistics. Related issue is HBASE-6025 . Although the change is trivial ,changing RegionServerDynamicStatistics to extend hbase-specific MetricsMBeanBase rather than hadoop-specific MetricsDynamicMBeanBase, we may want to note this, or revert that part. Backport patches does not include this change. Sorry for the trouble guys.
          Hide
          Ted Yu added a comment -

          @Enis:
          There're 5 files which have conflicts with 0.94 branch.
          Can you prepare patch for 0.94 ?

          Thanks

          Show
          Ted Yu added a comment - @Enis: There're 5 files which have conflicts with 0.94 branch. Can you prepare patch for 0.94 ? Thanks
          Hide
          Lars Hofhansl added a comment -

          +1 for 0.94, can't speak for 0.92.

          Show
          Lars Hofhansl added a comment - +1 for 0.94, can't speak for 0.92.
          Hide
          Ted Yu added a comment -

          This JIRA is marked as bug.
          The reason behind not choosing approach #3 is to facilitate backporting to 0.92 and 0.94

          I think 0.92.2 and 0.94.1 should carry this fix.

          Show
          Ted Yu added a comment - This JIRA is marked as bug. The reason behind not choosing approach #3 is to facilitate backporting to 0.92 and 0.94 I think 0.92.2 and 0.94.1 should carry this fix.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2919 (See https://builds.apache.org/job/HBase-TRUNK/2919/)
          HBASE-5986 Clients can see holes in the META table when regions are being split (Revision 1342099)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicStatistics.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2919 (See https://builds.apache.org/job/HBase-TRUNK/2919/ ) HBASE-5986 Clients can see holes in the META table when regions are being split (Revision 1342099) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicStatistics.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          Hide
          stack added a comment -

          Thanks for running tests Ted.

          Show
          stack added a comment - Thanks for running tests Ted.
          Hide
          stack added a comment -

          Committed to trunk. Thanks for the patch Enis.

          Show
          stack added a comment - Committed to trunk. Thanks for the patch Enis.
          Hide
          Ted Yu added a comment -

          I ran TestAssignmentManager with patch v3 and it passed:

          Running org.apache.hadoop.hbase.master.TestAssignmentManager
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.703 sec
          ...
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 17.004s
          
          Show
          Ted Yu added a comment - I ran TestAssignmentManager with patch v3 and it passed: Running org.apache.hadoop.hbase.master.TestAssignmentManager Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.703 sec ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 17.004s
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12528806/HBASE-5986_v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 34 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.TestReplication
          org.apache.hadoop.hbase.master.TestAssignmentManager
          org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
          org.apache.hadoop.hbase.replication.TestMasterReplication

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1972//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1972//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1972//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528806/HBASE-5986_v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 34 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.master.TestAssignmentManager org.apache.hadoop.hbase.replication.TestMultiSlaveReplication org.apache.hadoop.hbase.replication.TestMasterReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1972//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1972//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1972//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Patch v3 from Enis.

          Show
          Ted Yu added a comment - Patch v3 from Enis.
          Hide
          Ted Yu added a comment -

          @Stack:
          What do you think of patch v2 ?

          Show
          Ted Yu added a comment - @Stack: What do you think of patch v2 ?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12527760/5986-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 32 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1903//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1903//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1903//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527760/5986-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 32 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1903//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1903//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1903//console This message is automatically generated.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/#review7948
          -----------------------------------------------------------

          v2 looks much better.

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17287>

          We may need to mark this JIRA an incompatible change.

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          <https://reviews.apache.org/r/5133/#comment17283>

          'with' -> 'by'

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          <https://reviews.apache.org/r/5133/#comment17284>

          This class can be private.

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          <https://reviews.apache.org/r/5133/#comment17285>

          The interval can be made shorter.

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
          <https://reviews.apache.org/r/5133/#comment17286>

          Name this method verifyRegionsOfHTable ?

          • Ted

          On 2012-05-17 00:58:21, enis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/5133/

          -----------------------------------------------------------

          (Updated 2012-05-17 00:58:21)

          Review request for hbase.

          Summary

          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.

          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af

          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18

          src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing

          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/#review7948 ----------------------------------------------------------- v2 looks much better. src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17287 > We may need to mark this JIRA an incompatible change. src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java < https://reviews.apache.org/r/5133/#comment17283 > 'with' -> 'by' src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java < https://reviews.apache.org/r/5133/#comment17284 > This class can be private. src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java < https://reviews.apache.org/r/5133/#comment17285 > The interval can be made shorter. src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java < https://reviews.apache.org/r/5133/#comment17286 > Name this method verifyRegionsOfHTable ? Ted On 2012-05-17 00:58:21, enis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-17 00:58:21) Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs ----- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          Ted Yu added a comment -

          v2 from Enis

          Show
          Ted Yu added a comment - v2 from Enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/
          -----------------------------------------------------------

          (Updated 2012-05-17 00:58:21.965488)

          Review request for hbase.

          Changes
          -------

          v2:

          • addressed review comments
          • when timeout and interrupt throws RegionOfflineException.
          • reuse HTable instances in BMSV.
          • adds close() to MSV assuming it is indeed not public, evolving but private. we should decide the visibility.

          Summary
          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.
          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512
          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4
          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18
          src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing
          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-17 00:58:21.965488) Review request for hbase. Changes ------- v2: addressed review comments when timeout and interrupt throws RegionOfflineException. reuse HTable instances in BMSV. adds close() to MSV assuming it is indeed not public, evolving but private. we should decide the visibility. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs (updated) src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-05-16 21:32:56, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 473

          > <https://reviews.apache.org/r/5133/diff/1/?file=109116#file109116line473>

          >

          > Why you remove this? We don't return these any more? Offline I think is 'dead', unused now. Split not.

          If we discover that a region has been split in META, than, it is past point-of-no-return and the region cannot be seen un-split anymore, even though concurrent rs failures. for getStartEndKeys() we are returning whatever the getRegionLocation() provides. getRegionLocations() ignores offline regions, but returns daughter regions for split-parents (which are offline as well).

          On 2012-05-16 21:32:56, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 351

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line351>

          >

          > Should this be public? Should it remain internal to the HTable hidden?

          MetaScanner is confusing in its visibility. It's javadoc states "Although public visibility, this is not a public-facing API and may evolve in minor releases.", but it is annotated with @InterfaceAudience.Public @InterfaceStability.Evolving. I think for BlockingMetaScannerVisitor, we should set the visibility the same as MetaScannerVisitor. I think, MetaScannerVisitor itself is of little use without the BlockingMetaScannerVisitor functionality due to this issue.
          Shall we change MetaScanner, MSV and BMSV to be @InterfaceAudience.Private, wdyt?

          On 2012-05-16 21:32:56, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 310

          > <https://reviews.apache.org/r/5133/diff/1/?file=109115#file109115line310>

          >

          > Bit of javadoc to say this is best-effort.

          >

          > Also, does this belong in MetaReader (Won't hold you to it... these two classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog package needs killing).

          agreed

          On 2012-05-16 21:32:56, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 395

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line395>

          >

          > Yeah, so, what happens if daughter has split by the time I get here?

          The scanner provides the regions results in sorted order, and since the daughters are sorted after the parent, we always process parent first. When we process the daughter regions manually (processRow(resultA)), we also add them to the daughterRegions set. If the scanner also sees them, they are just skipped (line 373)

          On 2012-05-16 21:32:56, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 434

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line434>

          >

          > So if interrupted or we don't find it by the time the blocking time has passed, we just return null? What you reckon? We should at least complain?

          yeah, I think we can throw RegionOfflineException upon timeout and interrupt. The operation can be retried by the client upon timeout.

          On 2012-05-16 21:32:56, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 465

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line465>

          >

          > Does the scan of meta start at first table region?

          MetaScanner.metaScan() ensures that the scan starts at the first table region.

          On 2012-05-16 21:32:56, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line390>

          >

          > Yeah, what Ted says... Can you close when done? See MetaEditor/MetaReader. They do this a bunch. Closing means for sure the zk and connection resources will be cleaned up afterward. Its reference counting so keepign around an HTable could mess it up.

          We are already closing the HTable's. But let me see how we can best reuse Htable instances.

          • enis

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/#review7939
          -----------------------------------------------------------

          On 2012-05-16 01:53:09, enis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/5133/

          -----------------------------------------------------------

          (Updated 2012-05-16 01:53:09)

          Review request for hbase.

          Summary

          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.

          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512

          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing

          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-05-16 21:32:56, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 473 > < https://reviews.apache.org/r/5133/diff/1/?file=109116#file109116line473 > > > Why you remove this? We don't return these any more? Offline I think is 'dead', unused now. Split not. If we discover that a region has been split in META, than, it is past point-of-no-return and the region cannot be seen un-split anymore, even though concurrent rs failures. for getStartEndKeys() we are returning whatever the getRegionLocation() provides. getRegionLocations() ignores offline regions, but returns daughter regions for split-parents (which are offline as well). On 2012-05-16 21:32:56, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 351 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line351 > > > Should this be public? Should it remain internal to the HTable hidden? MetaScanner is confusing in its visibility. It's javadoc states "Although public visibility, this is not a public-facing API and may evolve in minor releases.", but it is annotated with @InterfaceAudience.Public @InterfaceStability.Evolving. I think for BlockingMetaScannerVisitor, we should set the visibility the same as MetaScannerVisitor. I think, MetaScannerVisitor itself is of little use without the BlockingMetaScannerVisitor functionality due to this issue. Shall we change MetaScanner, MSV and BMSV to be @InterfaceAudience.Private, wdyt? On 2012-05-16 21:32:56, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 310 > < https://reviews.apache.org/r/5133/diff/1/?file=109115#file109115line310 > > > Bit of javadoc to say this is best-effort. > > Also, does this belong in MetaReader (Won't hold you to it... these two classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog package needs killing). agreed On 2012-05-16 21:32:56, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 395 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line395 > > > Yeah, so, what happens if daughter has split by the time I get here? The scanner provides the regions results in sorted order, and since the daughters are sorted after the parent, we always process parent first. When we process the daughter regions manually (processRow(resultA)), we also add them to the daughterRegions set. If the scanner also sees them, they are just skipped (line 373) On 2012-05-16 21:32:56, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 434 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line434 > > > So if interrupted or we don't find it by the time the blocking time has passed, we just return null? What you reckon? We should at least complain? yeah, I think we can throw RegionOfflineException upon timeout and interrupt. The operation can be retried by the client upon timeout. On 2012-05-16 21:32:56, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 465 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line465 > > > Does the scan of meta start at first table region? MetaScanner.metaScan() ensures that the scan starts at the first table region. On 2012-05-16 21:32:56, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line390 > > > Yeah, what Ted says... Can you close when done? See MetaEditor/MetaReader. They do this a bunch. Closing means for sure the zk and connection resources will be cleaned up afterward. Its reference counting so keepign around an HTable could mess it up. We are already closing the HTable's. But let me see how we can best reuse Htable instances. enis ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/#review7939 ----------------------------------------------------------- On 2012-05-16 01:53:09, enis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-16 01:53:09) Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs ----- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          Lars Hofhansl added a comment -

          As a general comment I think it would be better if instead of blocking on the server, HTable.getStartEndKeys() could detect the inconsistency and try again.

          Show
          Lars Hofhansl added a comment - As a general comment I think it would be better if instead of blocking on the server, HTable.getStartEndKeys() could detect the inconsistency and try again.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/#review7939
          -----------------------------------------------------------

          Looks great Enis. Thanks for having a go at this. Loads of white space introduced which you might fix in v2. Otherwise a few little items below....

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          <https://reviews.apache.org/r/5133/#comment17262>

          Bit of javadoc to say this is best-effort.

          Also, does this belong in MetaReader (Won't hold you to it... these two classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog package needs killing).

          src/main/java/org/apache/hadoop/hbase/client/HTable.java
          <https://reviews.apache.org/r/5133/#comment17264>

          Why you remove this? We don't return these any more? Offline I think is 'dead', unused now. Split not.

          src/main/java/org/apache/hadoop/hbase/client/HTable.java
          <https://reviews.apache.org/r/5133/#comment17265>

          This seems like a good refactoring.

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17266>

          Good

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17267>

          Should this be public? Should it remain internal to the HTable hidden?

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17268>

          Yeah, it could be a long time before a region comes on line if its in a GC or some such.

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17269>

          Yeah, what Ted says... Can you close when done? See MetaEditor/MetaReader. They do this a bunch. Closing means for sure the zk and connection resources will be cleaned up afterward. Its reference counting so keepign around an HTable could mess it up.

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17270>

          Yeah, so, what happens if daughter has split by the time I get here?

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17271>

          So if interrupted or we don't find it by the time the blocking time has passed, we just return null? What you reckon? We should at least complain?

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17272>

          Does the scan of meta start at first table region?

          • Michael

          On 2012-05-16 01:53:09, enis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/5133/

          -----------------------------------------------------------

          (Updated 2012-05-16 01:53:09)

          Review request for hbase.

          Summary

          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.

          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512

          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing

          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/#review7939 ----------------------------------------------------------- Looks great Enis. Thanks for having a go at this. Loads of white space introduced which you might fix in v2. Otherwise a few little items below.... src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java < https://reviews.apache.org/r/5133/#comment17262 > Bit of javadoc to say this is best-effort. Also, does this belong in MetaReader (Won't hold you to it... these two classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog package needs killing). src/main/java/org/apache/hadoop/hbase/client/HTable.java < https://reviews.apache.org/r/5133/#comment17264 > Why you remove this? We don't return these any more? Offline I think is 'dead', unused now. Split not. src/main/java/org/apache/hadoop/hbase/client/HTable.java < https://reviews.apache.org/r/5133/#comment17265 > This seems like a good refactoring. src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17266 > Good src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17267 > Should this be public? Should it remain internal to the HTable hidden? src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17268 > Yeah, it could be a long time before a region comes on line if its in a GC or some such. src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17269 > Yeah, what Ted says... Can you close when done? See MetaEditor/MetaReader. They do this a bunch. Closing means for sure the zk and connection resources will be cleaned up afterward. Its reference counting so keepign around an HTable could mess it up. src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17270 > Yeah, so, what happens if daughter has split by the time I get here? src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17271 > So if interrupted or we don't find it by the time the blocking time has passed, we just return null? What you reckon? We should at least complain? src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17272 > Does the scan of meta start at first table region? Michael On 2012-05-16 01:53:09, enis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-16 01:53:09) Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs ----- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          stack added a comment -

          Regards 1. and 2., what guarantees do we have that the daughter will not have split by the time we go into our wait on the daughters to come online?

          Show
          stack added a comment - Regards 1. and 2., what guarantees do we have that the daughter will not have split by the time we go into our wait on the daughters to come online?
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/#review7938
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17261>

          I was asking how do we convey the fact that MetaScanner has timed out to client.

          I don't see exception being thrown which contains timeout information.

          • Ted

          On 2012-05-16 01:53:09, enis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/5133/

          -----------------------------------------------------------

          (Updated 2012-05-16 01:53:09)

          Review request for hbase.

          Summary

          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.

          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512

          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing

          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/#review7938 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17261 > I was asking how do we convey the fact that MetaScanner has timed out to client. I don't see exception being thrown which contains timeout information. Ted On 2012-05-16 01:53:09, enis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-16 01:53:09) Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs ----- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-05-16 05:20:29, ramkrishna vasudevan wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 402

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line402>

          >

          > Blocking time out is one factor on which we will be waiting. So this blockingtimeout should be in lieu with 'fileSplitTimeout'?

          From my understanding, fileSplitTimeout is a regionserver property, while the timeout parameter in this patch is a client side property, since the blocking will happen on the client. I think "hbase.client.operation.timeout" can serve us well. HConstants state that it is "Default HBase client operation timeout, which is tantamount to a blocking call".

          • enis

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/#review7928
          -----------------------------------------------------------

          On 2012-05-16 01:53:09, enis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/5133/

          -----------------------------------------------------------

          (Updated 2012-05-16 01:53:09)

          Review request for hbase.

          Summary

          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.

          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512

          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing

          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-05-16 05:20:29, ramkrishna vasudevan wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 402 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line402 > > > Blocking time out is one factor on which we will be waiting. So this blockingtimeout should be in lieu with 'fileSplitTimeout'? From my understanding, fileSplitTimeout is a regionserver property, while the timeout parameter in this patch is a client side property, since the blocking will happen on the client. I think "hbase.client.operation.timeout" can serve us well. HConstants state that it is "Default HBase client operation timeout, which is tantamount to a blocking call". enis ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/#review7928 ----------------------------------------------------------- On 2012-05-16 01:53:09, enis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-16 01:53:09) Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs ----- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-05-16 02:52:04, Ted Yu wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 391

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line391>

          >

          > null is always returned by getRegionResultBlocking() in case of time out.

          > How do we deal with that ?

          >

          we are checking for null return value for resultA, and resultB. But I'll also add a LOG warning.

          On 2012-05-16 02:52:04, Ted Yu wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line390>

          >

          > Do we have to instantiate HTable every time ?

          This is a bit tricky. Ideally we should not. But there is no close() on MetaScannerVisitor, so we cannot close the HTable properly if we reuse the HTable across calls to processRow(). We can add a close() method and call it, or obtain the HTable from the context, but that would imply changing the class signature for MetaScannerVisitor. I assumed since we are reusing HConnection's HTable creation is cheap, is that not the case, wdyt?

          On 2012-05-16 02:52:04, Ted Yu wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 360

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line360>

          >

          > Can we add a config param for blockingTimeout ?

          please see below

          On 2012-05-16 02:52:04, Ted Yu wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 398

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line398>

          >

          > We shouldn't be passing blockingTimeout here.

          > We need to consider the amount of time spent in the call @ line 388.

          agreed

          On 2012-05-16 02:52:04, Ted Yu wrote:

          > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 430

          > <https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line430>

          >

          > Please restore interrupted state of the thread.

          agreed

          • enis

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/#review7925
          -----------------------------------------------------------

          On 2012-05-16 01:53:09, enis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/5133/

          -----------------------------------------------------------

          (Updated 2012-05-16 01:53:09)

          Review request for hbase.

          Summary

          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.

          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512

          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing

          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-05-16 02:52:04, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 391 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line391 > > > null is always returned by getRegionResultBlocking() in case of time out. > How do we deal with that ? > we are checking for null return value for resultA, and resultB. But I'll also add a LOG warning. On 2012-05-16 02:52:04, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line390 > > > Do we have to instantiate HTable every time ? This is a bit tricky. Ideally we should not. But there is no close() on MetaScannerVisitor, so we cannot close the HTable properly if we reuse the HTable across calls to processRow(). We can add a close() method and call it, or obtain the HTable from the context, but that would imply changing the class signature for MetaScannerVisitor. I assumed since we are reusing HConnection's HTable creation is cheap, is that not the case, wdyt? On 2012-05-16 02:52:04, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 360 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line360 > > > Can we add a config param for blockingTimeout ? please see below On 2012-05-16 02:52:04, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 398 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line398 > > > We shouldn't be passing blockingTimeout here. > We need to consider the amount of time spent in the call @ line 388. agreed On 2012-05-16 02:52:04, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 430 > < https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line430 > > > Please restore interrupted state of the thread. agreed enis ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/#review7925 ----------------------------------------------------------- On 2012-05-16 01:53:09, enis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-16 01:53:09) Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs ----- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/#review7928
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17254>

          Blocking time out is one factor on which we will be waiting. So this blockingtimeout should be in lieu with 'fileSplitTimeout'?

          • ramkrishna

          On 2012-05-16 01:53:09, enis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/5133/

          -----------------------------------------------------------

          (Updated 2012-05-16 01:53:09)

          Review request for hbase.

          Summary

          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.

          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512

          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing

          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/#review7928 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17254 > Blocking time out is one factor on which we will be waiting. So this blockingtimeout should be in lieu with 'fileSplitTimeout'? ramkrishna On 2012-05-16 01:53:09, enis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-16 01:53:09) Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs ----- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/#review7925
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          <https://reviews.apache.org/r/5133/#comment17245>

          Remove white space.

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17246>

          Can we add a config param for blockingTimeout ?

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17247>

          Do we have to instantiate HTable every time ?

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17250>

          null is always returned by getRegionResultBlocking() in case of time out.
          How do we deal with that ?

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17249>

          We shouldn't be passing blockingTimeout here.
          We need to consider the amount of time spent in the call @ line 388.

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          <https://reviews.apache.org/r/5133/#comment17248>

          Please restore interrupted state of the thread.

          • Ted

          On 2012-05-16 01:53:09, enis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/5133/

          -----------------------------------------------------------

          (Updated 2012-05-16 01:53:09)

          Review request for hbase.

          Summary

          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.

          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512

          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing

          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/#review7925 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java < https://reviews.apache.org/r/5133/#comment17245 > Remove white space. src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17246 > Can we add a config param for blockingTimeout ? src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17247 > Do we have to instantiate HTable every time ? src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17250 > null is always returned by getRegionResultBlocking() in case of time out. How do we deal with that ? src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17249 > We shouldn't be passing blockingTimeout here. We need to consider the amount of time spent in the call @ line 388. src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java < https://reviews.apache.org/r/5133/#comment17248 > Please restore interrupted state of the thread. Ted On 2012-05-16 01:53:09, enis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-16 01:53:09) Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs ----- src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/
          -----------------------------------------------------------

          (Updated 2012-05-16 01:53:09.861265)

          Review request for hbase.

          Summary (updated)
          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk.

          Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475).

          This addresses bug HBASE-5986.
          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs


          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512
          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4
          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999
          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing
          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- (Updated 2012-05-16 01:53:09.861265) Review request for hbase. Summary (updated) ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This patch is the approach 2 mentioned in the issue comments, mainly during META scan, if we detect that the region is split, we block until the information for the child regions are available in META and manually feed those rows to the MetaScanner. Although approach 3 (using local region transactions) seems cleaner, they are not available under branch 0.92, which I think should also incorporate this fix. I'll provide ports once we are clear for trunk. Also this patch does not fix MetaReader (see https://issues.apache.org/jira/browse/HBASE-3475 ). This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/5133/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          We found this issue when running large scale ingestion tests for HBASE-5754. The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet.

          This addresses bug HBASE-5986.
          https://issues.apache.org/jira/browse/HBASE-5986

          Diffs


          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512
          src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4
          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999
          src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6

          Diff: https://reviews.apache.org/r/5133/diff

          Testing
          -------

          added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests.

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5133/ ----------------------------------------------------------- Review request for hbase. Summary ------- We found this issue when running large scale ingestion tests for HBASE-5754 . The problem is that the .META. table updates are not atomic while splitting a region. In SplitTransaction, there is a time lap between the marking the parent offline, and adding of daughters to the META table. This can result in clients using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) missing regions which are made just offline, but the daughters are not added yet. This addresses bug HBASE-5986 . https://issues.apache.org/jira/browse/HBASE-5986 Diffs src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java a8091e6 Diff: https://reviews.apache.org/r/5133/diff Testing ------- added extensive tests under TestEndToEndSplitTranscation, and ran existing unit tests. Thanks, enis
          Hide
          Lars Hofhansl added a comment -

          @Enis: yes HRegion.mutateRowsWithLock() has correct MVCC semantics even across multi row scans.

          The region boundary could be between start_key1 and mid_key1 if there rows inserted and deleted before... Still unlikely. To be safe we could just have a splitKeyPolicy that never splits the table prefix (i.e. all keys for the same table are guaranteed to be in the region).

          Show
          Lars Hofhansl added a comment - @Enis: yes HRegion.mutateRowsWithLock() has correct MVCC semantics even across multi row scans. The region boundary could be between start_key1 and mid_key1 if there rows inserted and deleted before... Still unlikely. To be safe we could just have a splitKeyPolicy that never splits the table prefix (i.e. all keys for the same table are guaranteed to be in the region).
          Hide
          Enis Soztutar added a comment -

          I have implemented approach 1 by adding split daughters to the returned map from MetaScanner.allTableRegions(). But then the problem is that, we are returning regions which does not yet exists in the META table, so any subsequent getRegion call will fail.

          Thinking a bit more about 3, I think we already guarantee that the region split parent, and daughters fall into the same META region. Let's say we have two regions region1 and region2, with start keys start_key*, and timestamps ts* respectively.

          Before split:

           
          <table> <start_key1> <ts1> <encoded_name1>
          <table> <start_key2> <ts2> <encoded_name2>
          

          Now, if we split region1, daughters will be sorted after region1, and before region2:

           
          <table> <start_key1> <ts1> <encoded_name1> <offline> <split>
          <table> <start_key1> <ts3> <encoded_name1>
          <table> <mid_key1> <ts3> <encoded_name1>
          <table> <start_key2> <ts2> <encoded_name2>
          

          we know this since we have the invariants ts3 > ts1 (SplitTransaction.getDaughterRegionIdTimestamp()) and start_key1 < mid_key1 < start_key2. Even if we have a region boundary between start_key1 and start_key2 in the META table, the daughters will be co-located with the parent. The only exception is that while the user table is split, we have a concurrent split for the META table, and the new region boundary is chosen to be between the parent and daughters. With some effort, we can prevent this, but it seems to be very highly unlikely.

          So, if my analysis is correct, that means option 3 seems like the best choice, since this will not complicate the meta scan code. The problem is that, there is no internal API to do multi-row transcations other than using the coprocessor. Should we think of allowing that w/o coprocessors?

          @Lars, does HRegion.mutateRowsWithLock() guarantee that a concurrent scanner won't see partial changes?

          Show
          Enis Soztutar added a comment - I have implemented approach 1 by adding split daughters to the returned map from MetaScanner.allTableRegions(). But then the problem is that, we are returning regions which does not yet exists in the META table, so any subsequent getRegion call will fail. Thinking a bit more about 3, I think we already guarantee that the region split parent, and daughters fall into the same META region. Let's say we have two regions region1 and region2, with start keys start_key*, and timestamps ts* respectively. Before split: <table> <start_key1> <ts1> <encoded_name1> <table> <start_key2> <ts2> <encoded_name2> Now, if we split region1, daughters will be sorted after region1, and before region2: <table> <start_key1> <ts1> <encoded_name1> <offline> <split> <table> <start_key1> <ts3> <encoded_name1> <table> <mid_key1> <ts3> <encoded_name1> <table> <start_key2> <ts2> <encoded_name2> we know this since we have the invariants ts3 > ts1 (SplitTransaction.getDaughterRegionIdTimestamp()) and start_key1 < mid_key1 < start_key2. Even if we have a region boundary between start_key1 and start_key2 in the META table, the daughters will be co-located with the parent. The only exception is that while the user table is split, we have a concurrent split for the META table, and the new region boundary is chosen to be between the parent and daughters. With some effort, we can prevent this, but it seems to be very highly unlikely. So, if my analysis is correct, that means option 3 seems like the best choice, since this will not complicate the meta scan code. The problem is that, there is no internal API to do multi-row transcations other than using the coprocessor. Should we think of allowing that w/o coprocessors? @Lars, does HRegion.mutateRowsWithLock() guarantee that a concurrent scanner won't see partial changes?
          Hide
          Ted Yu added a comment -

          Can MetaScanner.allTableRegions() return special exception so that client knows it needs to check back ?

          Show
          Ted Yu added a comment - Can MetaScanner.allTableRegions() return special exception so that client knows it needs to check back ?
          Hide
          Lars Hofhansl added a comment -

          True, META is not single region by design, but effectively it is (not really suggesting that we add more code that relies on this). I think you're right we could add a split policy that ensures that parent and daughters are always in the same region.

          As for the alternative:
          If the client has to recheck in these (rare) cases that seems OK, as long as the client can reliably tell that it does have to recheck.

          Show
          Lars Hofhansl added a comment - True, META is not single region by design, but effectively it is (not really suggesting that we add more code that relies on this). I think you're right we could add a split policy that ensures that parent and daughters are always in the same region. As for the alternative: If the client has to recheck in these (rare) cases that seems OK, as long as the client can reliably tell that it does have to recheck.
          Hide
          Enis Soztutar added a comment -

          I think we are assuming in many other places that META only has a single region.

          The ROOT is by-design one region, but META is not, right?

          Is there another alternative, such as adding the daughter regions first, and then have HTable disentangle conflicts?

          I have thought about this as well, but then there is a time window in which you have both the parent and daughter regions online, and parent not marked as split. So the client again has to resolve that the returned regions are overlapping.

          Show
          Enis Soztutar added a comment - I think we are assuming in many other places that META only has a single region. The ROOT is by-design one region, but META is not, right? Is there another alternative, such as adding the daughter regions first, and then have HTable disentangle conflicts? I have thought about this as well, but then there is a time window in which you have both the parent and daughter regions online, and parent not marked as split. So the client again has to resolve that the returned regions are overlapping.
          Hide
          Lars Hofhansl added a comment -

          I think we are assuming in many other places that META only has a single region.

          Is there another alternative, such as adding the daughter regions first, and then have HTable disentangle conflicts?

          Show
          Lars Hofhansl added a comment - I think we are assuming in many other places that META only has a single region. Is there another alternative, such as adding the daughter regions first, and then have HTable disentangle conflicts?
          Hide
          Enis Soztutar added a comment -

          Possible fixes I can think of:
          1. Keep MetaScanner/MetaReader as non-consistent (as it is), but allow for a consistent view for getting table regions. Since single row puts are atomic, when the parent region is mutated to be offline, the HRI for daughters are added to the row. So on MetaScanner.allTableRegions and similar calls, we can keep track of daughter regions from split parents and return them to the client.
          2. Make MetaScanner consistent, in that, whenever it sees a split parent, it blocks until the daughters are available.
          3. We have region-local transactions now, so if we ensure that the rows for parent and daughters will be served from the same META region, then we can update all three rows atomically. Maybe we can come up with a META-specific split policy to ensure split-regions go to the same META region.
          Thoughts?

          Show
          Enis Soztutar added a comment - Possible fixes I can think of: 1. Keep MetaScanner/MetaReader as non-consistent (as it is), but allow for a consistent view for getting table regions. Since single row puts are atomic, when the parent region is mutated to be offline, the HRI for daughters are added to the row. So on MetaScanner.allTableRegions and similar calls, we can keep track of daughter regions from split parents and return them to the client. 2. Make MetaScanner consistent, in that, whenever it sees a split parent, it blocks until the daughters are available. 3. We have region-local transactions now, so if we ensure that the rows for parent and daughters will be served from the same META region, then we can update all three rows atomically. Maybe we can come up with a META-specific split policy to ensure split-regions go to the same META region. Thoughts?
          Hide
          Enis Soztutar added a comment -

          Attaching a unit test to illustrate the problem. The test fails for me when splitting the first region, the client sees 0 regions for some time.

          Show
          Enis Soztutar added a comment - Attaching a unit test to illustrate the problem. The test fails for me when splitting the first region, the client sees 0 regions for some time.

            People

            • Assignee:
              Enis Soztutar
              Reporter:
              Enis Soztutar
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development