HBase
  1. HBase
  2. HBASE-5351

hbase completebulkload to a new table fails in a race

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0, 0.94.0
    • Fix Version/s: 0.92.1, 0.94.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I have a test that tests vanilla use of importtsv with importtsv.bulk.output option followed by completebulkload to a new table.

      This sometimes fails as follows:
      11/12/19 15:02:39 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch META table:
      org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for table: ml_items_copy, row=ml_items_copy,,99999999999999
      at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:157)
      at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:52)
      at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
      at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:127)
      at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
      at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:127)
      at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:103)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:875)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:929)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:817)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:781)
      at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:247)
      at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:211)
      at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
      at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.createTable(LoadIncrementalHFiles.java:673)
      at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:697)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
      at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:707)

      The race appears to be calling HbAdmin.createTableAsync(htd, keys) and then creating an HTable object before that call has actually completed.
      The following change to /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java appears to fix the problem, but I have not been able to reproduce the race reliably, in order to write a test.

      -    HTable table = new HTable(this.cfg, tableName);
      -
      -    HConnection conn = table.getConnection();
           int ctr = 0;
      -    while (!conn.isTableAvailable(table.getTableName()) && (ctr<TABLE_CREATE_MA
      +    while (!this.hbAdmin.isTableAvailable(tableName) && (ctr<TABLE_CREATE_MAX_R
      
      1. HBASE-5351-v2.patch
        2 kB
        Gregory Chanan
      2. HBASE-5351-v2.patch
        2 kB
        stack
      3. HBASE-5351-v1.patch
        2 kB
        Gregory Chanan
      4. HBASE-5351.patch
        2 kB
        Gregory Chanan

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #96 (See https://builds.apache.org/job/HBase-0.92-security/96/)
        HBASE-5351 hbase completebulkload to a new table fails in a race (Revision 1293480)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #96 (See https://builds.apache.org/job/HBase-0.92-security/96/ ) HBASE-5351 hbase completebulkload to a new table fails in a race (Revision 1293480) Result = FAILURE stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2669 (See https://builds.apache.org/job/HBase-TRUNK/2669/)
        HBASE-5351 hbase completebulkload to a new table fails in a race (Revision 1293479)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2669 (See https://builds.apache.org/job/HBase-TRUNK/2669/ ) HBASE-5351 hbase completebulkload to a new table fails in a race (Revision 1293479) Result = SUCCESS stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #122 (See https://builds.apache.org/job/HBase-TRUNK-security/122/)
        HBASE-5351 hbase completebulkload to a new table fails in a race (Revision 1293479)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #122 (See https://builds.apache.org/job/HBase-TRUNK-security/122/ ) HBASE-5351 hbase completebulkload to a new table fails in a race (Revision 1293479) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #304 (See https://builds.apache.org/job/HBase-0.92/304/)
        HBASE-5351 hbase completebulkload to a new table fails in a race (Revision 1293480)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #304 (See https://builds.apache.org/job/HBase-0.92/304/ ) HBASE-5351 hbase completebulkload to a new table fails in a race (Revision 1293480) Result = FAILURE stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Hide
        stack added a comment -

        Committed trunk and 0.92. Thanks for the patch Gregory.

        Show
        stack added a comment - Committed trunk and 0.92. Thanks for the patch Gregory.
        Hide
        Lars Hofhansl added a comment -

        +1 on V2.

        Show
        Lars Hofhansl added a comment - +1 on V2.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515977/HBASE-5351-v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -131 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1046//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1046//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1046//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515977/HBASE-5351-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -131 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1046//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1046//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1046//console This message is automatically generated.
        Hide
        stack added a comment -

        Uploading same patch so can redo submit patch to hadoopqa

        Show
        stack added a comment - Uploading same patch so can redo submit patch to hadoopqa
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515971/HBASE-5351-v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -131 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.master.TestLogsCleaner
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1045//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1045//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1045//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515971/HBASE-5351-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -131 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestLogsCleaner org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1045//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1045//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1045//console This message is automatically generated.
        Hide
        Gregory Chanan added a comment -

        attached HBASE-5351-v2.patch

        You are quite right – createTable catches the SocketTimeoutException anyway.

        Show
        Gregory Chanan added a comment - attached HBASE-5351 -v2.patch You are quite right – createTable catches the SocketTimeoutException anyway.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515962/HBASE-5351-v1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -131 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.replication.TestReplication
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1044//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1044//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1044//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515962/HBASE-5351-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -131 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1044//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1044//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1044//console This message is automatically generated.
        Hide
        Adrian Muraru added a comment -

        Great, what about try/catch java.net.SocketTimeoutException. Don't think is needed anymore when sync createTable is used. Let's let any exception thrown by createTable() call bubble up.
        What do you say?

        Show
        Adrian Muraru added a comment - Great, what about try/catch java.net.SocketTimeoutException. Don't think is needed anymore when sync createTable is used. Let's let any exception thrown by createTable() call bubble up. What do you say?
        Hide
        Gregory Chanan added a comment -

        *attached HBASE-5351-v1.patch

        @Adrian and stack:
        Agreed, I was just trying to make a minimal change.

        New patch as suggested.

        Show
        Gregory Chanan added a comment - *attached HBASE-5351 -v1.patch @Adrian and stack: Agreed, I was just trying to make a minimal change. New patch as suggested.
        Hide
        stack added a comment -

        @Adrian That seems like the way to go.

        Show
        stack added a comment - @Adrian That seems like the way to go.
        Hide
        Adrian Muraru added a comment -

        Saw the same issue in 0.92 branch and trace it down to the same

        this.hbAdmin.createTableAsync(htd, keys);

        and wondering why we wouldn't change this to:

        this.hbAdmin.createTable

        instead of looping and waiting for table to become available

        Show
        Adrian Muraru added a comment - Saw the same issue in 0.92 branch and trace it down to the same this.hbAdmin.createTableAsync(htd, keys); and wondering why we wouldn't change this to: this.hbAdmin.createTable instead of looping and waiting for table to become available
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515521/HBASE-5351.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -134 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.master.TestSplitLogManager
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestImportTsv

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1002//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1002//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1002//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515521/HBASE-5351.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -134 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1002//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1002//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1002//console This message is automatically generated.
        Hide
        Gregory Chanan added a comment -

        Patch, including comment that Jon requested.

        Show
        Gregory Chanan added a comment - Patch, including comment that Jon requested.
        Hide
        Ted Yu added a comment -

        @Gregory:
        Please attach a patch.

        Show
        Ted Yu added a comment - @Gregory: Please attach a patch.
        Hide
        Jonathan Hsieh added a comment -

        I buy that the patch should fix the problem, and don't think we need to have a test here. However, to prevent this problem in the future, can you update the javadoc comments in HBaseAdmin.createTableAsync to warn about this condition alerting devs that use this method to make sure the table is available before instantiating the HTable?

        Show
        Jonathan Hsieh added a comment - I buy that the patch should fix the problem, and don't think we need to have a test here. However, to prevent this problem in the future, can you update the javadoc comments in HBaseAdmin.createTableAsync to warn about this condition alerting devs that use this method to make sure the table is available before instantiating the HTable?

          People

          • Assignee:
            Gregory Chanan
            Reporter:
            Gregory Chanan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development