HBase
  1. HBase
  2. HBASE-4552

multi-CF bulk load is not atomic across column families

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0
    • Fix Version/s: 0.92.0
    • Component/s: regionserver
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      HRegion.bulkLoadHFile() has been removed in 0.92 and TRUNK.
      HRegion.bulkLoadHFiles() is introduced for multi-family bulk load.
      Show
      HRegion.bulkLoadHFile() has been removed in 0.92 and TRUNK. HRegion.bulkLoadHFiles() is introduced for multi-family bulk load.

      Description

      Currently the bulk load API simply imports one HFile at a time. With multi-column-family support, this is inappropriate, since different CFs show up separately. Instead, the IPC endpoint should take a of CF -> HFiles, so we can online them all under a single region-wide lock.

      1. hbase-4552.consolidated.v4.patch
        53 kB
        Ted Yu
      2. hbase-4552.consolidated.v3.patch
        55 kB
        Ted Yu
      3. hbase-4552.consolidated.v2.patch
        55 kB
        Jonathan Hsieh
      4. hbase-4552.consolidated.patch
        55 kB
        Jonathan Hsieh

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          One solution is to add the following method to HRegionInterface:

            public void bulkLoadHFile(List<Pair<byte[], String>> familyPaths, byte[] regionName)
            throws IOException;
          

          familyPaths is a list of <family, hfilePath> pairs for the same region identified by regionName.

          LoadIncrementalHFiles would need to group HFiles for the same region together before calling the above method.

          Show
          Ted Yu added a comment - One solution is to add the following method to HRegionInterface: public void bulkLoadHFile(List<Pair< byte [], String >> familyPaths, byte [] regionName) throws IOException; familyPaths is a list of <family, hfilePath> pairs for the same region identified by regionName. LoadIncrementalHFiles would need to group HFiles for the same region together before calling the above method.
          Hide
          Todd Lipcon added a comment -

          yep, that's what I meant, but the implementation isn't quite trivial since we have to do the locking at a higher level, so the change is made visible atomically.

          Show
          Todd Lipcon added a comment - yep, that's what I meant, but the implementation isn't quite trivial since we have to do the locking at a higher level, so the change is made visible atomically.
          Hide
          Ted Yu added a comment -

          For HRegion, we can introduce the following new method:

            public void bulkLoadHFile(List<Pair<byte[], String>> familyPaths)
            throws IOException {
          

          where familyPaths is a list of <family, hfilePath> pairs identifying HFile path and the underlying family the HFile should be loaded to.

          Show
          Ted Yu added a comment - For HRegion, we can introduce the following new method: public void bulkLoadHFile(List<Pair< byte [], String >> familyPaths) throws IOException { where familyPaths is a list of <family, hfilePath> pairs identifying HFile path and the underlying family the HFile should be loaded to.
          Hide
          Ted Yu added a comment -

          Since LoadIncrementalHFiles uses ExecutorService to achieve parallelism, we should use Queue<Pair<byte[], String>> in place of List above so that concurrent queue can be instantiated.

          Show
          Ted Yu added a comment - Since LoadIncrementalHFiles uses ExecutorService to achieve parallelism, we should use Queue<Pair<byte[], String>> in place of List above so that concurrent queue can be instantiated.
          Hide
          Todd Lipcon added a comment -

          The trick is making sure it's atomic inside the region server - not just that the client sends all of the files for a given region in one RPC. If there are any concurrent scanners, then they should either see all of the new data or none of the new data on a given row. So we need some region-wide coordination. I think probably we have to take a write-lock on HRegion#lock

          Show
          Todd Lipcon added a comment - The trick is making sure it's atomic inside the region server - not just that the client sends all of the files for a given region in one RPC. If there are any concurrent scanners, then they should either see all of the new data or none of the new data on a given row. So we need some region-wide coordination. I think probably we have to take a write-lock on HRegion#lock
          Hide
          Jonathan Hsieh added a comment -

          Plan

          1) Test to show there is an atomicity problem. Likely just does not use LoadIncrementalHFiles
          2) Fix for the region server side.
          3) Rewrite of LoadIncrementalHFiles so that it groups the proper HFiles into the new bulkLoadHFile calls. This will likely have two parallel steps - the first gather enough info to group HFiles and then the second that attempts to bulk load.

          Show
          Jonathan Hsieh added a comment - Plan 1) Test to show there is an atomicity problem. Likely just does not use LoadIncrementalHFiles 2) Fix for the region server side. 3) Rewrite of LoadIncrementalHFiles so that it groups the proper HFiles into the new bulkLoadHFile calls. This will likely have two parallel steps - the first gather enough info to group HFiles and then the second that attempts to bulk load.
          Hide
          Jonathan Hsieh added a comment -

          One more piece: Mechanism to atomically rollback if a partial failures encountered when attempting to bulk load multiple families.

          For example, let's say I want to bulk load a region with cfs A, B, C. I issue a call to an RS region to atomically bulkload the HFiles. The RS loads A and B successfully but fails on C (hdfs failure, or rs goes down, etc). We should rollback A and B – if we don't we would have A and B loaded but not C and have an atomicity violation.

          Show
          Jonathan Hsieh added a comment - One more piece: Mechanism to atomically rollback if a partial failures encountered when attempting to bulk load multiple families. For example, let's say I want to bulk load a region with cfs A, B, C. I issue a call to an RS region to atomically bulkload the HFiles. The RS loads A and B successfully but fails on C (hdfs failure, or rs goes down, etc). We should rollback A and B – if we don't we would have A and B loaded but not C and have an atomicity violation.
          Hide
          Ted Yu added a comment -

          In case of such faulty condition (hdfs failure), would it be easier if we record which column families encountered error and retry loading them after faulty condition recovers ?

          Show
          Ted Yu added a comment - In case of such faulty condition (hdfs failure), would it be easier if we record which column families encountered error and retry loading them after faulty condition recovers ?
          Hide
          Jonathan Hsieh added a comment -

          If we have an hdfs failure, we won't be able to record or update information about what failed.

          This make me think we need to journal/log the intended atomic actions. Once we have the log, we can act depending on the situation:

          • If we complete successfully, we remove/invalidate log and carry on.
          • If we fail (can't write, rs goes down and restarts), we check to see if everything is in. If it isn't we rollback the subset of hfile loads that had happened. If rollback fails, we still have the log, so we can try later or maybe we kill the RS?

          How about we make this a subtask/follow on jira. The first cut will just detect the situation and log error messages (similar to what currently happens). A follow-on task will discuss and add/implement a recovery mechanism?

          Show
          Jonathan Hsieh added a comment - If we have an hdfs failure, we won't be able to record or update information about what failed. This make me think we need to journal/log the intended atomic actions. Once we have the log, we can act depending on the situation: If we complete successfully, we remove/invalidate log and carry on. If we fail (can't write, rs goes down and restarts), we check to see if everything is in. If it isn't we rollback the subset of hfile loads that had happened. If rollback fails, we still have the log, so we can try later or maybe we kill the RS? How about we make this a subtask/follow on jira. The first cut will just detect the situation and log error messages (similar to what currently happens). A follow-on task will discuss and add/implement a recovery mechanism?
          Hide
          Ted Yu added a comment -

          It is fine to implement recovery in another JIRA.

          Show
          Ted Yu added a comment - It is fine to implement recovery in another JIRA.
          Hide
          Jonathan Hsieh added a comment -

          Created recovery mechanism jira at HBASE-4652

          Show
          Jonathan Hsieh added a comment - Created recovery mechanism jira at HBASE-4652
          Hide
          Jonathan Hsieh added a comment -

          Consolidated patch with HBASE-4649 v3, HBASE-4650 v3, and HBASE-4677 v1.

          Show
          Jonathan Hsieh added a comment - Consolidated patch with HBASE-4649 v3, HBASE-4650 v3, and HBASE-4677 v1.
          Hide
          Jonathan Hsieh added a comment -

          Posted consolidated patch.

          Show
          Jonathan Hsieh added a comment - Posted consolidated patch.
          Hide
          Ted Yu added a comment -

          After applying the consolidated patch, I got:

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile (default-testCompile) on project hbase: Compilation failure: Compilation failure:
          [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java:[190,34] cannot find symbol
          [ERROR] symbol  : method getTestDir(java.lang.String)
          [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility
          [ERROR] 
          [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java:[216,34] cannot find symbol
          [ERROR] symbol  : method getTestDir(java.lang.String)
          [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility
          [ERROR] 
          [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java:[309,36] cannot find symbol
          [ERROR] symbol  : method getTestDir(java.lang.String)
          [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility
          [ERROR] 
          [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java:[123,36] cannot find symbol
          [ERROR] symbol  : method getTestDir(java.lang.String)
          [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility
          

          For TRUNK, there is no such error.

          Show
          Ted Yu added a comment - After applying the consolidated patch, I got: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile ( default -testCompile) on project hbase: Compilation failure: Compilation failure: [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java:[190,34] cannot find symbol [ERROR] symbol : method getTestDir(java.lang. String ) [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility [ERROR] [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java:[216,34] cannot find symbol [ERROR] symbol : method getTestDir(java.lang. String ) [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility [ERROR] [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java:[309,36] cannot find symbol [ERROR] symbol : method getTestDir(java.lang. String ) [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility [ERROR] [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java:[123,36] cannot find symbol [ERROR] symbol : method getTestDir(java.lang. String ) [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility For TRUNK, there is no such error.
          Hide
          Jonathan Hsieh added a comment -

          This was due to HBASE-4634 which got committed two days ago. The old getTestDir was a public method and apparently was just removed. This will probably break on trunk as well.

          https://github.com/apache/hbase/commit/ed21cd6c4c266f610352d76d3d4b6f5cff492a97#src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java

          I think this should be replaced with getDataTestDir calls (thats what the old bulk load test calls to getTestDir were changed to).

          Show
          Jonathan Hsieh added a comment - This was due to HBASE-4634 which got committed two days ago. The old getTestDir was a public method and apparently was just removed. This will probably break on trunk as well. https://github.com/apache/hbase/commit/ed21cd6c4c266f610352d76d3d4b6f5cff492a97#src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java I think this should be replaced with getDataTestDir calls (thats what the old bulk load test calls to getTestDir were changed to).
          Hide
          Jonathan Hsieh added a comment -

          Consolidated patch compatible with trunk/0.92 after HBASE-4634

          Show
          Jonathan Hsieh added a comment - Consolidated patch compatible with trunk/0.92 after HBASE-4634
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501415/hbase-4552.consolidated.v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -166 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestMultiParallel
          org.apache.hadoop.hbase.coprocessor.TestMasterObserver
          org.apache.hadoop.hbase.TestRegionRebalancing
          org.apache.hadoop.hbase.master.TestDefaultLoadBalancer
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/92//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/92//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/92//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501415/hbase-4552.consolidated.v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestMultiParallel org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.master.TestDefaultLoadBalancer org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/92//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/92//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/92//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501415/hbase-4552.consolidated.v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -166 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.coprocessor.TestMasterObserver
          org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
          org.apache.hadoop.hbase.master.TestMasterFailover
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/95//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/95//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/95//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501415/hbase-4552.consolidated.v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/95//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/95//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/95//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501459/hbase-4552.consolidated.v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -166 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
          org.apache.hadoop.hbase.master.TestMasterFailover
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/99//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/99//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/99//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501459/hbase-4552.consolidated.v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/99//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/99//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/99//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Fix HTable creation in tests.

          Show
          Ted Yu added a comment - Fix HTable creation in tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501481/hbase-4552.consolidated.v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -166 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestMasterFailover
          org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting
          org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/101//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/101//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/101//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501481/hbase-4552.consolidated.v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/101//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/101//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/101//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          I verified that except for TestRegionServerCoprocessorExceptionWithXXX tests, the other failures were caused by too many open files.

          Show
          Ted Yu added a comment - I verified that except for TestRegionServerCoprocessorExceptionWithXXX tests, the other failures were caused by too many open files.
          Hide
          ramkrishna.s.vasudevan added a comment -

          In test case TestLoadIncrementalHFilesSplitRecovery
          new HTable(tableName) is getting used.

          +      LOG.info("Creating table " + table);
          +      HTableDescriptor htd = new HTableDescriptor(table);
          

          As part of HBASE-4253 it was found that using new HTableDescriptor(conf, tablename) is the best way. Also check HBASE-4138(comment 25/Aug/11 09:19) for reference.
          This will prevent the failure that happened in
          https://builds.apache.org/job/PreCommit-HBASE-Build/99/testReport/org.apache.hadoop.hbase.mapreduce/TestLoadIncrementalHFilesSplitRecovery/testBulkLoadPhaseRecovery/

          Correct me if am wrong.

          Show
          ramkrishna.s.vasudevan added a comment - In test case TestLoadIncrementalHFilesSplitRecovery new HTable(tableName) is getting used. + LOG.info( "Creating table " + table); + HTableDescriptor htd = new HTableDescriptor(table); As part of HBASE-4253 it was found that using new HTableDescriptor(conf, tablename) is the best way. Also check HBASE-4138 (comment 25/Aug/11 09:19) for reference. This will prevent the failure that happened in https://builds.apache.org/job/PreCommit-HBASE-Build/99/testReport/org.apache.hadoop.hbase.mapreduce/TestLoadIncrementalHFilesSplitRecovery/testBulkLoadPhaseRecovery/ Correct me if am wrong.
          Hide
          Jonathan Hsieh added a comment -

          @Ram, on trunk or 0.92 branches, HTableDescriptor(conf,tablename) doesn't seem to be in the api. In patch v4, it seems like all the HTable constructors have been updated to explicitly take a the configuration reference.

          I'm assuming you meant HTable?

          Show
          Jonathan Hsieh added a comment - @Ram, on trunk or 0.92 branches, HTableDescriptor(conf,tablename) doesn't seem to be in the api. In patch v4, it seems like all the HTable constructors have been updated to explicitly take a the configuration reference. I'm assuming you meant HTable?
          Hide
          Ted Yu added a comment -

          Integrated to 0.92 and TRUNK.

          Thanks for the patch Jonathan.

          Thanks for the review Todd.

          Show
          Ted Yu added a comment - Integrated to 0.92 and TRUNK. Thanks for the patch Jonathan. Thanks for the review Todd.
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Jon
          Yes.. i was wrong...Sorry for that. So may be will chk the reason for failure once again..Thanks Jon

          Show
          ramkrishna.s.vasudevan added a comment - @Jon Yes.. i was wrong...Sorry for that. So may be will chk the reason for failure once again..Thanks Jon
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2392 (See https://builds.apache.org/job/HBase-TRUNK/2392/)
          HBASE-4552 multi-CF bulk load is not atomic across column families (Jonathan Hsieh)

          tedyu :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/src/main/resources/hbase-default.xml
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2392 (See https://builds.apache.org/job/HBase-TRUNK/2392/ ) HBASE-4552 multi-CF bulk load is not atomic across column families (Jonathan Hsieh) tedyu : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/src/main/resources/hbase-default.xml /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #90 (See https://builds.apache.org/job/HBase-0.92/90/)
          HBASE-4552 multi-CF bulk load is not atomic across column families (Jonathan Hsieh)

          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.92/src/main/resources/hbase-default.xml
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #90 (See https://builds.apache.org/job/HBase-0.92/90/ ) HBASE-4552 multi-CF bulk load is not atomic across column families (Jonathan Hsieh) tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.92/src/main/resources/hbase-default.xml /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #119 (See https://builds.apache.org/job/HBase-0.92/119/)
          HBASE-4740 [bulk load] the HBASE-4552 API can't tell if errors on region server are recoverable
          (Jonathan Hsieh)

          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.92/src/main/resources/hbase-default.xml
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #119 (See https://builds.apache.org/job/HBase-0.92/119/ ) HBASE-4740 [bulk load] the HBASE-4552 API can't tell if errors on region server are recoverable (Jonathan Hsieh) tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/resources/hbase-default.xml /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2420 (See https://builds.apache.org/job/HBase-TRUNK/2420/)
          HBASE-4740 [bulk load] the HBASE-4552 API can't tell if errors on region server are recoverable
          (Jonathan Hsieh)

          tedyu :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/src/main/resources/hbase-default.xml
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2420 (See https://builds.apache.org/job/HBase-TRUNK/2420/ ) HBASE-4740 [bulk load] the HBASE-4552 API can't tell if errors on region server are recoverable (Jonathan Hsieh) tedyu : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/resources/hbase-default.xml /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development