HBase
  1. HBase
  2. HBASE-6853

IllegalArgumentException is thrown when an empty region is splitted

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.1, 0.94.1
    • Fix Version/s: 0.94.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is w.r.t a mail sent in the dev mail list.

      Empty region split should be handled gracefully. Either we should not allow the split to happen if we know that the region is empty or we should allow the split to happen by setting the no of threads to the thread pool executor as 1.

      int nbFiles = hstoreFilesToSplit.size();
      ThreadFactoryBuilder builder = new ThreadFactoryBuilder();
          builder.setNameFormat("StoreFileSplitter-%1$d");
          ThreadFactory factory = builder.build();
          ThreadPoolExecutor threadPool =
            (ThreadPoolExecutor) Executors.newFixedThreadPool(nbFiles, factory);
          List<Future<Void>> futures = new ArrayList<Future<Void>>(nbFiles);
      
      

      Here the nbFiles needs to be a non zero positive value.

      1. HBASE-6853_splitfailure.patch
        3 kB
        Priyadarshini
      2. HBASE-6853_2_splitsuccess.patch
        4 kB
        Priyadarshini
      3. HBASE-6853.patch
        4 kB
        ramkrishna.s.vasudevan
      4. HBASE-6853_addendum.patch
        0.8 kB
        ramkrishna.s.vasudevan
      5. HBASE-6853_0.94
        1 kB
        ramkrishna.s.vasudevan

        Activity

        Hide
        Priyadarshini added a comment -

        Attached 2 patches refers to 2 scenarios.

        SCENARIO 1 (success) :
        ---------------------
        Making the split to succeed by setting the number of threads for the FixedThreadPool to 1 even if the hstoreFilesToSplit size is zero.

        SCENARIO 2 (failure) :
        ----------------------
        Making the split to fail when there are no store files to split.

        Show
        Priyadarshini added a comment - Attached 2 patches refers to 2 scenarios. SCENARIO 1 (success) : --------------------- Making the split to succeed by setting the number of threads for the FixedThreadPool to 1 even if the hstoreFilesToSplit size is zero. SCENARIO 2 (failure) : ---------------------- Making the split to fail when there are no store files to split.
        Hide
        stack added a comment -

        Nice patches and tests Priya. Scenario #2 seems cleanest (thats patch #1, right)? What do you think?

        Show
        stack added a comment - Nice patches and tests Priya. Scenario #2 seems cleanest (thats patch #1, right)? What do you think?
        Hide
        Priyadarshini added a comment -

        Yes stack..
        We can fail the split if there are no store files to split..

        Show
        Priyadarshini added a comment - Yes stack.. We can fail the split if there are no store files to split..
        Hide
        ramkrishna.s.vasudevan added a comment -

        @Stack
        If patch 1 is fine, can we commit it?

        Show
        ramkrishna.s.vasudevan added a comment - @Stack If patch 1 is fine, can we commit it?
        Hide
        ramkrishna.s.vasudevan added a comment -

        @Stack
        Can we commit patch 1?

        Show
        ramkrishna.s.vasudevan added a comment - @Stack Can we commit patch 1?
        Hide
        stack added a comment -

        ramkrishna.s.vasudevan Yes. +1 on HBASE-6853_splitfailure.patch

        Show
        stack added a comment - ramkrishna.s.vasudevan Yes. +1 on HBASE-6853 _splitfailure.patch
        Hide
        ramkrishna.s.vasudevan added a comment -

        Hadoop QA trial

        Show
        ramkrishna.s.vasudevan added a comment - Hadoop QA trial
        Hide
        ramkrishna.s.vasudevan added a comment -

        Patch for 0.94. Wanted to rebase the patch for trunk but am getting some compilation error. Will do it after resolving.

        Show
        ramkrishna.s.vasudevan added a comment - Patch for 0.94. Wanted to rebase the patch for trunk but am getting some compilation error. Will do it after resolving.
        Hide
        stack added a comment -

        ramkrishna.s.vasudevan Sorry Ram. I broke the build. Fixed now.

        Show
        stack added a comment - ramkrishna.s.vasudevan Sorry Ram. I broke the build. Fixed now.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Committed to trunk and 0.96. Thanks for the patch Priya.
        Thanks for the review Stack.

        Show
        ramkrishna.s.vasudevan added a comment - Committed to trunk and 0.96. Thanks for the patch Priya. Thanks for the review Stack.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3394 (See https://builds.apache.org/job/HBase-TRUNK/3394/)
        HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391759)

        Result = FAILURE
        ramkrishna :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3394 (See https://builds.apache.org/job/HBase-TRUNK/3394/ ) HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391759) Result = FAILURE ramkrishna : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #497 (See https://builds.apache.org/job/HBase-0.94/497/)
        HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391760)

        Result = SUCCESS
        ramkrishna :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #497 (See https://builds.apache.org/job/HBase-0.94/497/ ) HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391760) Result = SUCCESS ramkrishna : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #199 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/199/)
        HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391759)

        Result = FAILURE
        ramkrishna :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #199 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/199/ ) HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391759) Result = FAILURE ramkrishna : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #58 (See https://builds.apache.org/job/HBase-0.94-security/58/)
        HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391760)

        Result = SUCCESS
        ramkrishna :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #58 (See https://builds.apache.org/job/HBase-0.94-security/58/ ) HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391760) Result = SUCCESS ramkrishna : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #8 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/8/)
        HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391760)

        Result = FAILURE
        ramkrishna :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #8 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/8/ ) HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Priya) (Revision 1391760) Result = FAILURE ramkrishna : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Hide
        Lars Hofhansl added a comment -

        testShouldThrowIOExceptionIfStoreFileSizeIsEmptyAndSHouldSuccessfullyExecuteRollback fails very frequently in 0.94 builds.

        Show
        Lars Hofhansl added a comment - testShouldThrowIOExceptionIfStoreFileSizeIsEmptyAndSHouldSuccessfullyExecuteRollback fails very frequently in 0.94 builds.
        Hide
        Lars Hofhansl added a comment -

        nm, I think these are related to env issues on the ubuntu2 jenkins machine. Sorry for the noise

        Show
        Lars Hofhansl added a comment - nm, I think these are related to env issues on the ubuntu2 jenkins machine. Sorry for the noise
        Show
        Lars Hofhansl added a comment - Actually, I do see the test failing with assertions every (without environment issues): https://builds.apache.org/job/HBase-0.94/533/testReport/org.apache.hadoop.hbase.regionserver/TestSplitTransactionOnCluster/testShouldThrowIOExceptionIfStoreFileSizeIsEmptyAndSHouldSuccessfullyExecuteRollback/ https://builds.apache.org/job/HBase-0.94/532/testReport/junit/org.apache.hadoop.hbase.regionserver/TestSplitTransactionOnCluster/testShouldThrowIOExceptionIfStoreFileSizeIsEmptyAndSHouldSuccessfullyExecuteRollback/ Ram or Priyadarshini: Do you think you have time to look at these (it's not blocking the RC, I think, so it's not urgent)?
        Hide
        ramkrishna.s.vasudevan added a comment -

        @Lars
        Locally i once again verified its running fine. But i can think of one possible reason for the failure in Jenkins. I can give an addendum. And its not the functionality introduced by this test case is failing, it is a general behaviour where we expect a region to be created but may be in the Jenkins it takes more time and not getting created yet.

        Show
        ramkrishna.s.vasudevan added a comment - @Lars Locally i once again verified its running fine. But i can think of one possible reason for the failure in Jenkins. I can give an addendum. And its not the functionality introduced by this test case is failing, it is a general behaviour where we expect a region to be created but may be in the Jenkins it takes more time and not getting created yet.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Addendum attached. This should avoid the failure in Jenkins builds.

        Show
        ramkrishna.s.vasudevan added a comment - Addendum attached. This should avoid the failure in Jenkins builds.
        Hide
        Lars Hofhansl added a comment -

        I thought HBaseAdmin.createTable was a synchronous operation. Like you, I would have expected the region to have been create when it returns.

        +1 on Addendum. Thanks Ram!

        Show
        Lars Hofhansl added a comment - I thought HBaseAdmin.createTable was a synchronous operation. Like you, I would have expected the region to have been create when it returns. +1 on Addendum. Thanks Ram!
        Hide
        Lars Hofhansl added a comment -

        Should we commit the addendum? (Also should we limit the waiting, or just wait until the test is killed?)

        Show
        Lars Hofhansl added a comment - Should we commit the addendum? (Also should we limit the waiting, or just wait until the test is killed?)
        Hide
        stack added a comment -

        +1 on committing addendum and +1 on adding a wait time on the test.

        Show
        stack added a comment - +1 on committing addendum and +1 on adding a wait time on the test.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Ok, I shall commit this later today with a wait time on test.
        @Lars/Stack
        You mind committing it?

        Show
        ramkrishna.s.vasudevan added a comment - Ok, I shall commit this later today with a wait time on test. @Lars/Stack You mind committing it?
        Hide
        ramkrishna.s.vasudevan added a comment -

        This is what i will be committing.

        Show
        ramkrishna.s.vasudevan added a comment - This is what i will be committing.
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #535 (See https://builds.apache.org/job/HBase-0.94/535/)
        HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Ram) : Addendum for testcase failure (Revision 1396708)

        Result = SUCCESS
        ramkrishna :
        Files :

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #535 (See https://builds.apache.org/job/HBase-0.94/535/ ) HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Ram) : Addendum for testcase failure (Revision 1396708) Result = SUCCESS ramkrishna : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/)
        HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Ram) : Addendum for testcase failure (Revision 1396708)

        Result = FAILURE
        ramkrishna :
        Files :

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/ ) HBASE-6853 IllegalArgument Exception is thrown when an empty region is spliitted(Ram) : Addendum for testcase failure (Revision 1396708) Result = FAILURE ramkrishna : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
        Hide
        stack added a comment -

        Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl)

        Show
        stack added a comment - Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl)

          People

          • Assignee:
            Priyadarshini
            Reporter:
            ramkrishna.s.vasudevan
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development