HBase
  1. HBase
  2. HBASE-5693

When creating a region, the master initializes it and creates a memstore within the master server

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.94.4, 0.95.0
    • Component/s: master, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I didn't do a complete analysis, but the attached patch saves more than 0.25s for each region creation and locally all the unit tests work.

      1. 5693.v1.patch
        4 kB
        Nicolas Liochon
      2. 5593.v2.patch
        3 kB
        Nicolas Liochon

        Activity

        Hide
        Ted Yu added a comment -

        CreateTableHandler isn't initializing the regions.
        Who will initialize them ?

        Show
        Ted Yu added a comment - CreateTableHandler isn't initializing the regions. Who will initialize them ?
        Hide
        Nicolas Liochon added a comment -

        I didn't look very far in the code. CreateTableHandler is executed on the
        master. It does not need to initialize the memstore & so on.
        The underlying method is called from the region server as well; and here
        the initialization code is called. May be there is some thing more complex
        I didn't see, but at least all the unit tests went well.

        On Sun, Apr 1, 2012 at 5:28 PM, Ted Yu (Commented) (JIRA)

        Show
        Nicolas Liochon added a comment - I didn't look very far in the code. CreateTableHandler is executed on the master. It does not need to initialize the memstore & so on. The underlying method is called from the region server as well; and here the initialization code is called. May be there is some thing more complex I didn't see, but at least all the unit tests went well. On Sun, Apr 1, 2012 at 5:28 PM, Ted Yu (Commented) (JIRA)
        Hide
        Ted Yu added a comment -

        It is called from OpenRegionHandler.openRegion()

        I once made some threads daemon which passed unit tests but resulted in master and region server failing to start.

        Testing on a real cluster is desirable.

        Show
        Ted Yu added a comment - It is called from OpenRegionHandler.openRegion() I once made some threads daemon which passed unit tests but resulted in master and region server failing to start. Testing on a real cluster is desirable.
        Hide
        Ted Yu added a comment -

        @N:
        Can you rebased the patch for trunk ?

        Hunk #3 FAILED at 3613.
        1 out of 3 hunks FAILED -- saving rejects to file src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java.rej
        
        Show
        Ted Yu added a comment - @N: Can you rebased the patch for trunk ? Hunk #3 FAILED at 3613. 1 out of 3 hunks FAILED -- saving rejects to file src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java.rej
        Hide
        Nicolas Liochon added a comment -

        Ok, I will do that + a test on a real cluster.

        On Sun, Apr 1, 2012 at 6:12 PM, Ted Yu (Commented) (JIRA)

        Show
        Nicolas Liochon added a comment - Ok, I will do that + a test on a real cluster. On Sun, Apr 1, 2012 at 6:12 PM, Ted Yu (Commented) (JIRA)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12520822/5693.v1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1358//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520822/5693.v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1358//console This message is automatically generated.
        Hide
        stack added a comment -

        +1 on patch. Its silly we initialize the region over on master on creation.

        CreateTableHandler isn't initializing the regions. Who will initialize them ?

        The regionserver when its assigned a region.

        May be there is some thing more complex I didn't see, but at least all the unit tests went well.

        Nothing complex here.

        Regards test on a real cluster, not necessary on a patch this small. Unit tests run clusters anyways.

        Please make a patch that applies N and run it by hadoopqa. Thanks.

        Show
        stack added a comment - +1 on patch. Its silly we initialize the region over on master on creation. CreateTableHandler isn't initializing the regions. Who will initialize them ? The regionserver when its assigned a region. May be there is some thing more complex I didn't see, but at least all the unit tests went well. Nothing complex here. Regards test on a real cluster, not necessary on a patch this small. Unit tests run clusters anyways. Please make a patch that applies N and run it by hadoopqa. Thanks.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12520876/5593.v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestFromClientSide
        org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
        org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1364//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1364//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1364//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520876/5593.v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1364//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1364//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1364//console This message is automatically generated.
        Hide
        stack added a comment -

        The failures because of your patch N?

        Show
        stack added a comment - The failures because of your patch N?
        Hide
        Nicolas Liochon added a comment -

        I don't think so. I didn't see them locally.

        Show
        Nicolas Liochon added a comment - I don't think so. I didn't see them locally.
        Hide
        stack added a comment -

        I tried the first three locally. They pass for me. Committed trunk. Thanks for the patch N.

        Show
        stack added a comment - I tried the first three locally. They pass for me. Committed trunk. Thanks for the patch N.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2704 (See https://builds.apache.org/job/HBase-TRUNK/2704/)
        HBASE-5693 When creating a region, the master initializes it and creates a memstore within the master server (Revision 1308535)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2704 (See https://builds.apache.org/job/HBase-TRUNK/2704/ ) HBASE-5693 When creating a region, the master initializes it and creates a memstore within the master server (Revision 1308535) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Lars Hofhansl added a comment -

        For 0.94?

        Show
        Lars Hofhansl added a comment - For 0.94?
        Hide
        stack added a comment -

        You could but benefit is little, probably not worth the risk.

        Show
        stack added a comment - You could but benefit is little, probably not worth the risk.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #156 (See https://builds.apache.org/job/HBase-TRUNK-security/156/)
        HBASE-5693 When creating a region, the master initializes it and creates a memstore within the master server (Revision 1308535)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #156 (See https://builds.apache.org/job/HBase-TRUNK-security/156/ ) HBASE-5693 When creating a region, the master initializes it and creates a memstore within the master server (Revision 1308535) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        stack added a comment -

        Committed to 0.94 as part of HBASE-7253

        Show
        stack added a comment - Committed to 0.94 as part of HBASE-7253
        Hide
        stack added a comment -

        Committed to 0.94 as part of HBASE-7253

        Show
        stack added a comment - Committed to 0.94 as part of HBASE-7253
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #611 (See https://builds.apache.org/job/HBase-0.94/611/)
        HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616, HBASE-5693, HBASE-6327, HBASE-7253 (Revision 1417559)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #611 (See https://builds.apache.org/job/HBase-0.94/611/ ) HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616 , HBASE-5693 , HBASE-6327 , HBASE-7253 (Revision 1417559) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #86 (See https://builds.apache.org/job/HBase-0.94-security/86/)
        HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616, HBASE-5693, HBASE-6327, HBASE-7253 (Revision 1417559)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #86 (See https://builds.apache.org/job/HBase-0.94-security/86/ ) HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616 , HBASE-5693 , HBASE-6327 , HBASE-7253 (Revision 1417559) Result = SUCCESS stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/)
        HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616, HBASE-5693, HBASE-6327, HBASE-7253 (Revision 1417559)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/ ) HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616 , HBASE-5693 , HBASE-6327 , HBASE-7253 (Revision 1417559) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java

          People

          • Assignee:
            Nicolas Liochon
            Reporter:
            Nicolas Liochon
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development