HBase
  1. HBase
  2. HBASE-4932

Block cache can be mistakenly instantiated by tools

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None

      Description

      Map Reduce tasks that create a writer to write HFiles inadvertently end up creating block cache.

      1. HBASE-4932.patch
        7 kB
        Nicolas Spiegelberg

        Activity

        Hide
        stack added a comment -

        Yeah. This is actually a pain. Master splitting WALs used to be creating blockcaches until ugly-hacked it not too.

        Show
        stack added a comment - Yeah. This is actually a pain. Master splitting WALs used to be creating blockcaches until ugly-hacked it not too.
        Hide
        Nicolas Spiegelberg added a comment -

        Phabricator is being a little buggy. Working with Marek to diagnose the problem. See https://reviews.facebook.net/D573

        Show
        Nicolas Spiegelberg added a comment - Phabricator is being a little buggy. Working with Marek to diagnose the problem. See https://reviews.facebook.net/D573
        Hide
        Nicolas Spiegelberg added a comment -

        looks good for now. sounds like we need to clean up the caching semantics for a writer in trunk :\ Save that for another JIRA

        Show
        Nicolas Spiegelberg added a comment - looks good for now. sounds like we need to clean up the caching semantics for a writer in trunk :\ Save that for another JIRA
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12506185/HBASE-4932.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 25 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/449//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506185/HBASE-4932.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 25 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/449//console This message is automatically generated.
        Hide
        stack added a comment -

        I was going to say this is probably really expensive but its just cloning of in memory Arrays and Properties so probably not too bad:

        -  public static final WriterFactory getWriterFactory(Configuration conf) {
        -    return HFile.getWriterFactory(conf, new CacheConfig(conf));
        +  public static final WriterFactory getWriterFactoryNoCache(Configuration
        +       conf) {
        +    Configuration tempConf = new Configuration(conf);
        +    tempConf.setFloat(CacheConfig.HFILE_BLOCK_CACHE_SIZE_KEY, 0.0f);
        +    return HFile.getWriterFactory(conf, new CacheConfig(tempConf));
        

        +1 on patch.

        Show
        stack added a comment - I was going to say this is probably really expensive but its just cloning of in memory Arrays and Properties so probably not too bad: - public static final WriterFactory getWriterFactory(Configuration conf) { - return HFile.getWriterFactory(conf, new CacheConfig(conf)); + public static final WriterFactory getWriterFactoryNoCache(Configuration + conf) { + Configuration tempConf = new Configuration(conf); + tempConf.setFloat(CacheConfig.HFILE_BLOCK_CACHE_SIZE_KEY, 0.0f); + return HFile.getWriterFactory(conf, new CacheConfig(tempConf)); +1 on patch.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2519 (See https://builds.apache.org/job/HBase-TRUNK/2519/)
        HBASE-4932 Block cache can be mistakenly instantiated by tools

        Summary: disable block cache for HFile writers created for tools

        Test Plan:
        ran the following tests that call getWriterFactory

        test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
        test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
        test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
        test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
        test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java

        Reviewers: dhruba, jgray, nspiegelberg, JIRA, mbautin

        Reviewed By: nspiegelberg

        CC: HBase Diffs Facebook Group, mbautin, khemani, nspiegelberg

        Differential Revision: 573

        nspiegelberg :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2519 (See https://builds.apache.org/job/HBase-TRUNK/2519/ ) HBASE-4932 Block cache can be mistakenly instantiated by tools Summary: disable block cache for HFile writers created for tools Test Plan: ran the following tests that call getWriterFactory test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java Reviewers: dhruba, jgray, nspiegelberg, JIRA, mbautin Reviewed By: nspiegelberg CC: HBase Diffs Facebook Group, mbautin, khemani, nspiegelberg Differential Revision: 573 nspiegelberg : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #23 (See https://builds.apache.org/job/HBase-TRUNK-security/23/)
        HBASE-4932 Block cache can be mistakenly instantiated by tools

        Summary: disable block cache for HFile writers created for tools

        Test Plan:
        ran the following tests that call getWriterFactory

        test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
        test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
        test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
        test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
        test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
        test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java

        Reviewers: dhruba, jgray, nspiegelberg, JIRA, mbautin

        Reviewed By: nspiegelberg

        CC: HBase Diffs Facebook Group, mbautin, khemani, nspiegelberg

        Differential Revision: 573

        nspiegelberg :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #23 (See https://builds.apache.org/job/HBase-TRUNK-security/23/ ) HBASE-4932 Block cache can be mistakenly instantiated by tools Summary: disable block cache for HFile writers created for tools Test Plan: ran the following tests that call getWriterFactory test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java Reviewers: dhruba, jgray, nspiegelberg, JIRA, mbautin Reviewed By: nspiegelberg CC: HBase Diffs Facebook Group, mbautin, khemani, nspiegelberg Differential Revision: 573 nspiegelberg : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12506185/HBASE-4932.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 25 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/511//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506185/HBASE-4932.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 25 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/511//console This message is automatically generated.
        Hide
        Mikhail Bautin added a comment -

        @Prakash: I've noticed that this is not there in 89-fb. Do we need this patch there as well?

        Show
        Mikhail Bautin added a comment - @Prakash: I've noticed that this is not there in 89-fb. Do we need this patch there as well?
        Hide
        Prakash Khemani added a comment -

        Yes ... It is a good to have patch. Thanks.

        On 2/24/12 12:03 PM, "Mikhail Bautin (Commented) (JIRA)" <jira@apache.org>

        Show
        Prakash Khemani added a comment - Yes ... It is a good to have patch. Thanks. On 2/24/12 12:03 PM, "Mikhail Bautin (Commented) (JIRA)" <jira@apache.org>
        Hide
        Lars Hofhansl added a comment -

        Is this ready for commit to 0.94?

        Show
        Lars Hofhansl added a comment - Is this ready for commit to 0.94?
        Hide
        Lars Hofhansl added a comment -

        Ah, NM, was already committed, just not marked fixed.

        Show
        Lars Hofhansl added a comment - Ah, NM, was already committed, just not marked fixed.

          People

          • Assignee:
            Prakash Khemani
            Reporter:
            Prakash Khemani
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development