HBase
  1. HBase
  2. HBASE-11007

BLOCKCACHE in schema descriptor seems not aptly named

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.94.19
    • Fix Version/s: 0.99.0
    • Component/s: None
    • Labels:
      None

      Description

      Hi,

      It seems that setting BLOCKCACHE key to false will disable the Data blocks from being cached but will continue to cache bloom and index blocks. This same property seems to be called cacheDataOnRead inside CacheConfig.

      Should this be called CACHE_DATA_ON_READ instead of BLOCKCACHE similar to the other CACHE_DATA_ON_WRITE/CACHE_INDEX_ON_WRITE. We got quite confused and ended up adding our own property CACHE_DATA_ON_READ - we also added some unit tests for the same.

      What do folks think about this ?

      Thanks
      Varun

      1. 11007.txt
        11 kB
        stack
      2. 11007v2.txt
        9 kB
        stack

        Activity

        Hide
        Lars Hofhansl added a comment -

        Debatable. There aren't many index/bloomfilter blocks and not caching those would lead to truly terrible performance. Hence the only useful option is whether enable or disable caching of data blocks.
        We can't rename those without backward compatibility issues.

        Show
        Lars Hofhansl added a comment - Debatable. There aren't many index/bloomfilter blocks and not caching those would lead to truly terrible performance. Hence the only useful option is whether enable or disable caching of data blocks. We can't rename those without backward compatibility issues.
        Hide
        Varun Sharma added a comment -

        I agree on the backward compatibility issue. How about at least adding a test, as part of this issue, on this property which exercises the code path and makes sure that it only touches Data Blocks ? I found several tests on CACHE_DATA_ON_WRITE etc. but none for this option ? Did I miss the unit test by any chance ?

        Thanks !
        Varun

        Show
        Varun Sharma added a comment - I agree on the backward compatibility issue. How about at least adding a test, as part of this issue, on this property which exercises the code path and makes sure that it only touches Data Blocks ? I found several tests on CACHE_DATA_ON_WRITE etc. but none for this option ? Did I miss the unit test by any chance ? Thanks ! Varun
        Hide
        Lars Hofhansl added a comment -

        There's TestForceCacheImportantBlocks, which seems to test that... At least in 0.94.

        The test exists in trunk, however it looks like the test is not actually verifying anything anymore.
        stack, Andrew Purtell, FYI. Looks like the test should either be fixed to verify something or be removed.

        Show
        Lars Hofhansl added a comment - There's TestForceCacheImportantBlocks, which seems to test that... At least in 0.94. The test exists in trunk, however it looks like the test is not actually verifying anything anymore. stack , Andrew Purtell , FYI. Looks like the test should either be fixed to verify something or be removed.
        Hide
        Nick Dimiduk added a comment -

        On 0.94, this test uses SchemaMetrics, which was removed as part of HBASE-6410.

        Show
        Nick Dimiduk added a comment - On 0.94, this test uses SchemaMetrics, which was removed as part of HBASE-6410 .
        Hide
        stack added a comment -

        Changed javadoc for BLOCKCACHE and for its methods to explain that a better name would have been CACHE_DATA_ON_READ explaining that this attribute enables DATA block caching, yes/no.

        The TestForceCacheImportantBlocks in trunk was testing nothing (since removal of SchemaMetrics). Cache stats are opaque on whether DATA or META (INDEX/BLOOM). Let me fix that elsewhere. Meantime made TestForceCacheImportantBlocks do a very basic verification that when BLOCKCACHE is on/off, that the miss count reflects at least a difference.

        Show
        stack added a comment - Changed javadoc for BLOCKCACHE and for its methods to explain that a better name would have been CACHE_DATA_ON_READ explaining that this attribute enables DATA block caching, yes/no. The TestForceCacheImportantBlocks in trunk was testing nothing (since removal of SchemaMetrics). Cache stats are opaque on whether DATA or META (INDEX/BLOOM). Let me fix that elsewhere. Meantime made TestForceCacheImportantBlocks do a very basic verification that when BLOCKCACHE is on/off, that the miss count reflects at least a difference.
        Hide
        stack added a comment -

        Making the patch for trunk. I could make a patch for 0.94 to backport the javadoc piece (since it already has a test, a test that is better than what I'm adding to trunk). You want it Lars Hofhansl

        Show
        stack added a comment - Making the patch for trunk. I could make a patch for 0.94 to backport the javadoc piece (since it already has a test, a test that is better than what I'm adding to trunk). You want it Lars Hofhansl
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12645430/11007.txt
        against trunk revision .
        ATTACHMENT ID: 12645430

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9533//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645430/11007.txt against trunk revision . ATTACHMENT ID: 12645430 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9533//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12645436/11007v2.txt
        against trunk revision .
        ATTACHMENT ID: 12645436

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645436/11007v2.txt against trunk revision . ATTACHMENT ID: 12645436 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9538//console This message is automatically generated.
        Hide
        stack added a comment -

        Committed these javadoc and minor refactor of a test to 0.98 and to trunk.

        Show
        stack added a comment - Committed these javadoc and minor refactor of a test to 0.98 and to trunk.
        Hide
        stack added a comment -

        Reverted from 0.98. Some issue in the test. Will work on it if wanted.

        Show
        stack added a comment - Reverted from 0.98. Some issue in the test. Will work on it if wanted.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #309 (See https://builds.apache.org/job/HBase-0.98/309/)
        HBASE-11007 BLOCKCACHE in schema descriptor seems not aptly named; REVERT (stack: rev 1595942)

        • /hbase/branches/0.98/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
          HBASE-11007 BLOCKCACHE in schema descriptor seems not aptly named (stack: rev 1595939)
        • /hbase/branches/0.98/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #309 (See https://builds.apache.org/job/HBase-0.98/309/ ) HBASE-11007 BLOCKCACHE in schema descriptor seems not aptly named; REVERT (stack: rev 1595942) /hbase/branches/0.98/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java HBASE-11007 BLOCKCACHE in schema descriptor seems not aptly named (stack: rev 1595939) /hbase/branches/0.98/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #5135 (See https://builds.apache.org/job/HBase-TRUNK/5135/)
        HBASE-11007 BLOCKCACHE in schema descriptor seems not aptly named (stack: rev 1595938)

        • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5135 (See https://builds.apache.org/job/HBase-TRUNK/5135/ ) HBASE-11007 BLOCKCACHE in schema descriptor seems not aptly named (stack: rev 1595938) /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
        Hide
        Enis Soztutar added a comment -

        Closing this issue after 0.99.0 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

          People

          • Assignee:
            Varun Sharma
            Reporter:
            Varun Sharma
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development