HBase
  1. HBase
  2. HBASE-4027

Enable direct byte buffers LruBlockCache

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Setting hbase.offheapcachesize in hbase-site.xml and -XX:MaxDirectMemorySize in hbase-env.sh to enable this feature. The file already has a line you can uncomment and you need to set the size of the direct memory (your total memory - size allocated to memstores - size allocated to the normal block cache - and some head room for the other functionalities).
      Show
      Setting hbase.offheapcachesize in hbase-site.xml and -XX:MaxDirectMemorySize in hbase-env.sh to enable this feature. The file already has a line you can uncomment and you need to set the size of the direct memory (your total memory - size allocated to memstores - size allocated to the normal block cache - and some head room for the other functionalities).

      Description

      Java offers the creation of direct byte buffers which are allocated outside of the heap.

      They need to be manually free'd, which can be accomplished using an documented clean method.

      The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.

      1. hbase-4027v15.3.diff
        91 kB
        Li Pi
      2. hbase4027v15.2.diff
        91 kB
        Li Pi
      3. hbase-4027v13.1.diff
        83 kB
        Li Pi
      4. hbase4027v12.1.diff
        108 kB
        Li Pi
      5. hbase4027v12.diff
        113 kB
        Li Pi
      6. hbase4027v11.7.diff
        68 kB
        Li Pi
      7. hbase4027v11.6.diff
        68 kB
        Li Pi
      8. hbase4027v11.5.diff
        68 kB
        Li Pi
      9. hbase4027v11.diff
        68 kB
        Li Pi
      10. hbase-4027v10.6.diff
        68 kB
        Li Pi
      11. hbase-4027-v10.5.diff
        62 kB
        Li Pi
      12. hbase-4027-v10.diff
        155 kB
        Li Pi
      13. HBase4027v9.diff
        56 kB
        Li Pi
      14. HBase4027v8.diff
        56 kB
        Li Pi
      15. 4027v7.diff
        52 kB
        Li Pi
      16. HBase-4027 (1).pdf
        54 kB
        Li Pi
      17. hbase-4027v6.diff
        50 kB
        Li Pi
      18. HBase-4027.pdf
        54 kB
        Li Pi
      19. 4027-v5.diff
        50 kB
        Li Pi
      20. slabcachepatchv4.5.diff
        38 kB
        Li Pi
      21. slabcachepatchv4.diff
        46 kB
        Li Pi
      22. slabcachepatchv3.2.diff
        37 kB
        Li Pi
      23. slabcachepatchv3.1.diff
        37 kB
        Li Pi
      24. slabcachepatchv3.diff
        34 kB
        Li Pi
      25. slabcachepatchv2.diff
        33 kB
        Li Pi
      26. slabcachepatch.diff
        34 kB
        Li Pi
      There are no Sub-Tasks for this issue.

        Activity

        Hide
        Lars Francke added a comment -

        This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

        Show
        Lars Francke added a comment - This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).
        Hide
        Jean-Daniel Cryans added a comment -

        I changed the release note, the patch doesn't have a "hbase.offheapcachesize" configuration and it's enabled as soon as you set -XX:MaxDirectMemorySize (which is actually a big problem when you consider this: http://hbase.apache.org/book.html#trouble.client.oome.directmemory.leak). I'm opening followup jiras.

        Show
        Jean-Daniel Cryans added a comment - I changed the release note, the patch doesn't have a "hbase.offheapcachesize" configuration and it's enabled as soon as you set -XX:MaxDirectMemorySize (which is actually a big problem when you consider this: http://hbase.apache.org/book.html#trouble.client.oome.directmemory.leak ). I'm opening followup jiras.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2172 (See https://builds.apache.org/job/HBase-TRUNK/2172/)
        HBASE-4027 Off Heap Cache never creates Slabs

        stack :
        Files :

        • /hbase/trunk/CHANGES.txt
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2172 (See https://builds.apache.org/job/HBase-TRUNK/2172/ ) HBASE-4027 Off Heap Cache never creates Slabs stack : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        Hide
        stack added a comment -

        I get this in TRUNK at mo:

        Running org.apache.hadoop.hbase.client.TestAdmin
        Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 227.229 sec <<< FAILURE!
        

        I'm taking a looksee.

        Show
        stack added a comment - I get this in TRUNK at mo: Running org.apache.hadoop.hbase.client.TestAdmin Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 227.229 sec <<< FAILURE! I'm taking a looksee.
        Hide
        Li Pi added a comment -

        Huh:

        Running org.apache.hadoop.hbase.client.TestAdmin

        Results :

        Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

        [INFO]
        ------------------------------------------------------------------------
        [INFO] BUILD FAILURE

        without the patch. Tests don't run?

        Show
        Li Pi added a comment - Huh: Running org.apache.hadoop.hbase.client.TestAdmin Results : Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE without the patch. Tests don't run?
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2147 (See https://builds.apache.org/job/HBase-TRUNK/2147/)
        HBASE-4027 Enable direct byte buffers LruBlockCache

        stack :
        Files :

        • /hbase/trunk/CHANGES.txt
        • /hbase/trunk/conf/hbase-env.sh
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2147 (See https://builds.apache.org/job/HBase-TRUNK/2147/ ) HBASE-4027 Enable direct byte buffers LruBlockCache stack : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/conf/hbase-env.sh /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
        Hide
        Li Pi added a comment -

        I'm running a test against the previous version without 4027. I'll let you know what fails.

        Show
        Li Pi added a comment - I'm running a test against the previous version without 4027. I'll let you know what fails.
        Hide
        stack added a comment -

        Before commit I ran tests.... about 2/3rds completed. TestAdmin failed but figured that not because of this patch.

        Show
        stack added a comment - Before commit I ran tests.... about 2/3rds completed. TestAdmin failed but figured that not because of this patch.
        Hide
        stack added a comment -

        Thank you for the fat feature Li. Thanks to all the reviewers too. Committed to TRUNK.

        Show
        stack added a comment - Thank you for the fat feature Li. Thanks to all the reviewers too. Committed to TRUNK.
        Hide
        Li Pi added a comment -

        final one!

        Show
        Li Pi added a comment - final one!
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-26 06:39:36, Michael Stack wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 1

        > <https://reviews.apache.org/r/1214/diff/25/?file=34984#file34984line1>

        >

        > When would we have an instance of Cacheable where we did not need to serialize AND deserialize? (Wondering if worth having this extra CacheableDeserializer)

        If you wanted something to only reside in the on heap cache, you'd return 0 for getSerializedLength, and return null for getDeserializer.

        On 2011-08-26 06:39:36, Michael Stack wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 44

        > <https://reviews.apache.org/r/1214/diff/25/?file=34983#file34983line44>

        >

        > Is it ok that "already serialize" and "can't be serialized" return same response out of this method?

        Theres no longer an "already serialized" state. I now remove the object entirely from the heap. I'll edit the comments.

        On 2011-08-26 06:39:36, Michael Stack wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 49

        > <https://reviews.apache.org/r/1214/diff/25/?file=34990#file34990line49>

        >

        > Whats this mean ('an exception will be thrown..') Will we ever try to put something into a block that is larger than the block? Its a programming error if we do? If a serialized object is larger than a 'block', what do we do?

        Its a programming error if we do, we shouldn't ever do it, the sizer should make sure of that. I'll add that to the comments.

        On 2011-08-26 06:39:36, Michael Stack wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 52

        > <https://reviews.apache.org/r/1214/diff/25/?file=34990#file34990line52>

        >

        > Where is this ConcurrentLinkedHashMap? Its not in this class? I see a ConcurrentMap. Its made in the depths of guava?

        Its made by Guava's MapMaker. I'll add it to the comments.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1653
        -----------------------------------------------------------

        On 2011-08-24 04:08:40, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-24 04:08:40)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-26 06:39:36, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 1 > < https://reviews.apache.org/r/1214/diff/25/?file=34984#file34984line1 > > > When would we have an instance of Cacheable where we did not need to serialize AND deserialize? (Wondering if worth having this extra CacheableDeserializer) If you wanted something to only reside in the on heap cache, you'd return 0 for getSerializedLength, and return null for getDeserializer. On 2011-08-26 06:39:36, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 44 > < https://reviews.apache.org/r/1214/diff/25/?file=34983#file34983line44 > > > Is it ok that "already serialize" and "can't be serialized" return same response out of this method? Theres no longer an "already serialized" state. I now remove the object entirely from the heap. I'll edit the comments. On 2011-08-26 06:39:36, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 49 > < https://reviews.apache.org/r/1214/diff/25/?file=34990#file34990line49 > > > Whats this mean ('an exception will be thrown..') Will we ever try to put something into a block that is larger than the block? Its a programming error if we do? If a serialized object is larger than a 'block', what do we do? Its a programming error if we do, we shouldn't ever do it, the sizer should make sure of that. I'll add that to the comments. On 2011-08-26 06:39:36, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 52 > < https://reviews.apache.org/r/1214/diff/25/?file=34990#file34990line52 > > > Where is this ConcurrentLinkedHashMap? Its not in this class? I see a ConcurrentMap. Its made in the depths of guava? Its made by Guava's MapMaker. I'll add it to the comments. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1653 ----------------------------------------------------------- On 2011-08-24 04:08:40, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 04:08:40) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1653
        -----------------------------------------------------------

        Ship it!

        Excellent. Minor queries from a know-nothing below. If you have a mo, make a few answers. You'll need to make a fat release note in the issue so folks get a clue on what this sweet addition is about and how to run it (release note should be good enough one of us can make a short paragraph on this new facility). Good on you Li.

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3691>

        Is it ok that "already serialize" and "can't be serialized" return same response out of this method?

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
        <https://reviews.apache.org/r/1214/#comment3690>

        When would we have an instance of Cacheable where we did not need to serialize AND deserialize? (Wondering if worth having this extra CacheableDeserializer)

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3692>

        Whats this mean ('an exception will be thrown..') Will we ever try to put something into a block that is larger than the block? Its a programming error if we do? If a serialized object is larger than a 'block', what do we do?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3693>

        Where is this ConcurrentLinkedHashMap? Its not in this class? I see a ConcurrentMap. Its made in the depths of guava?

        • Michael

        On 2011-08-24 04:08:40, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-24 04:08:40)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1653 ----------------------------------------------------------- Ship it! Excellent. Minor queries from a know-nothing below. If you have a mo, make a few answers. You'll need to make a fat release note in the issue so folks get a clue on what this sweet addition is about and how to run it (release note should be good enough one of us can make a short paragraph on this new facility). Good on you Li. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3691 > Is it ok that "already serialize" and "can't be serialized" return same response out of this method? src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java < https://reviews.apache.org/r/1214/#comment3690 > When would we have an instance of Cacheable where we did not need to serialize AND deserialize? (Wondering if worth having this extra CacheableDeserializer) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3692 > Whats this mean ('an exception will be thrown..') Will we ever try to put something into a block that is larger than the block? Its a programming error if we do? If a serialized object is larger than a 'block', what do we do? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3693 > Where is this ConcurrentLinkedHashMap? Its not in this class? I see a ConcurrentMap. Its made in the depths of guava? Michael On 2011-08-24 04:08:40, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 04:08:40) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        The latest patch from reviewboard.

        Show
        Li Pi added a comment - The latest patch from reviewboard.
        Hide
        Ted Yu added a comment -

        +1 on latest patch.

        Show
        Ted Yu added a comment - +1 on latest patch.
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-24 04:00:58, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 10

        > <https://reviews.apache.org/r/1214/diff/24/?file=34934#file34934line10>

        >

        > @return should stay with deserialize() method.

        fixed comment.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1614
        -----------------------------------------------------------

        On 2011-08-24 04:08:40, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-24 04:08:40)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-24 04:00:58, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 10 > < https://reviews.apache.org/r/1214/diff/24/?file=34934#file34934line10 > > > @return should stay with deserialize() method. fixed comment. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1614 ----------------------------------------------------------- On 2011-08-24 04:08:40, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 04:08:40) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-24 04:08:40.398370)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed comment.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 04:08:40.398370) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed comment. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1614
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
        <https://reviews.apache.org/r/1214/#comment3647>

        @return should stay with deserialize() method.

        • Ted

        On 2011-08-24 03:57:16, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-24 03:57:16)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1614 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java < https://reviews.apache.org/r/1214/#comment3647 > @return should stay with deserialize() method. Ted On 2011-08-24 03:57:16, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 03:57:16) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-24 03:46:12, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 52

        > <https://reviews.apache.org/r/1214/diff/23/?file=34898#file34898line52>

        >

        > User may get confused by this javadoc since CacheableDeserializer is interface.

        > How about:

        > Returns CacheableDeserializer instance which reconstructs original object from ByteBuffer.

        fixed.

        On 2011-08-24 03:46:12, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 8

        > <https://reviews.apache.org/r/1214/diff/23/?file=34899#file34899line8>

        >

        > This javadoc should be above interface declaration.

        fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1612
        -----------------------------------------------------------

        On 2011-08-24 03:57:16, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-24 03:57:16)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-24 03:46:12, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 52 > < https://reviews.apache.org/r/1214/diff/23/?file=34898#file34898line52 > > > User may get confused by this javadoc since CacheableDeserializer is interface. > How about: > Returns CacheableDeserializer instance which reconstructs original object from ByteBuffer. fixed. On 2011-08-24 03:46:12, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 8 > < https://reviews.apache.org/r/1214/diff/23/?file=34899#file34899line8 > > > This javadoc should be above interface declaration. fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1612 ----------------------------------------------------------- On 2011-08-24 03:57:16, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 03:57:16) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-24 03:57:16.993844)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        addressed ted yu's reviews above.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 03:57:16.993844) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- addressed ted yu's reviews above. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1612
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3644>

        User may get confused by this javadoc since CacheableDeserializer is interface.
        How about:
        Returns CacheableDeserializer instance which reconstructs original object from ByteBuffer.

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
        <https://reviews.apache.org/r/1214/#comment3643>

        This javadoc should be above interface declaration.

        • Ted

        On 2011-08-24 02:46:25, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-24 02:46:25)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1612 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3644 > User may get confused by this javadoc since CacheableDeserializer is interface. How about: Returns CacheableDeserializer instance which reconstructs original object from ByteBuffer. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java < https://reviews.apache.org/r/1214/#comment3643 > This javadoc should be above interface declaration. Ted On 2011-08-24 02:46:25, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 02:46:25) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-19 22:49:55, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 55

        > <https://reviews.apache.org/r/1214/diff/17/?file=33818#file33818line55>

        >

        > does this always return itself? I still think that once something has been serialized, we should throw away the original object, and when we re-instantiate it, it should create a new object.

        Fixed. Changed the interface.

        On 2011-08-19 22:49:55, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1467

        > <https://reviews.apache.org/r/1214/diff/17/?file=33821#file33821line1467>

        >

        > this object confuses me. once we've serialized something, all we should need is the ByteBuffer.

        Fixed it.

        On 2011-08-19 22:49:55, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1518

        > <https://reviews.apache.org/r/1214/diff/17/?file=33821#file33821line1518>

        >

        > this isn't right - you need to take into account arrayOffset also. I think this .equals method should move to the tests, or a static method like blocksAndContentsEqual()

        Fixed. Comparisons are done with if (this.buf.compareTo(castedComparison.buf) != 0) { instead.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1575
        -----------------------------------------------------------

        On 2011-08-24 02:46:25, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-24 02:46:25)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-19 22:49:55, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 55 > < https://reviews.apache.org/r/1214/diff/17/?file=33818#file33818line55 > > > does this always return itself? I still think that once something has been serialized, we should throw away the original object, and when we re-instantiate it, it should create a new object. Fixed. Changed the interface. On 2011-08-19 22:49:55, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1467 > < https://reviews.apache.org/r/1214/diff/17/?file=33821#file33821line1467 > > > this object confuses me. once we've serialized something, all we should need is the ByteBuffer. Fixed it. On 2011-08-19 22:49:55, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1518 > < https://reviews.apache.org/r/1214/diff/17/?file=33821#file33821line1518 > > > this isn't right - you need to take into account arrayOffset also. I think this .equals method should move to the tests, or a static method like blocksAndContentsEqual() Fixed. Comparisons are done with if (this.buf.compareTo(castedComparison.buf) != 0) { instead. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1575 ----------------------------------------------------------- On 2011-08-24 02:46:25, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 02:46:25) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-24 02:46:25.061572)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        updated test case.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 02:46:25.061572) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- updated test case. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 39

        > <https://reviews.apache.org/r/1214/diff/19/?file=34156#file34156line39>

        >

        > i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same

        Li Pi wrote:

        I'm probably gonna change this to the MemCached model, where the slabs are "pages", and blocks are "chunks". Or we can just go our own way.

        We're going to call it Slabs and Blocks. Either way, easy enough change. Will make it consistent.

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 89

        > <https://reviews.apache.org/r/1214/diff/19/?file=34157#file34157line89>

        >

        > javadoc

        Li Pi wrote:

        Will do.

        fixed.

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 28

        > <https://reviews.apache.org/r/1214/diff/19/?file=34158#file34158line28>

        >

        > add @param @return javadoc

        Li Pi wrote:

        Will do.

        Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1594
        -----------------------------------------------------------

        On 2011-08-24 02:09:44, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-24 02:09:44)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 39 > < https://reviews.apache.org/r/1214/diff/19/?file=34156#file34156line39 > > > i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same Li Pi wrote: I'm probably gonna change this to the MemCached model, where the slabs are "pages", and blocks are "chunks". Or we can just go our own way. We're going to call it Slabs and Blocks. Either way, easy enough change. Will make it consistent. On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 89 > < https://reviews.apache.org/r/1214/diff/19/?file=34157#file34157line89 > > > javadoc Li Pi wrote: Will do. fixed. On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 28 > < https://reviews.apache.org/r/1214/diff/19/?file=34158#file34158line28 > > > add @param @return javadoc Li Pi wrote: Will do. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1594 ----------------------------------------------------------- On 2011-08-24 02:09:44, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 02:09:44) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-24 02:09:44.785814)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        added documentation according to jgrays changes.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-24 02:09:44.785814) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- added documentation according to jgrays changes. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-23 23:02:05.583625)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed race condition. rebased on top of another patch. addressed some of jgray's comments.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-23 23:02:05.583625) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed race condition. rebased on top of another patch. addressed some of jgray's comments. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java, line 45

        > <https://reviews.apache.org/r/1214/diff/19/?file=34146#file34146line45>

        >

        > is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated?

        Fixed. Also updated Javadocs in BlockCache.

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 27

        > <https://reviews.apache.org/r/1214/diff/19/?file=34147#file34147line27>

        >

        > extra newline (maybe move it to below the public class).

        >

        > also missing a space after CacheStats

        fixed.

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 55

        > <https://reviews.apache.org/r/1214/diff/19/?file=34151#file34151line55>

        >

        > looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor

        fixed.

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 39

        > <https://reviews.apache.org/r/1214/diff/19/?file=34156#file34156line39>

        >

        > i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same

        I'm probably gonna change this to the MemCached model, where the slabs are "pages", and blocks are "chunks". Or we can just go our own way.

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 89

        > <https://reviews.apache.org/r/1214/diff/19/?file=34157#file34157line89>

        >

        > javadoc

        Will do.

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 28

        > <https://reviews.apache.org/r/1214/diff/19/?file=34158#file34158line28>

        >

        > add @param @return javadoc

        Will do.

        On 2011-08-22 23:32:16, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 93

        > <https://reviews.apache.org/r/1214/diff/19/?file=34152#file34152line93>

        >

        > should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members

        Where should I declare statics? Moving it to the bottom would work.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1594
        -----------------------------------------------------------

        On 2011-08-23 23:02:05, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-23 23:02:05)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java, line 45 > < https://reviews.apache.org/r/1214/diff/19/?file=34146#file34146line45 > > > is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated? Fixed. Also updated Javadocs in BlockCache. On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 27 > < https://reviews.apache.org/r/1214/diff/19/?file=34147#file34147line27 > > > extra newline (maybe move it to below the public class). > > also missing a space after CacheStats fixed. On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 55 > < https://reviews.apache.org/r/1214/diff/19/?file=34151#file34151line55 > > > looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor fixed. On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 39 > < https://reviews.apache.org/r/1214/diff/19/?file=34156#file34156line39 > > > i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same I'm probably gonna change this to the MemCached model, where the slabs are "pages", and blocks are "chunks". Or we can just go our own way. On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 89 > < https://reviews.apache.org/r/1214/diff/19/?file=34157#file34157line89 > > > javadoc Will do. On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 28 > < https://reviews.apache.org/r/1214/diff/19/?file=34158#file34158line28 > > > add @param @return javadoc Will do. On 2011-08-22 23:32:16, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 93 > < https://reviews.apache.org/r/1214/diff/19/?file=34152#file34152line93 > > > should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members Where should I declare statics? Moving it to the bottom would work. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1594 ----------------------------------------------------------- On 2011-08-23 23:02:05, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-23 23:02:05) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1594
        -----------------------------------------------------------

        Looking good! Still some lines > 80 chars and whitespace to fix, but otherwise not much to comment on. As we discussed, I find the Cacheable interface name a bit confusing. Maybe rename it, or just fill the classes and methods with awesome javadoc that explains how it all works

        Also, should probably add a section to the book about this? Some of these parameters are pretty important and the defaults could be wasteful for some of the default use cases (for example, always giving 20% of heap to blocks > 1.1X block size)

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
        <https://reviews.apache.org/r/1214/#comment3591>

        is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated?

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment3592>

        extra newline (maybe move it to below the public class).

        also missing a space after CacheStats

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3594>

        looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3595>

        should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3596>

        is this necessary since we throw a runtime exception? could this leave the RS in a weird state? should we actually halt or abort? or will that happen when this gets thrown?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3599>

        i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3597>

        javadoc

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment3600>

        add @param @return javadoc

        • Jonathan

        On 2011-08-22 23:09:44, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-22 23:09:44)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt 0bb7c65

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 8f5600d

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java a817c37

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d6fa01b

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java cc4abc6

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java f8a854c

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1594 ----------------------------------------------------------- Looking good! Still some lines > 80 chars and whitespace to fix, but otherwise not much to comment on. As we discussed, I find the Cacheable interface name a bit confusing. Maybe rename it, or just fill the classes and methods with awesome javadoc that explains how it all works Also, should probably add a section to the book about this? Some of these parameters are pretty important and the defaults could be wasteful for some of the default use cases (for example, always giving 20% of heap to blocks > 1.1X block size) src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java < https://reviews.apache.org/r/1214/#comment3591 > is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated? src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment3592 > extra newline (maybe move it to below the public class). also missing a space after CacheStats src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3594 > looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3595 > should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3596 > is this necessary since we throw a runtime exception? could this leave the RS in a weird state? should we actually halt or abort? or will that happen when this gets thrown? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3599 > i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3597 > javadoc src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment3600 > add @param @return javadoc Jonathan On 2011-08-22 23:09:44, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-22 23:09:44) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt 0bb7c65 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 8f5600d src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java a817c37 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d6fa01b src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java cc4abc6 src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java f8a854c Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-22 23:09:44.370020)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed concurrency issue. now with readwritelocks.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        CHANGES.txt 0bb7c65
        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 8f5600d
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java a817c37
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d6fa01b
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java cc4abc6
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java f8a854c

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-22 23:09:44.370020) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed concurrency issue. now with readwritelocks. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) CHANGES.txt 0bb7c65 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 8f5600d src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java a817c37 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d6fa01b src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java cc4abc6 src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java f8a854c Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        latest version. new cacheableserialization interface.

        Show
        Li Pi added a comment - latest version. new cacheableserialization interface.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-22 22:01:35.140426)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        added CacheableDeserializer.java.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-22 22:01:35.140426) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- added CacheableDeserializer.java. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-22 21:55:23.810893)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        CacheableSerializer interface added. HFileBlock now fully serialized.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-22 21:55:23.810893) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- CacheableSerializer interface added. HFileBlock now fully serialized. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1585
        -----------------------------------------------------------

        In SingleSizeCache.cacheBlock():
        CacheablePair newEntry = new CacheablePair(
        toBeCached.serialize(storedBlock), storedBlock);
        The above operation splits toBeCached into two parts: the first is for on-heap and is slim, storedBlock is for off-heap.

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3574>

        I think the word 'itself' in the javadoc above introduced confusion. It should be removed.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3573>

        As Pi explained in Cacheable interface, serialize() offloads majority of data to off-heap ByteBuffer. What gets returned is the skeleton that lives on-heap.

        • Ted

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1585 ----------------------------------------------------------- In SingleSizeCache.cacheBlock(): CacheablePair newEntry = new CacheablePair( toBeCached.serialize(storedBlock), storedBlock); The above operation splits toBeCached into two parts: the first is for on-heap and is slim, storedBlock is for off-heap. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3574 > I think the word 'itself' in the javadoc above introduced confusion. It should be removed. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3573 > As Pi explained in Cacheable interface, serialize() offloads majority of data to off-heap ByteBuffer. What gets returned is the skeleton that lives on-heap. Ted On 2011-08-19 20:21:35, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-19 22:49:55, Todd Lipcon wrote:

        >

        I haven't added in the serializer interface yet. That will be there very very soon - and should allow us to reduce the heapsize by another 60 bytes or so per entry.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1575
        -----------------------------------------------------------

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-19 22:49:55, Todd Lipcon wrote: > I haven't added in the serializer interface yet. That will be there very very soon - and should allow us to reduce the heapsize by another 60 bytes or so per entry. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1575 ----------------------------------------------------------- On 2011-08-19 20:21:35, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-19 14:36:26, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 78

        > <https://reviews.apache.org/r/1214/diff/16/?file=33706#file33706line78>

        >

        > The following 10 lines of code is very similar to those starting line 65.

        > Should be able to write another method so that both of them can call.

        fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1549
        -----------------------------------------------------------

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-19 14:36:26, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 78 > < https://reviews.apache.org/r/1214/diff/16/?file=33706#file33706line78 > > > The following 10 lines of code is very similar to those starting line 65. > Should be able to write another method so that both of them can call. fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1549 ----------------------------------------------------------- On 2011-08-19 20:21:35, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1575
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3551>

        does this always return itself? I still think that once something has been serialized, we should throw away the original object, and when we re-instantiate it, it should create a new object.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3553>

        this object confuses me. once we've serialized something, all we should need is the ByteBuffer.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3555>

        this isn't right - you need to take into account arrayOffset also. I think this .equals method should move to the tests, or a static method like blocksAndContentsEqual()

        • Todd

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1575 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3551 > does this always return itself? I still think that once something has been serialized, we should throw away the original object, and when we re-instantiate it, it should create a new object. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3553 > this object confuses me. once we've serialized something, all we should need is the ByteBuffer. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3555 > this isn't right - you need to take into account arrayOffset also. I think this .equals method should move to the tests, or a static method like blocksAndContentsEqual() Todd On 2011-08-19 20:21:35, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1573
        -----------------------------------------------------------

        Ship it!

        All tests passed based on hbase-4027v12.7.diff

        • Ted

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1573 ----------------------------------------------------------- Ship it! All tests passed based on hbase-4027v12.7.diff Ted On 2011-08-19 20:21:35, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-19 13:30:39, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 67

        > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line67>

        >

        > Still some white spaces.

        fixed.

        On 2011-08-19 13:30:39, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 244

        > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line244>

        >

        > This is an important class related to Cacheable. Should have added javadoc for class and methods.

        it's just a pair, added a comment though.

        On 2011-08-19 13:30:39, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 255

        > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line255>

        >

        > REFERENCE refers to serializedData, right ?

        > I think it would be clearer if the order of second and third parameters gets exchanged.

        fixed. added comment.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1546
        -----------------------------------------------------------

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-19 13:30:39, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 67 > < https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line67 > > > Still some white spaces. fixed. On 2011-08-19 13:30:39, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 244 > < https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line244 > > > This is an important class related to Cacheable. Should have added javadoc for class and methods. it's just a pair, added a comment though. On 2011-08-19 13:30:39, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 255 > < https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line255 > > > REFERENCE refers to serializedData, right ? > I think it would be clearer if the order of second and third parameters gets exchanged. fixed. added comment. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1546 ----------------------------------------------------------- On 2011-08-19 20:21:35, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35.572967)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Forgot to override .equals when redoing HFileBlock. Fixed, tests now work.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35.572967) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Forgot to override .equals when redoing HFileBlock. Fixed, tests now work. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1549
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3524>

        The following 10 lines of code is very similar to those starting line 65.
        Should be able to write another method so that both of them can call.

        • Ted

        On 2011-08-19 04:16:44, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 04:16:44)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt 0478003

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1549 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3524 > The following 10 lines of code is very similar to those starting line 65. Should be able to write another method so that both of them can call. Ted On 2011-08-19 04:16:44, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 04:16:44) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt 0478003 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1546
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3522>

        Still some white spaces.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3518>

        This is an important class related to Cacheable. Should have added javadoc for class and methods.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3519>

        REFERENCE refers to serializedData, right ?
        I think it would be clearer if the order of second and third parameters gets exchanged.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment3521>

        Can you explain the meaning of callAssignedCache ?
        It is not obvious by name.

        • Ted

        On 2011-08-19 04:16:44, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 04:16:44)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt 0478003

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1546 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3522 > Still some white spaces. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3518 > This is an important class related to Cacheable. Should have added javadoc for class and methods. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3519 > REFERENCE refers to serializedData, right ? I think it would be clearer if the order of second and third parameters gets exchanged. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment3521 > Can you explain the meaning of callAssignedCache ? It is not obvious by name. Ted On 2011-08-19 04:16:44, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 04:16:44) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt 0478003 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-19 04:10:41, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 49

        > <https://reviews.apache.org/r/1214/diff/15/?file=33671#file33671line49>

        >

        > If self is always returned, why do we need the return value here ?

        it returns a copy of itself, unless it doesn't need deserialization.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1544
        -----------------------------------------------------------

        On 2011-08-19 04:16:44, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 04:16:44)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt 0478003

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-19 04:10:41, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 49 > < https://reviews.apache.org/r/1214/diff/15/?file=33671#file33671line49 > > > If self is always returned, why do we need the return value here ? it returns a copy of itself, unless it doesn't need deserialization. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1544 ----------------------------------------------------------- On 2011-08-19 04:16:44, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 04:16:44) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt 0478003 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-19 04:16:44.682053)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed as per ted yu's comments above.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        CHANGES.txt 0478003
        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 04:16:44.682053) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed as per ted yu's comments above. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) CHANGES.txt 0478003 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1544
        -----------------------------------------------------------

        Cacheable interface is much more intuitive now.

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3516>

        Change an to the.

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3515>

        Do I see an incomplete sentence here ?

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3514>

        If self is always returned, why do we need the return value here ?

        • Ted

        On 2011-08-19 03:05:29, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 03:05:29)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        CHANGES.txt 763ddbc

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313

        src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1544 ----------------------------------------------------------- Cacheable interface is much more intuitive now. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3516 > Change an to the. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3515 > Do I see an incomplete sentence here ? src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3514 > If self is always returned, why do we need the return value here ? Ted On 2011-08-19 03:05:29, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 03:05:29) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d CHANGES.txt 763ddbc conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313 src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39 src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1543
        -----------------------------------------------------------

        Todd: I'll get the fully serialized model out by tonight. This still leaves around 60 bytes of stuff on the heap.

        • Li

        On 2011-08-19 03:05:29, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 03:05:29)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        CHANGES.txt 763ddbc

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313

        src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1543 ----------------------------------------------------------- Todd: I'll get the fully serialized model out by tonight. This still leaves around 60 bytes of stuff on the heap. Li On 2011-08-19 03:05:29, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 03:05:29) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d CHANGES.txt 763ddbc conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313 src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39 src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-19 03:05:29.951253)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Cacheable interface now far less confusing. HFileBlock has reformatting reverted.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        CHANGES.txt 763ddbc
        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313
        src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 03:05:29.951253) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Cacheable interface now far less confusing. HFileBlock has reformatting reverted. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d CHANGES.txt 763ddbc conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313 src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39 src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1525
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3476>

        Whitespace.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3477>

        Whitespace.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3478>

        Auto-formatter weirdness.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3480>

        this and all other formatting fixes should probably be in a seperate jira. filed.

        • Li

        On 2011-08-18 08:31:59, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-18 08:31:59)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1525 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3476 > Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3477 > Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3478 > Auto-formatter weirdness. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3480 > this and all other formatting fixes should probably be in a seperate jira. filed. Li On 2011-08-18 08:31:59, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-18 08:31:59) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 12

        > <https://reviews.apache.org/r/1214/diff/13/?file=32972#file32972line12>

        >

        > Please add javadoc for these methods.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 92

        > <https://reviews.apache.org/r/1214/diff/13/?file=32974#file32974line92>

        >

        > This shows that Cacheable can reside in on-heap cache.

        > The description for Cacheable should be refined.

        Done. Cacheable can reside in either.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 79

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line79>

        >

        > HeapSize is covered by Cacheable so is not needed here.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1528

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1528>

        >

        > Indentation is incorrect here.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1535

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1535>

        >

        > Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means.

        > Please add javadoc.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1540

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1540>

        >

        > Please add javadoc for what this method does.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1541

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1541>

        >

        > Whitespace.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1554

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1554>

        >

        > Indentation.

        Done.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1489
        -----------------------------------------------------------

        On 2011-08-18 08:31:59, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-18 08:31:59)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 12 > < https://reviews.apache.org/r/1214/diff/13/?file=32972#file32972line12 > > > Please add javadoc for these methods. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 92 > < https://reviews.apache.org/r/1214/diff/13/?file=32974#file32974line92 > > > This shows that Cacheable can reside in on-heap cache. > The description for Cacheable should be refined. Done. Cacheable can reside in either. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 79 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line79 > > > HeapSize is covered by Cacheable so is not needed here. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1528 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1528 > > > Indentation is incorrect here. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1535 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1535 > > > Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means. > Please add javadoc. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1540 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1540 > > > Please add javadoc for what this method does. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1541 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1541 > > > Whitespace. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1554 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1554 > > > Indentation. Done. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1489 ----------------------------------------------------------- On 2011-08-18 08:31:59, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-18 08:31:59) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>

        >

        > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000

        Li Pi wrote:

        changed it to returnBlock.put(contentBlock.duplicate()) instead.

        Todd Lipcon wrote:

        don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot?

        Since we duplicate it, we don't need to rewind after using it. The initial copy is already rewound, so we just duplicate it, do our put, and then discard it. TestHammerSingleKey has been written.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45

        > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>

        >

        > hrm, this is identical to the other method?

        Li Pi wrote:

        Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.

        Todd Lipcon wrote:

        but you should be able to extract a method, even if the method has to take a type parameter. too much dup code

        We have a Cacheable interface now. No more dup code.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        On 2011-08-18 08:31:59, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-18 08:31:59)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118 > > > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000 Li Pi wrote: changed it to returnBlock.put(contentBlock.duplicate()) instead. Todd Lipcon wrote: don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot? Since we duplicate it, we don't need to rewind after using it. The initial copy is already rewound, so we just duplicate it, do our put, and then discard it. TestHammerSingleKey has been written. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45 > < https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45 > > > hrm, this is identical to the other method? Li Pi wrote: Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize. Todd Lipcon wrote: but you should be able to extract a method, even if the method has to take a type parameter. too much dup code We have a Cacheable interface now. No more dup code. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- On 2011-08-18 08:31:59, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-18 08:31:59) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-18 08:31:59.585101)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        heapsize added. also addressed ted's review above. fixed formatting in hfileblock. (was inconsistent).

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-18 08:31:59.585101) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- heapsize added. also addressed ted's review above. fixed formatting in hfileblock. (was inconsistent). Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        added heapsize, fixed formatting, addressed ted yu's reviews.

        Show
        Li Pi added a comment - added heapsize, fixed formatting, addressed ted yu's reviews.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1489
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3402>

        Please add javadoc for these methods.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3403>

        This shows that Cacheable can reside in on-heap cache.
        The description for Cacheable should be refined.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3404>

        HeapSize is covered by Cacheable so is not needed here.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3405>

        Indentation is incorrect here.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3406>

        Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means.
        Please add javadoc.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3407>

        Please add javadoc for what this method does.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3408>

        Whitespace.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3409>

        Indentation.

        • Ted

        On 2011-08-17 04:16:06, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-17 04:16:06)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1489 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3402 > Please add javadoc for these methods. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3403 > This shows that Cacheable can reside in on-heap cache. The description for Cacheable should be refined. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3404 > HeapSize is covered by Cacheable so is not needed here. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3405 > Indentation is incorrect here. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3406 > Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means. Please add javadoc. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3407 > Please add javadoc for what this method does. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3408 > Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3409 > Indentation. Ted On 2011-08-17 04:16:06, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-17 04:16:06) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        a few more changes, substantially re-factored.

        Show
        Li Pi added a comment - a few more changes, substantially re-factored.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-17 04:16:06.047294)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        refactored, no more casting, cacheable added. multithreaded test that hammers single key added. general code cleanup stuff as well. heapsize not implemented completely yet.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-17 04:16:06.047294) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- refactored, no more casting, cacheable added. multithreaded test that hammers single key added. general code cleanup stuff as well. heapsize not implemented completely yet. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-12 22:30:10.310736)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed two bugs as per ted yu's reviews.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 22:30:10.310736) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed two bugs as per ted yu's reviews. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        fixed two typos.

        Show
        Li Pi added a comment - fixed two typos.
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-12 21:52:43, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 38

        > <https://reviews.apache.org/r/1214/diff/11/?file=32400#file32400line38>

        >

        > Still some white spaces to remove.

        Got it.

        On 2011-08-12 21:52:43, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 37

        > <https://reviews.apache.org/r/1214/diff/11/?file=32401#file32401line37>

        >

        > Incorrect class name.

        Doh. Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1430
        -----------------------------------------------------------

        On 2011-08-12 20:26:21, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 20:26:21)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-12 21:52:43, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 38 > < https://reviews.apache.org/r/1214/diff/11/?file=32400#file32400line38 > > > Still some white spaces to remove. Got it. On 2011-08-12 21:52:43, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 37 > < https://reviews.apache.org/r/1214/diff/11/?file=32401#file32401line37 > > > Incorrect class name. Doh. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1430 ----------------------------------------------------------- On 2011-08-12 20:26:21, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 20:26:21) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1430
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3294>

        Still some white spaces to remove.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3295>

        Incorrect class name.

        • Ted

        On 2011-08-12 20:26:21, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 20:26:21)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1430 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3294 > Still some white spaces to remove. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3295 > Incorrect class name. Ted On 2011-08-12 20:26:21, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 20:26:21) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        fixed broken test and race condition.

        Show
        Li Pi added a comment - fixed broken test and race condition.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-12 20:26:21.230751)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Fixed another broken test case. (Didn't reset buffer position before doing compare) and fixed race.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 20:26:21.230751) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Fixed another broken test case. (Didn't reset buffer position before doing compare) and fixed race. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-12 17:26:40, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>

        >

        > this can race against getBlock() though:

        >

        > Thread A: backingMap.get(key) returns object

        > Thread B: put() returns same object

        > Thread B: free(object)

        > Thread A: use object. boom?

        >

        > putIfAbsent shouldn't be any slower than put, may as well make use of it

        Ah, gotcha! I see it now. Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1423
        -----------------------------------------------------------

        On 2011-08-12 08:41:37, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-12 17:26:40, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101 > > > this can race against getBlock() though: > > Thread A: backingMap.get(key) returns object > Thread B: put() returns same object > Thread B: free(object) > Thread A: use object. boom? > > putIfAbsent shouldn't be any slower than put, may as well make use of it Ah, gotcha! I see it now. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1423 ----------------------------------------------------------- On 2011-08-12 08:41:37, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1423
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3285>

        this can race against getBlock() though:

        Thread A: backingMap.get(key) returns object
        Thread B: put() returns same object
        Thread B: free(object)
        Thread A: use object. boom?

        putIfAbsent shouldn't be any slower than put, may as well make use of it

        • Todd

        On 2011-08-12 08:41:37, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1423 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3285 > this can race against getBlock() though: Thread A: backingMap.get(key) returns object Thread B: put() returns same object Thread B: free(object) Thread A: use object. boom? putIfAbsent shouldn't be any slower than put, may as well make use of it Todd On 2011-08-12 08:41:37, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>

        >

        > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1

        Li Pi wrote:

        http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)

        Returns:

        -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.

        Todd Lipcon wrote:

        http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object)

        "Returns:

        a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object."

        so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1

        Okay. Will fix. Thanks!

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>

        >

        > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:

        >

        > ByteBuffer storedBlock = ...allloc

        > ... fill it in...

        > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);

        > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > }

        >

        > make sense?

        Li Pi wrote:

        Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."

        Good change, still though, I changed it to free alreadyCached instead.

        Todd Lipcon wrote:

        ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right?

        Yes, but that someone else stores a reference to the singlesizecache and a copy of the key. They'll look it up by key again, and will hit the new item.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>

        >

        > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?

        Li Pi wrote:

        Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.

        Todd Lipcon wrote:

        is that true? wouldn't the 2G offheap always have a subset of what's in heap?

        ...ah yes, good point. i'll add this check in.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        On 2011-08-12 08:41:37, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114 > > > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1 Li Pi wrote: http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal ) Returns: -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val. Todd Lipcon wrote: http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object ) "Returns: a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object." so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1 Okay. Will fix. Thanks! On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101 > > > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like: > > ByteBuffer storedBlock = ...allloc > ... fill it in... > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock); > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > } > > make sense? Li Pi wrote: Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value." Good change, still though, I changed it to free alreadyCached instead. Todd Lipcon wrote: ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right? Yes, but that someone else stores a reference to the singlesizecache and a copy of the key. They'll look it up by key again, and will hit the new item. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64 > > > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here? Li Pi wrote: Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap. Todd Lipcon wrote: is that true? wouldn't the 2G offheap always have a subset of what's in heap? ...ah yes, good point. i'll add this check in. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- On 2011-08-12 08:41:37, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>

        >

        > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?

        Li Pi wrote:

        Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.

        is that true? wouldn't the 2G offheap always have a subset of what's in heap?

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67>

        >

        > missing space - " bytes ..."

        Li Pi wrote:

        The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different?

        oh, I see - you want it to say "1.9mbytes" instead of "1.9m bytes". That's fine

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60>

        >

        > vertically collapse this - one line per param

        Li Pi wrote:

        80 char limit?. I don't know why eclipse keeps doing this. Will fix!

        i think it'll still fit in 80 chars without all that vertical space

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>

        >

        > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:

        >

        > ByteBuffer storedBlock = ...allloc

        > ... fill it in...

        > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);

        > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > }

        >

        > make sense?

        Li Pi wrote:

        Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."

        Good change, still though, I changed it to free alreadyCached instead.

        ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right?

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>

        >

        > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000

        Li Pi wrote:

        changed it to returnBlock.put(contentBlock.duplicate()) instead.

        don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot?

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>

        >

        > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1

        Li Pi wrote:

        http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)

        Returns:

        -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.

        http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object)

        "Returns:
        a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object."

        so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45

        > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>

        >

        > hrm, this is identical to the other method?

        Li Pi wrote:

        Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.

        but you should be able to extract a method, even if the method has to take a type parameter. too much dup code

        • Todd

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        On 2011-08-12 08:41:37, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64 > > > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here? Li Pi wrote: Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap. is that true? wouldn't the 2G offheap always have a subset of what's in heap? On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67 > > > missing space - " bytes ..." Li Pi wrote: The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different? oh, I see - you want it to say "1.9mbytes" instead of "1.9m bytes". That's fine On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60 > > > vertically collapse this - one line per param Li Pi wrote: 80 char limit?. I don't know why eclipse keeps doing this. Will fix! i think it'll still fit in 80 chars without all that vertical space On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101 > > > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like: > > ByteBuffer storedBlock = ...allloc > ... fill it in... > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock); > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > } > > make sense? Li Pi wrote: Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value." Good change, still though, I changed it to free alreadyCached instead. ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right? On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118 > > > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000 Li Pi wrote: changed it to returnBlock.put(contentBlock.duplicate()) instead. don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot? On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114 > > > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1 Li Pi wrote: http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal ) Returns: -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val. http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object ) "Returns: a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object." so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1 On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45 > < https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45 > > > hrm, this is identical to the other method? Li Pi wrote: Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize. but you should be able to extract a method, even if the method has to take a type parameter. too much dup code Todd ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- On 2011-08-12 08:41:37, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37.017483)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed test failure, as ted yu reported.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37.017483) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed test failure, as ted yu reported. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        fixed test failure, as reported by ted yu.

        Show
        Li Pi added a comment - fixed test failure, as reported by ted yu.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-11 07:52:55.019423)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Fixed as per ted/todd's reviews.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f22fb6e
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-11 07:52:55.019423) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Fixed as per ted/todd's reviews. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f22fb6e src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        fixed things as per ted/todd's reviews

        Show
        Li Pi added a comment - fixed things as per ted/todd's reviews
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > CHANGES.txt, line 197

        > <https://reviews.apache.org/r/1214/diff/8/?file=31762#file31762line197>

        >

        > your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it.

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, lines 24-26

        > <https://reviews.apache.org/r/1214/diff/8/?file=31766#file31766line24>

        >

        > style:

        > /**

        > * ...

        > */

        > public class CacheStates {

        >

        > (comment formatting and space before '{')

        Fixed

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 58

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line58>

        >

        > hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no?

        This is currently taken care of in StoreFile.java, but, good idea, I'll change that.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>

        >

        > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?

        Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 65

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line65>

        >

        > hyphenate 'on-heap' and 'off-heap' for clarity

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67>

        >

        > missing space - " bytes ..."

        The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different?

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 137

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line137>

        >

        > we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well.

        Will take care of next revision.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>

        >

        > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000

        changed it to returnBlock.put(contentBlock.duplicate()) instead.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 176

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line176>

        >

        > this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock.

        >

        > In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now?

        Definitely. Will do. At that point, might also change DoubleBlockCache to not cache things that aren't CacheableBlocks.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 190

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line190>

        >

        > maybe rename to getOccupiedSize?

        Done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 148

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line148>

        >

        > this.size() is in units of bytes, not blocks

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45

        > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>

        >

        > hrm, this is identical to the other method?

        Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60>

        >

        > vertically collapse this - one line per param

        80 char limit?. I don't know why eclipse keeps doing this. Will fix!

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>

        >

        > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:

        >

        > ByteBuffer storedBlock = ...allloc

        > ... fill it in...

        > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);

        > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > }

        >

        > make sense?

        Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."

        Good change, still though, I changed it to free alreadyCached instead.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 26

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line26>

        >

        > wrong Log class - should use org.apache.commons.logging.Log

        fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 35

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line35>

        >

        > remove extra whitespace

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 37-38

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line37>

        >

        > hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout?

        Okay. Will figure out names.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 90-91

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line90>

        >

        > LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message – eg "Unable to deallocate direct memory during shutdown".

        done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 104

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line104>

        >

        > getBlock*s*Remaining, right?

        Done

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 111

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line111>

        >

        > incomplete comment here

        I have no idea what that comment is supposed to be, removed it.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 88

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line88>

        >

        > this isn't calling addSlabByConf here, so its javadoc seems out of date?

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 93

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line93>

        >

        > spelling: proportions

        >

        > Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix

        Fixed. All prefixes changed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 43

        > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line43>

        >

        > this class looks like mostly dup code

        It is, but SingleSizeCache still caches bytebuffers rather than HFileBlocks.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 42

        > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line42>

        >

        > rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests – otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals

        equals was never implemented beforehand. I added it. I think default equals is just ==? I don't think anyone was using .equals beforehand.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 1

        > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line1>

        >

        > add license

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java, line 30

        > <https://reviews.apache.org/r/1214/diff/8/?file=31778#file31778line30>

        >

        > patch accidentally reverting another recent commit

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 38

        > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line38>

        >

        > I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below:

        > long total = totalQueries.incrementAndGet();

        > if (total % 10000 == 0) { bq. > LOG.debug("Ran " + total + " queries"); bq. > }

        I just removed the printout - I don't think its needed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 29

        > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line29>

        >

        > long line

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 100

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line100>

        >

        > "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is.

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 110

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line110>

        >

        > just to be thorough, check that they aren't negative?

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>

        >

        > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1

        http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)

        Returns:
        -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 116

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line116>

        >

        > should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1."

        Done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 117

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line117>

        >

        > should we also check that they sum up to at least 0.99 or something?

        added a LOG.warn if it doesn't add up to 0.99

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 121

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line121>

        >

        > wrap line

        Done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 143

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line143>

        >

        > you should .trim() the strings so whitespace is ignored

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 159

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line159>

        >

        > is that true? I thought it threw an RTE

        Yes, it does throw a RTE. Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 161-164

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line161>

        >

        > collapse vertical space

        Done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 375-376

        > <https://reviews.apache.org/r/1214/diff/8/?file=31776#file31776line375>

        >

        > see note about consistency of conf params above

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 30

        > <https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line30>

        >

        > you don't need abstract inside an interface

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 23

        > <https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line23>

        >

        > this can be package-private right?

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 231

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line231>

        >

        > javadoc to indicate where this is called from.

        >

        > Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class:

        > new SlabItemEvictionWatcher() {

        > public void onEviction(String key) { bq. > internalBlockEvicted(key); bq. > }

        > }

        >

        >

        > that way it doesn't show up as a public API to anyone else

        this is now package private.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 60

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line60>

        >

        > this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 237

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line237>

        >

        > I think that the assignedCache's evict() counter is going to be double incremented here... call stack like:

        >

        > SingleSizeCache.ConcurrentMap.eviction

        > -> SingleSizeCache's eviction watcher

        > ---> calls stats.evict()

        > ---> removes from backingMap

        > ---> calls SlabCache.onEviction()

        > -----> calls assignedCache.evict(key)

        > -------> calls stats.evict() again

        > -------> removes from map which is always a no-op

        >

        > right?

        Whoops. Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 245

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line245>

        >

        > may as well just iterate over sizer.values()

        fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 315

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line315>

        >

        > again can iterate over values()

        fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 1

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line1>

        >

        > Apache license

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18>

        >

        > this is unused?

        Not anymore

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, lines 59-63

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line59>

        >

        > remove empty javadocs

        Done.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-09 22:05:13, Todd Lipcon wrote: > CHANGES.txt, line 197 > < https://reviews.apache.org/r/1214/diff/8/?file=31762#file31762line197 > > > your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it. Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, lines 24-26 > < https://reviews.apache.org/r/1214/diff/8/?file=31766#file31766line24 > > > style: > /** > * ... > */ > public class CacheStates { > > (comment formatting and space before '{') Fixed On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 58 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line58 > > > hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no? This is currently taken care of in StoreFile.java, but, good idea, I'll change that. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64 > > > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here? Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 65 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line65 > > > hyphenate 'on-heap' and 'off-heap' for clarity Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67 > > > missing space - " bytes ..." The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different? On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 137 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line137 > > > we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well. Will take care of next revision. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118 > > > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000 changed it to returnBlock.put(contentBlock.duplicate()) instead. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 176 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line176 > > > this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock. > > In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now? Definitely. Will do. At that point, might also change DoubleBlockCache to not cache things that aren't CacheableBlocks. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 190 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line190 > > > maybe rename to getOccupiedSize? Done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 148 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line148 > > > this.size() is in units of bytes, not blocks Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45 > < https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45 > > > hrm, this is identical to the other method? Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60 > > > vertically collapse this - one line per param 80 char limit?. I don't know why eclipse keeps doing this. Will fix! On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101 > > > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like: > > ByteBuffer storedBlock = ...allloc > ... fill it in... > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock); > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > } > > make sense? Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value." Good change, still though, I changed it to free alreadyCached instead. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 26 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line26 > > > wrong Log class - should use org.apache.commons.logging.Log fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 35 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line35 > > > remove extra whitespace Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 37-38 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line37 > > > hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout? Okay. Will figure out names. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 90-91 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line90 > > > LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message – eg "Unable to deallocate direct memory during shutdown". done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 104 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line104 > > > getBlock*s*Remaining, right? Done On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 111 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line111 > > > incomplete comment here I have no idea what that comment is supposed to be, removed it. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 88 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line88 > > > this isn't calling addSlabByConf here, so its javadoc seems out of date? Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 93 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line93 > > > spelling: proportions > > Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix Fixed. All prefixes changed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 43 > < https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line43 > > > this class looks like mostly dup code It is, but SingleSizeCache still caches bytebuffers rather than HFileBlocks. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 42 > < https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line42 > > > rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests – otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals equals was never implemented beforehand. I added it. I think default equals is just ==? I don't think anyone was using .equals beforehand. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 1 > < https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line1 > > > add license Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java, line 30 > < https://reviews.apache.org/r/1214/diff/8/?file=31778#file31778line30 > > > patch accidentally reverting another recent commit Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 38 > < https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line38 > > > I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below: > long total = totalQueries.incrementAndGet(); > if (total % 10000 == 0) { bq. > LOG.debug("Ran " + total + " queries"); bq. > } I just removed the printout - I don't think its needed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 29 > < https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line29 > > > long line Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 100 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line100 > > > "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is. Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 110 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line110 > > > just to be thorough, check that they aren't negative? Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114 > > > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1 http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal ) Returns: -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 116 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line116 > > > should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1." Done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 117 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line117 > > > should we also check that they sum up to at least 0.99 or something? added a LOG.warn if it doesn't add up to 0.99 On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 121 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line121 > > > wrap line Done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 143 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line143 > > > you should .trim() the strings so whitespace is ignored Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 159 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line159 > > > is that true? I thought it threw an RTE Yes, it does throw a RTE. Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 161-164 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line161 > > > collapse vertical space Done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 375-376 > < https://reviews.apache.org/r/1214/diff/8/?file=31776#file31776line375 > > > see note about consistency of conf params above Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 30 > < https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line30 > > > you don't need abstract inside an interface Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 23 > < https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line23 > > > this can be package-private right? Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 231 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line231 > > > javadoc to indicate where this is called from. > > Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class: > new SlabItemEvictionWatcher() { > public void onEviction(String key) { bq. > internalBlockEvicted(key); bq. > } > } > > > that way it doesn't show up as a public API to anyone else this is now package private. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 60 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line60 > > > this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 237 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line237 > > > I think that the assignedCache's evict() counter is going to be double incremented here... call stack like: > > SingleSizeCache.ConcurrentMap.eviction > -> SingleSizeCache's eviction watcher > ---> calls stats.evict() > ---> removes from backingMap > ---> calls SlabCache.onEviction() > -----> calls assignedCache.evict(key) > -------> calls stats.evict() again > -------> removes from map which is always a no-op > > right? Whoops. Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 245 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line245 > > > may as well just iterate over sizer.values() fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 315 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line315 > > > again can iterate over values() fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 1 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line1 > > > Apache license Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18 > > > this is unused? Not anymore On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, lines 59-63 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line59 > > > remove empty javadocs Done. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-30 02:57:26, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 41

        > <https://reviews.apache.org/r/1214/diff/3/?file=28543#file28543line41>

        >

        > I don't see SkipList being used in this class.

        Fixed. It got changed to an unsynchronized TreeMap, as we aren't supporting online cache resizing.

        On 2011-07-30 02:57:26, Ted Yu wrote:

        > conf/hbase-env.sh, line 40

        > <https://reviews.apache.org/r/1214/diff/3/?file=28534#file28534line40>

        >

        > Still some whitespaces.

        Fixed.

        On 2011-07-30 02:57:26, Ted Yu wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java, line 31

        > <https://reviews.apache.org/r/1214/diff/3/?file=28549#file28549line31>

        >

        > This doesn't match the actual test.

        Fixed.

        On 2011-07-30 02:57:26, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 31

        > <https://reviews.apache.org/r/1214/diff/3/?file=28542#file28542line31>

        >

        > Doesn't match actual name of class.

        Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1238
        -----------------------------------------------------------

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-30 02:57:26, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 41 > < https://reviews.apache.org/r/1214/diff/3/?file=28543#file28543line41 > > > I don't see SkipList being used in this class. Fixed. It got changed to an unsynchronized TreeMap, as we aren't supporting online cache resizing. On 2011-07-30 02:57:26, Ted Yu wrote: > conf/hbase-env.sh, line 40 > < https://reviews.apache.org/r/1214/diff/3/?file=28534#file28534line40 > > > Still some whitespaces. Fixed. On 2011-07-30 02:57:26, Ted Yu wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java, line 31 > < https://reviews.apache.org/r/1214/diff/3/?file=28549#file28549line31 > > > This doesn't match the actual test. Fixed. On 2011-07-30 02:57:26, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 31 > < https://reviews.apache.org/r/1214/diff/3/?file=28542#file28542line31 > > > Doesn't match actual name of class. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1238 ----------------------------------------------------------- On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-10 21:28:51, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 21

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line21>

        >

        > I believe this was a typo.

        > You meant 1 (for bytes), right ?

        Good catch. Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1392
        -----------------------------------------------------------

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-10 21:28:51, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 21 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line21 > > > I believe this was a typo. > You meant 1 (for bytes), right ? Good catch. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1392 ----------------------------------------------------------- On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1392
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3179>

        I believe this was a typo.
        You meant 1 (for bytes), right ?

        • Ted

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1392 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3179 > I believe this was a typo. You meant 1 (for bytes), right ? Ted On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-10 01:44:22, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18>

        >

        > I think this was intended for use by hbase-env.sh

        Yup, it's used by StoreFile.java now. We now specify the size of the off heap cache as a proportion of total direct memory size.

        long offHeapCacheSize = (long) (conf.getFloat("hbase.offheapcachepercentage", (float) 0.95) * DirectMemoryUtils.getDirectMemorySize());

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1371
        -----------------------------------------------------------

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-10 01:44:22, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18 > > > I think this was intended for use by hbase-env.sh Yup, it's used by StoreFile.java now. We now specify the size of the off heap cache as a proportion of total direct memory size. long offHeapCacheSize = (long) (conf.getFloat("hbase.offheapcachepercentage", (float) 0.95) * DirectMemoryUtils.getDirectMemorySize()); Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1371 ----------------------------------------------------------- On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1371
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3155>

        I think this was intended for use by hbase-env.sh

        • Ted

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1371 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3155 > I think this was intended for use by hbase-env.sh Ted On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        CHANGES.txt
        <https://reviews.apache.org/r/1214/#comment3063>

        your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it.

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment3064>

        style:
        /**

        • ...
          */
          public class CacheStates {

        (comment formatting and space before '{')

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3068>

        hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no?

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3069>

        does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3065>

        hyphenate 'on-heap' and 'off-heap' for clarity

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3066>

        missing space - " bytes ..."

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3067>

        same as above

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3070>

        we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3071>

        vertically collapse this - one line per param

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3072>

        when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:

        ByteBuffer storedBlock = ...allloc
        ... fill it in...
        ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);
        if (alreadyCached != null)

        { // we didn't insert the new one, so free it and throw an exception backingStore.free(storedBlock); throw new RuntimeException("already cached xxxxx"); }

        make sense?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3073>

        I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3074>

        this.size() is in units of bytes, not blocks

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3075>

        maybe rename to getOccupiedSize?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3076>

        wrong Log class - should use org.apache.commons.logging.Log

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3077>

        remove extra whitespace

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3078>

        hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3079>

        LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message – eg "Unable to deallocate direct memory during shutdown".

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3080>

        getBlock*s*Remaining, right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3081>

        incomplete comment here

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3083>

        this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3085>

        this isn't calling addSlabByConf here, so its javadoc seems out of date?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3087>

        spelling: proportions

        Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3088>

        "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3094>

        just to be thorough, check that they aren't negative?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3089>

        > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3091>

        should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1."

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3095>

        should we also check that they sum up to at least 0.99 or something?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3092>

        wrap line

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3093>

        you should .trim() the strings so whitespace is ignored

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3096>

        is that true? I thought it threw an RTE

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3097>

        collapse vertical space

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3098>

        this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock.

        In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3100>

        javadoc to indicate where this is called from.

        Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class:
        new SlabItemEvictionWatcher() {
        public void onEviction(String key)

        { internalBlockEvicted(key); }

        }

        that way it doesn't show up as a public API to anyone else

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3102>

        I think that the assignedCache's evict() counter is going to be double incremented here... call stack like:

        SingleSizeCache.ConcurrentMap.eviction
        -> SingleSizeCache's eviction watcher
        ---> calls stats.evict()
        ---> removes from backingMap
        ---> calls SlabCache.onEviction()
        -----> calls assignedCache.evict(key)
        -------> calls stats.evict() again
        -------> removes from map which is always a no-op

        right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3103>

        may as well just iterate over sizer.values()

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3104>

        again can iterate over values()

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3105>

        move this function up in the file so it's with other functions. Also needs @Override right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3106>

        also move this function up above the inner classes

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3099>

        does this make a copy? it shouldn't need to at this point, right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment3108>

        this can be package-private right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment3107>

        you don't need abstract inside an interface

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        <https://reviews.apache.org/r/1214/#comment3109>

        see note about consistency of conf params above

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3110>

        Apache license

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3111>

        this is unused?

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3112>

        remove empty javadocs

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java
        <https://reviews.apache.org/r/1214/#comment3113>

        patch accidentally reverting another recent commit

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3114>

        add license

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3115>

        long line

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3117>

        using a queue here ensures that two threads never request the same block in this test - that's one good test, but there should be another that just caches a small number of blocks and pounds on get only

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3116>

        I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below:
        long total = totalQueries.incrementAndGet();
        if (total % 10000 == 0)

        { LOG.debug("Ran " + total + " queries"); }

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3118>

        rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests – otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3120>

        this class looks like mostly dup code

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3119>

        hrm, this is identical to the other method?

        • Todd

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- CHANGES.txt < https://reviews.apache.org/r/1214/#comment3063 > your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment3064 > style: /** ... */ public class CacheStates { (comment formatting and space before '{') src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3068 > hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no? src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3069 > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here? src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3065 > hyphenate 'on-heap' and 'off-heap' for clarity src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3066 > missing space - " bytes ..." src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3067 > same as above src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3070 > we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3071 > vertically collapse this - one line per param src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3072 > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like: ByteBuffer storedBlock = ...allloc ... fill it in... ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock); if (alreadyCached != null) { // we didn't insert the new one, so free it and throw an exception backingStore.free(storedBlock); throw new RuntimeException("already cached xxxxx"); } make sense? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3073 > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3074 > this.size() is in units of bytes, not blocks src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3075 > maybe rename to getOccupiedSize? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3076 > wrong Log class - should use org.apache.commons.logging.Log src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3077 > remove extra whitespace src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3078 > hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3079 > LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message – eg "Unable to deallocate direct memory during shutdown". src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3080 > getBlock*s*Remaining, right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3081 > incomplete comment here src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3083 > this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3085 > this isn't calling addSlabByConf here, so its javadoc seems out of date? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3087 > spelling: proportions Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3088 > "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3094 > just to be thorough, check that they aren't negative? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3089 > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3091 > should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1." src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3095 > should we also check that they sum up to at least 0.99 or something? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3092 > wrap line src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3093 > you should .trim() the strings so whitespace is ignored src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3096 > is that true? I thought it threw an RTE src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3097 > collapse vertical space src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3098 > this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock. In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3100 > javadoc to indicate where this is called from. Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class: new SlabItemEvictionWatcher() { public void onEviction(String key) { internalBlockEvicted(key); } } that way it doesn't show up as a public API to anyone else src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3102 > I think that the assignedCache's evict() counter is going to be double incremented here... call stack like: SingleSizeCache.ConcurrentMap.eviction -> SingleSizeCache's eviction watcher ---> calls stats.evict() ---> removes from backingMap ---> calls SlabCache.onEviction() -----> calls assignedCache.evict(key) -------> calls stats.evict() again -------> removes from map which is always a no-op right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3103 > may as well just iterate over sizer.values() src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3104 > again can iterate over values() src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3105 > move this function up in the file so it's with other functions. Also needs @Override right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3106 > also move this function up above the inner classes src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3099 > does this make a copy? it shouldn't need to at this point, right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment3108 > this can be package-private right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment3107 > you don't need abstract inside an interface src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java < https://reviews.apache.org/r/1214/#comment3109 > see note about consistency of conf params above src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3110 > Apache license src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3111 > this is unused? src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3112 > remove empty javadocs src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java < https://reviews.apache.org/r/1214/#comment3113 > patch accidentally reverting another recent commit src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3114 > add license src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3115 > long line src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3117 > using a queue here ensures that two threads never request the same block in this test - that's one good test, but there should be another that just caches a small number of blocks and pounds on get only src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3116 > I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below: long total = totalQueries.incrementAndGet(); if (total % 10000 == 0) { LOG.debug("Ran " + total + " queries"); } src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3118 > rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests – otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3120 > this class looks like mostly dup code src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3119 > hrm, this is identical to the other method? Todd On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        Moved HFileBlockCacheTestUtils/SingleSizeCacheTestUtils to the tests folder, so it builds under mvn.

        Show
        Li Pi added a comment - Moved HFileBlockCacheTestUtils/SingleSizeCacheTestUtils to the tests folder, so it builds under mvn.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55.719673)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Fixed compilation error. Corrected package location of cachetestutils.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        CHANGES.txt e9c0478
        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55.719673) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Fixed compilation error. Corrected package location of cachetestutils. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        ignore v10 diff, that is just broken. this is the fixed one.

        Show
        Li Pi added a comment - ignore v10 diff, that is just broken. this is the fixed one.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-09 08:36:02.175711)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Fixed diff. Rebase on 3857 works!

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 08:36:02.175711) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Fixed diff. Rebase on 3857 works! Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-09 07:52:09.666709)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Rebased onto 3857. Configuration added as well.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        CHANGES.txt 7c6f592
        conf/hbase-env.sh 2d55d27
        pom.xml 0f24681
        src/docbkx/book.xml 2c19cef
        src/docbkx/configuration.xml 3595e76
        src/docbkx/developer.xml a3e22ea
        src/docbkx/performance.xml d8e104f
        src/docbkx/troubleshooting.xml 9f93cd9
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/ipc/Delayable.java 56f0650
        src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f223e46
        src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 6fb1da7
        src/main/java/org/apache/hadoop/hbase/mapred/Driver.java d38956c
        src/main/java/org/apache/hadoop/hbase/mapreduce/Driver.java dda4241
        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff05df8
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
        src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 8e87c83
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 887f736
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java 9d87fa5
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 151c90b
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 8e1f6e0
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937
        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java edca7f4
        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 0729410
        src/main/java/org/apache/hadoop/hbase/util/PoolMap.java ebf3078
        src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java a0bef34
        src/main/ruby/hbase/admin.rb 4460d6e
        src/site/xdoc/index.xml 8e1b531
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java 0b21c6c
        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6d83b00
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java b4c407b
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java dc43eb2
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 381ac90
        src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java 2c565d7

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 07:52:09.666709) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Rebased onto 3857. Configuration added as well. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) CHANGES.txt 7c6f592 conf/hbase-env.sh 2d55d27 pom.xml 0f24681 src/docbkx/book.xml 2c19cef src/docbkx/configuration.xml 3595e76 src/docbkx/developer.xml a3e22ea src/docbkx/performance.xml d8e104f src/docbkx/troubleshooting.xml 9f93cd9 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/Delayable.java 56f0650 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f223e46 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 6fb1da7 src/main/java/org/apache/hadoop/hbase/mapred/Driver.java d38956c src/main/java/org/apache/hadoop/hbase/mapreduce/Driver.java dda4241 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff05df8 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 8e87c83 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 887f736 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java 9d87fa5 src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 151c90b src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 8e1f6e0 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java edca7f4 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 0729410 src/main/java/org/apache/hadoop/hbase/util/PoolMap.java ebf3078 src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java a0bef34 src/main/ruby/hbase/admin.rb 4460d6e src/site/xdoc/index.xml 8e1b531 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java 0b21c6c src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6d83b00 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java b4c407b src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java dc43eb2 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 381ac90 src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java 2c565d7 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-02 00:46:57.567666)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        configuration changed to percentage of MaxDirectMemorySize rather than specifying size of offHeapCache.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-02 00:46:57.567666) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- configuration changed to percentage of MaxDirectMemorySize rather than specifying size of offHeapCache. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        configuration changed to percentage of MaxDirectMemorySize.

        Show
        Li Pi added a comment - configuration changed to percentage of MaxDirectMemorySize.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-01 23:54:08.232314)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Addressed most of Ted Yu/JGray's comments. Will do another patch later. Fixed build errors. Tested patch/build on fresh pull from trunk.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-01 23:54:08.232314) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Addressed most of Ted Yu/JGray's comments. Will do another patch later. Fixed build errors. Tested patch/build on fresh pull from trunk. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        Fixed build error, addressed most of Ted Yu/JGray's review.

        Show
        Li Pi added a comment - Fixed build error, addressed most of Ted Yu/JGray's review.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1238
        -----------------------------------------------------------

        conf/hbase-env.sh
        <https://reviews.apache.org/r/1214/#comment2812>

        Still some whitespaces.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment2813>

        Doesn't match actual name of class.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2814>

        I don't see SkipList being used in this class.

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
        <https://reviews.apache.org/r/1214/#comment2815>

        This doesn't match the actual test.

        • Ted

        On 2011-07-30 00:39:48, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-30 00:39:48)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1238 ----------------------------------------------------------- conf/hbase-env.sh < https://reviews.apache.org/r/1214/#comment2812 > Still some whitespaces. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment2813 > Doesn't match actual name of class. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2814 > I don't see SkipList being used in this class. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java < https://reviews.apache.org/r/1214/#comment2815 > This doesn't match the actual test. Ted On 2011-07-30 00:39:48, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-30 00:39:48) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100

        > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100>

        >

        > Should we add hitCount for offHeapCache ?

        > This can be addressed in a follow-on JIRA.

        Li Pi wrote:

        I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics.

        What I meant to say, is, that it does have a hitcount. It just isn't displayed anywhere.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1220
        -----------------------------------------------------------

        On 2011-07-30 00:39:48, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-30 00:39:48)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100 > < https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100 > > > Should we add hitCount for offHeapCache ? > This can be addressed in a follow-on JIRA. Li Pi wrote: I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics. What I meant to say, is, that it does have a hitcount. It just isn't displayed anywhere. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1220 ----------------------------------------------------------- On 2011-07-30 00:39:48, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-30 00:39:48) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > conf/hbase-env.sh, lines 44-45

        > <https://reviews.apache.org/r/1214/diff/1/?file=28125#file28125line44>

        >

        > Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is)

        I haven't figured out a way. Just asked StackOverflow - hopefully they'll have an answer.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2

        > <https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line2>

        >

        > 2011

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 1

        > <https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line1>

        >

        > license

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 41

        > <https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line41>

        >

        > whitespace

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 5

        > <https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line5>

        >

        > class comment

        Added

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 39

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line39>

        >

        > Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc).

        >

        > If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks)

        It can be made more generic easily. I just haven't done it. Will do though - just create a constructor that takes two BlockCaches?

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 66

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line66>

        >

        > longer than 80 chars

        fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-79

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line77>

        >

        > This seems like a behavior that we may not always want.

        >

        > If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?)

        >

        > we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap)

        I was thinking of sending all initial caches to the off heap cache, giving it an inbuilt scan resistance, but LRUBlockCache already does partitioning to deal with scans.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line115>

        >

        > this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both.

        The idea was the combined evicted stats of both, but yeah, CacheStats are implemented for both LruBlockCache and SlabCache, so exposing both might be a good idea.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 74

        > <https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line74>

        >

        > huh?

        Just a typo. Fixed. Don't know why that line is there.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 84

        > <https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line84>

        >

        > line > 80 chars

        fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 122-125

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line122>

        >

        > seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed

        If the size of the item being cached is too big to fit into any slab, we can't cache it. Again, I'm moving away from MemCached's model, so this should work better.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 151

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line151>

        >

        > is this really a cache miss? this cache will never take this block.

        Good point. Taking that part out.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 155

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line155>

        >

        > but here we need to check for null and increment cache miss rather than hit if it is null

        Yup, doing so as well.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 170-173

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line170>

        >

        > this non-javadoc comment is not necessary, just the @Override has been normal format in hbase

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 269

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line269>

        >

        > should this really be a bunch of INFO logs? Could we have these exported as metrics instead?

        Will take care of this next pass.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 1

        > <https://reviews.apache.org/r/1214/diff/1/?file=28135#file28135line1>

        >

        > license

        added.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 374-375

        > <https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line374>

        >

        > formatting looks a little strange here (missing spaces and such)

        fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 28

        > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line28>

        >

        > This test looks more specific than testing "the concurrent LruBlockCache"

        Woah, typo, fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 34

        > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line34>

        >

        > you are missing a space here and throughout this file and some others... our format is to always have a space before a {

        fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java, line 1

        > <https://reviews.apache.org/r/1214/diff/1/?file=28139#file28139line1>

        >

        > license

        added.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 2

        > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line2>

        >

        > 2011

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 376

        > <https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line376>

        >

        > line > 80 chars

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 63-65

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line63>

        >

        > Why these ratios? At the least, this should all be configurable (even if just in code and undocumented).

        >

        > Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive.

        >

        > I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated.

        Its sort of expensive to change cache sizes with DirectByteBuffer after instantiation. I have a plan to restructure this away from MemCached's model though.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 37

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line37>

        >

        > whitespace here and throughout this file

        Removed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 48

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line48>

        >

        > I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab?

        I'm using MemCached's allocation model - although I have one written that does it in a wholly different manner. But MemCached gets around fragmentation by allocating a certain range of sizes to a Single Slab.

        If something is smaller than the max size of a particular slab, the extra space is just wasted. Each SingleSizeCache can cache things up to a certain size. Things above that size cannot be cached, and things below that size will waste space.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1211
        -----------------------------------------------------------

        On 2011-07-30 00:39:48, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-30 00:39:48)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-29 17:41:50, Jonathan Gray wrote: > conf/hbase-env.sh, lines 44-45 > < https://reviews.apache.org/r/1214/diff/1/?file=28125#file28125line44 > > > Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is) I haven't figured out a way. Just asked StackOverflow - hopefully they'll have an answer. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2 > < https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line2 > > > 2011 Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 1 > < https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line1 > > > license Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 41 > < https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line41 > > > whitespace Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 5 > < https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line5 > > > class comment Added On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 39 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line39 > > > Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc). > > If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks) It can be made more generic easily. I just haven't done it. Will do though - just create a constructor that takes two BlockCaches? On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 66 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line66 > > > longer than 80 chars fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-79 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line77 > > > This seems like a behavior that we may not always want. > > If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?) > > we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap) I was thinking of sending all initial caches to the off heap cache, giving it an inbuilt scan resistance, but LRUBlockCache already does partitioning to deal with scans. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line115 > > > this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both. The idea was the combined evicted stats of both, but yeah, CacheStats are implemented for both LruBlockCache and SlabCache, so exposing both might be a good idea. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 74 > < https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line74 > > > huh? Just a typo. Fixed. Don't know why that line is there. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 84 > < https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line84 > > > line > 80 chars fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 122-125 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line122 > > > seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed If the size of the item being cached is too big to fit into any slab, we can't cache it. Again, I'm moving away from MemCached's model, so this should work better. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 151 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line151 > > > is this really a cache miss? this cache will never take this block. Good point. Taking that part out. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 155 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line155 > > > but here we need to check for null and increment cache miss rather than hit if it is null Yup, doing so as well. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 170-173 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line170 > > > this non-javadoc comment is not necessary, just the @Override has been normal format in hbase Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 269 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line269 > > > should this really be a bunch of INFO logs? Could we have these exported as metrics instead? Will take care of this next pass. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 1 > < https://reviews.apache.org/r/1214/diff/1/?file=28135#file28135line1 > > > license added. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 374-375 > < https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line374 > > > formatting looks a little strange here (missing spaces and such) fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 28 > < https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line28 > > > This test looks more specific than testing "the concurrent LruBlockCache" Woah, typo, fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 34 > < https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line34 > > > you are missing a space here and throughout this file and some others... our format is to always have a space before a { fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java, line 1 > < https://reviews.apache.org/r/1214/diff/1/?file=28139#file28139line1 > > > license added. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 2 > < https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line2 > > > 2011 Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 376 > < https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line376 > > > line > 80 chars Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 63-65 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line63 > > > Why these ratios? At the least, this should all be configurable (even if just in code and undocumented). > > Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive. > > I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated. Its sort of expensive to change cache sizes with DirectByteBuffer after instantiation. I have a plan to restructure this away from MemCached's model though. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 37 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line37 > > > whitespace here and throughout this file Removed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 48 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line48 > > > I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab? I'm using MemCached's allocation model - although I have one written that does it in a wholly different manner. But MemCached gets around fragmentation by allocating a certain range of sizes to a Single Slab. If something is smaller than the max size of a particular slab, the extra space is just wasted. Each SingleSizeCache can cache things up to a certain size. Things above that size cannot be cached, and things below that size will waste space. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1211 ----------------------------------------------------------- On 2011-07-30 00:39:48, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-30 00:39:48) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        addressed some of Ted Yu/JGrays comments on reviewboard.

        Show
        Li Pi added a comment - addressed some of Ted Yu/JGrays comments on reviewboard.
        Hide
        Li Pi added a comment -

        New updated benchmark results. Tested with 1gb dataset, as per todd's suggestion. the FS cache does better, but still can't match the off heap cache.

        Show
        Li Pi added a comment - New updated benchmark results. Tested with 1gb dataset, as per todd's suggestion. the FS cache does better, but still can't match the off heap cache.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-07-30 00:39:48.920025)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Addressed TedYu/JGray's comments.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-30 00:39:48.920025) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Addressed TedYu/JGray's comments. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > conf/hbase-env.sh, line 42

        > <https://reviews.apache.org/r/1214/diff/2/?file=28356#file28356line42>

        >

        > Extra whitespace is obvious on review board.

        Fixed.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2

        > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line2>

        >

        > Year of copyright.

        Fixed.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 48

        > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line48>

        >

        > Long line should wrap.

        Fixed.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 64

        > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line64>

        >

        > Indentation should be corrected for the if block.

        Fixed

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 2

        > <https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line2>

        >

        > Copyright missing.

        Fixed

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 25

        > <https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line25>

        >

        > This implies that an eviction may involve fewer than one block.

        > If so, this count can be named evictedBlockCount.

        Fixed.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100

        > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100>

        >

        > Should we add hitCount for offHeapCache ?

        > This can be addressed in a follow-on JIRA.

        I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115

        > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line115>

        >

        > In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts.

        Agreed. I may be redoing this substantially in a follow on Jira.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 145

        > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line145>

        >

        > Is this a typo ?

        Yup, and fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1220
        -----------------------------------------------------------

        On 2011-07-28 23:02:50, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-28 23:02:50)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-29 05:14:48, Ted Yu wrote: > conf/hbase-env.sh, line 42 > < https://reviews.apache.org/r/1214/diff/2/?file=28356#file28356line42 > > > Extra whitespace is obvious on review board. Fixed. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2 > < https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line2 > > > Year of copyright. Fixed. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 48 > < https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line48 > > > Long line should wrap. Fixed. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 64 > < https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line64 > > > Indentation should be corrected for the if block. Fixed On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 2 > < https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line2 > > > Copyright missing. Fixed On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 25 > < https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line25 > > > This implies that an eviction may involve fewer than one block. > If so, this count can be named evictedBlockCount. Fixed. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100 > < https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100 > > > Should we add hitCount for offHeapCache ? > This can be addressed in a follow-on JIRA. I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115 > < https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line115 > > > In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts. Agreed. I may be redoing this substantially in a follow on Jira. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 145 > < https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line145 > > > Is this a typo ? Yup, and fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1220 ----------------------------------------------------------- On 2011-07-28 23:02:50, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 23:02:50) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1211
        -----------------------------------------------------------

        conf/hbase-env.sh
        <https://reviews.apache.org/r/1214/#comment2675>

        Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is)

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment2676>

        2011

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment2677>

        whitespace

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment2678>

        license

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment2679>

        class comment

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2680>

        whitespace here and throughout this file

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2681>

        Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc).

        If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks)

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2682>

        longer than 80 chars

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2683>

        This seems like a behavior that we may not always want.

        If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?)

        we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap)

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2684>

        this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment2685>

        huh?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment2686>

        line > 80 chars

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment2687>

        getTotalNumBlocks() and getRemainingNumBlocks() or something? i find the method names a little unclear (or just add some javadoc)

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment2688>

        javadoc on these

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2689>

        I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2690>

        Why these ratios? At the least, this should all be configurable (even if just in code and undocumented).

        Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive.

        I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2691>

        seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2692>

        is this really a cache miss? this cache will never take this block.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2693>

        but here we need to check for null and increment cache miss rather than hit if it is null

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2694>

        this non-javadoc comment is not necessary, just the @Override has been normal format in hbase

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2762>

        should this really be a bunch of INFO logs? Could we have these exported as metrics instead?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment2763>

        license

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        <https://reviews.apache.org/r/1214/#comment2764>

        formatting looks a little strange here (missing spaces and such)

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        <https://reviews.apache.org/r/1214/#comment2765>

        line > 80 chars

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1214/#comment2766>

        2011

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1214/#comment2767>

        This test looks more specific than testing "the concurrent LruBlockCache"

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1214/#comment2768>

        you are missing a space here and throughout this file and some others... our format is to always have a space before a {

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
        <https://reviews.apache.org/r/1214/#comment2769>

        license

        • Jonathan

        On 2011-07-28 23:02:50, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-28 23:02:50)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1211 ----------------------------------------------------------- conf/hbase-env.sh < https://reviews.apache.org/r/1214/#comment2675 > Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is) src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment2676 > 2011 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment2677 > whitespace src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment2678 > license src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment2679 > class comment src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2680 > whitespace here and throughout this file src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2681 > Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc). If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks) src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2682 > longer than 80 chars src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2683 > This seems like a behavior that we may not always want. If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?) we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap) src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2684 > this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment2685 > huh? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment2686 > line > 80 chars src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment2687 > getTotalNumBlocks() and getRemainingNumBlocks() or something? i find the method names a little unclear (or just add some javadoc) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment2688 > javadoc on these src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2689 > I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2690 > Why these ratios? At the least, this should all be configurable (even if just in code and undocumented). Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive. I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2691 > seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2692 > is this really a cache miss? this cache will never take this block. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2693 > but here we need to check for null and increment cache miss rather than hit if it is null src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2694 > this non-javadoc comment is not necessary, just the @Override has been normal format in hbase src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2762 > should this really be a bunch of INFO logs? Could we have these exported as metrics instead? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment2763 > license src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java < https://reviews.apache.org/r/1214/#comment2764 > formatting looks a little strange here (missing spaces and such) src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java < https://reviews.apache.org/r/1214/#comment2765 > line > 80 chars src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java < https://reviews.apache.org/r/1214/#comment2766 > 2011 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java < https://reviews.apache.org/r/1214/#comment2767 > This test looks more specific than testing "the concurrent LruBlockCache" src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java < https://reviews.apache.org/r/1214/#comment2768 > you are missing a space here and throughout this file and some others... our format is to always have a space before a { src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java < https://reviews.apache.o