HBase
  1. HBase
  2. HBASE-4027

Enable direct byte buffers LruBlockCache

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Setting hbase.offheapcachesize in hbase-site.xml and -XX:MaxDirectMemorySize in hbase-env.sh to enable this feature. The file already has a line you can uncomment and you need to set the size of the direct memory (your total memory - size allocated to memstores - size allocated to the normal block cache - and some head room for the other functionalities).
      Show
      Setting hbase.offheapcachesize in hbase-site.xml and -XX:MaxDirectMemorySize in hbase-env.sh to enable this feature. The file already has a line you can uncomment and you need to set the size of the direct memory (your total memory - size allocated to memstores - size allocated to the normal block cache - and some head room for the other functionalities).

      Description

      Java offers the creation of direct byte buffers which are allocated outside of the heap.

      They need to be manually free'd, which can be accomplished using an documented clean method.

      The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.

      1. hbase-4027v15.3.diff
        91 kB
        Li Pi
      2. hbase4027v15.2.diff
        91 kB
        Li Pi
      3. hbase-4027v13.1.diff
        83 kB
        Li Pi
      4. hbase4027v12.1.diff
        108 kB
        Li Pi
      5. hbase4027v12.diff
        113 kB
        Li Pi
      6. hbase4027v11.7.diff
        68 kB
        Li Pi
      7. hbase4027v11.6.diff
        68 kB
        Li Pi
      8. hbase4027v11.5.diff
        68 kB
        Li Pi
      9. hbase4027v11.diff
        68 kB
        Li Pi
      10. hbase-4027v10.6.diff
        68 kB
        Li Pi
      11. hbase-4027-v10.5.diff
        62 kB
        Li Pi
      12. hbase-4027-v10.diff
        155 kB
        Li Pi
      13. HBase4027v9.diff
        56 kB
        Li Pi
      14. HBase4027v8.diff
        56 kB
        Li Pi
      15. 4027v7.diff
        52 kB
        Li Pi
      16. HBase-4027 (1).pdf
        54 kB
        Li Pi
      17. hbase-4027v6.diff
        50 kB
        Li Pi
      18. HBase-4027.pdf
        54 kB
        Li Pi
      19. 4027-v5.diff
        50 kB
        Li Pi
      20. slabcachepatchv4.5.diff
        38 kB
        Li Pi
      21. slabcachepatchv4.diff
        46 kB
        Li Pi
      22. slabcachepatchv3.2.diff
        37 kB
        Li Pi
      23. slabcachepatchv3.1.diff
        37 kB
        Li Pi
      24. slabcachepatchv3.diff
        34 kB
        Li Pi
      25. slabcachepatchv2.diff
        33 kB
        Li Pi
      26. slabcachepatch.diff
        34 kB
        Li Pi

        Activity

        Hide
        Jason Rutherglen added a comment -

        Here's the relevant code from hadoop-lzo:

        Object cleaner = buf.getClass().getMethod("cleaner").invoke(buf);
        cleaner.getClass().getMethod("clean").invoke(cleaner);
        

        and

        ByteBuffer.allocateDirect(newSize);
        
        Show
        Jason Rutherglen added a comment - Here's the relevant code from hadoop-lzo: Object cleaner = buf.getClass().getMethod( "cleaner" ).invoke(buf); cleaner.getClass().getMethod( "clean" ).invoke(cleaner); and ByteBuffer.allocateDirect(newSize);
        Hide
        Jonathan Gray added a comment -

        In the new HFile v2 over in HBASE-3857 the block cache interface changes from ByteBuffer to HeapSize. So you can now put anything you want into the cache that implements HeapSize (there is a new HFileBlock that is used in HFile v2).

        One big question is whether you're going to make copies out of the direct byte buffers on each read of that block, or if you're going to change KeyValue to use the ByteBuffer interface (or some other) instead of the byte[] directly. With a DBB you can't get access to an underlying byte[].

        Show
        Jonathan Gray added a comment - In the new HFile v2 over in HBASE-3857 the block cache interface changes from ByteBuffer to HeapSize. So you can now put anything you want into the cache that implements HeapSize (there is a new HFileBlock that is used in HFile v2). One big question is whether you're going to make copies out of the direct byte buffers on each read of that block, or if you're going to change KeyValue to use the ByteBuffer interface (or some other) instead of the byte[] directly. With a DBB you can't get access to an underlying byte[].
        Hide
        Li Pi added a comment -

        This would be really useful. I think even making copies out of the direct byte buffers would confer a substantial performance advantage over the FS cache.

        Show
        Li Pi added a comment - This would be really useful. I think even making copies out of the direct byte buffers would confer a substantial performance advantage over the FS cache.
        Hide
        Jason Rutherglen added a comment -

        One big question is whether you're going to make copies out of the direct byte buffers on each read of that block, or if you're going to change KeyValue to use the ByteBuffer interface (or some other) instead of the byte[] directly

        Right the HFile.Scanner.getKeyValue() method is calling block.array(). We'd need to track down all byte[] references, and convert them to ByteBuffer. That's more of a separate Jira.

        I think converting a direct ByteBuffer to byte[] will generate a fair amount of garbage, though of a different (smaller and more numerous) kind than the blocks.

        Show
        Jason Rutherglen added a comment - One big question is whether you're going to make copies out of the direct byte buffers on each read of that block, or if you're going to change KeyValue to use the ByteBuffer interface (or some other) instead of the byte[] directly Right the HFile.Scanner.getKeyValue() method is calling block.array() . We'd need to track down all byte[] references, and convert them to ByteBuffer . That's more of a separate Jira. I think converting a direct ByteBuffer to byte[] will generate a fair amount of garbage, though of a different (smaller and more numerous) kind than the blocks.
        Hide
        Jason Rutherglen added a comment -

        This would be really useful. I think even making copies out of the direct byte buffers would confer a substantial performance advantage over the FS cache.

        The filesystem cache doesn't help because HBase needs quick access to uncompressed blocks for scanning. For what duration does an uncompressed block need to be cached? In either case, accessing compressed blocks from the FS cache will be faster than hitting the disk or network. I am guessing one can maintain a small'ish block cache, ensure HDFS blocks are local, provide extra space for the FS cache, and gain in read throughput.

        Snappy should decompress fast enough for this to be more viable than maintaining a large-ish block cache. The problem [today] with a small'ish block cache is the GC is driven mad.

        Show
        Jason Rutherglen added a comment - This would be really useful. I think even making copies out of the direct byte buffers would confer a substantial performance advantage over the FS cache. The filesystem cache doesn't help because HBase needs quick access to uncompressed blocks for scanning. For what duration does an uncompressed block need to be cached? In either case, accessing compressed blocks from the FS cache will be faster than hitting the disk or network. I am guessing one can maintain a small'ish block cache, ensure HDFS blocks are local, provide extra space for the FS cache, and gain in read throughput. Snappy should decompress fast enough for this to be more viable than maintaining a large-ish block cache. The problem [today] with a small'ish block cache is the GC is driven mad.
        Hide
        Li Pi added a comment -

        I have a copy based variant working right now, I'll submit the patch by the end of this week for that. Benchmarks show roughly 75% of the speed of the existing LruBlockCache, and around 4x the speed of FS cache reads. Realistically, we'll be using a two stage cache solution, with the LruBlockCache also holding some values, so for the majority of reads, it'll be the same speed as the LruBlockCache.

        Currently pursueing a 0 copy solution - going to need to make everything run off ByteBuffers rather than byte[] though.

        Show
        Li Pi added a comment - I have a copy based variant working right now, I'll submit the patch by the end of this week for that. Benchmarks show roughly 75% of the speed of the existing LruBlockCache, and around 4x the speed of FS cache reads. Realistically, we'll be using a two stage cache solution, with the LruBlockCache also holding some values, so for the majority of reads, it'll be the same speed as the LruBlockCache. Currently pursueing a 0 copy solution - going to need to make everything run off ByteBuffers rather than byte[] though.
        Hide
        stack added a comment -

        Currently pursueing a 0 copy solution - going to need to make everything run off ByteBuffers rather than byte[] though.

        Go for it. That should fill your summer Li. Good stuff.

        Show
        stack added a comment - Currently pursueing a 0 copy solution - going to need to make everything run off ByteBuffers rather than byte[] though. Go for it. That should fill your summer Li. Good stuff.
        Hide
        Li Pi added a comment -

        Slab allocated cache for hbase. 80% of the speed of the original lrublockcache.

        Show
        Li Pi added a comment - Slab allocated cache for hbase. 80% of the speed of the original lrublockcache.
        Hide
        Li Pi added a comment -

        I'll do the writeup and pretty benchmarked graphs and stuff tomorrow.

        Show
        Li Pi added a comment - I'll do the writeup and pretty benchmarked graphs and stuff tomorrow.
        Hide
        Ted Yu added a comment -

        onHeapCache should be used in getBlock():

        +  /*  if ((cachedBlock = onHeapCache.getBlock(blockName, caching)) != null) {
        +      return cachedBlock;
        +    } else */ if ((cachedBlock = offHeapCache.getBlock(blockName, caching)) != null) {
        +     // onHeapCache.cacheBlock(blockName, cachedBlock);
        

        Otherwise onHeapCache.cacheBlock() should be commented out in:

        +  public void cacheBlock(String blockName, ByteBuffer buf, boolean inMemory) {
        +    onHeapCache.cacheBlock(blockName, buf, inMemory);
        +    offHeapCache.cacheBlock(blockName, buf);
        +  }
        
        Show
        Ted Yu added a comment - onHeapCache should be used in getBlock(): + /* if ((cachedBlock = onHeapCache.getBlock(blockName, caching)) != null ) { + return cachedBlock; + } else */ if ((cachedBlock = offHeapCache.getBlock(blockName, caching)) != null ) { + // onHeapCache.cacheBlock(blockName, cachedBlock); Otherwise onHeapCache.cacheBlock() should be commented out in: + public void cacheBlock( String blockName, ByteBuffer buf, boolean inMemory) { + onHeapCache.cacheBlock(blockName, buf, inMemory); + offHeapCache.cacheBlock(blockName, buf); + }
        Hide
        Li Pi added a comment -

        whoops. I have no idea why thats still commented out.

        Show
        Li Pi added a comment - whoops. I have no idea why thats still commented out.
        Hide
        Li Pi added a comment -

        fixed disabled on heap cache.

        Show
        Li Pi added a comment - fixed disabled on heap cache.
        Hide
        Li Pi added a comment -

        I'm gonna upload a PDF of benchmark results, but results are interesting. Running HBase standalone, with a small enough dataset so that all files are in local FS cache, the SlabCache substantially outperforms the FS Cache, even if checksumming is disabled.

        Compression is disabled as well.

        Show
        Li Pi added a comment - I'm gonna upload a PDF of benchmark results, but results are interesting. Running HBase standalone, with a small enough dataset so that all files are in local FS cache, the SlabCache substantially outperforms the FS Cache, even if checksumming is disabled. Compression is disabled as well.
        Hide
        Ted Yu added a comment -

        Why is the handling below different from that in cacheBlock(String blockName, ByteBuffer buf, boolean inMemory) ?

          public void cacheBlock(String blockName, ByteBuffer buf) {
            // onHeapCache.cacheBlock(blockName, buf);
        

        Overall, more javadoc would help others grasp the functionality for each class.

        Show
        Ted Yu added a comment - Why is the handling below different from that in cacheBlock(String blockName, ByteBuffer buf, boolean inMemory) ? public void cacheBlock( String blockName, ByteBuffer buf) { // onHeapCache.cacheBlock(blockName, buf); Overall, more javadoc would help others grasp the functionality for each class.
        Hide
        stack added a comment -

        Patch looks excellent!

        You want that change to surefire heap size in pom.xml? Are you saying here that tests won't pass unless you have 5G of RAM?

        In apache code base, you cannot have '@author' tags.

        WTF does DoubleBlockCache dooo? (smile). Usually folks will fill in the class comment on the what/why a class exists. Would be good here; e.g. why is it called DoubleBlockCache? (Similar for other new classes you've added)

        I love this:

        +  private LruBlockCache onHeapCache;
        +  private SlabCache offHeapCache;
        

        Thats sweet.

        Remove code rather than comment it out: e.g. + // onHeapCache.cacheBlock(blockName, buf);

        And when we cache a block, how we know where to put it? On or off heap? How you decide where to put it (Looks like you put it off heap always here).

        Any chance of some offheap stats (is this getStats used?)

        We only return heapsize of onheap cache. You think we should not include offheap?

        Yeah, a bunch of these state methods go unimplemented. Can we do any of them? Or is it that they just don't make sense in offheap context?

        Do the assign when you declare the blockSize and slabs data members rather than wait till the constructor runs. Make them final while you are at it. Similar comment for SingleSizeCache, etc.

        What is this limit? int blocksPerSlab = Integer.MAX_VALUE / blockSize; Max of 4G per slab?

        Should slabsize just be hardcoded as Integer.MAX_VALUE?

        The kung fu going on in the shutdown of metaslab needs a comment. I think I know whats going on. Explain what 'cleaner' is.

        Is there something up w/ the naming in MetaSlab or is it me misreading? We are iterating up to blocksPerSlab but we are creating slabs, not blocks. I'd think we'd be making blocks in a slab, not slabs.

        You think this is going to happen or how could it happen?

        +    if(backingMap.containsKey(blockName)){
        +      throw new RuntimeException("Cached an already cached block");
        +    }
        

        Whats going on here (I see this in a few places):

        +    } catch (Exception e) {
        +
        +    }
        

        The get from backingStore will never return a null here?

        + contentBlock = backingStore.get(key).getBlock(key, caching);

        Is this a good name for the default blocksize config? "hbase.mapreduce.hfileoutputformat.blocksize" The 'outputformat' would seem to come from mapreduce world (Which seems to be where you found it if I grep src). SHould we be using DEFAULT_BLOCKSIZE from hfile here instead?

        There are some copy paste issues with the javadoc you have in your TestSingleSlabCache.

        Hey Li, you get an F in documentation (smile) but the code is great. Good stuff.

        Show
        stack added a comment - Patch looks excellent! You want that change to surefire heap size in pom.xml? Are you saying here that tests won't pass unless you have 5G of RAM? In apache code base, you cannot have '@author' tags. WTF does DoubleBlockCache dooo? (smile). Usually folks will fill in the class comment on the what/why a class exists. Would be good here; e.g. why is it called DoubleBlockCache? (Similar for other new classes you've added) I love this: + private LruBlockCache onHeapCache; + private SlabCache offHeapCache; Thats sweet. Remove code rather than comment it out: e.g. + // onHeapCache.cacheBlock(blockName, buf); And when we cache a block, how we know where to put it? On or off heap? How you decide where to put it (Looks like you put it off heap always here). Any chance of some offheap stats (is this getStats used?) We only return heapsize of onheap cache. You think we should not include offheap? Yeah, a bunch of these state methods go unimplemented. Can we do any of them? Or is it that they just don't make sense in offheap context? Do the assign when you declare the blockSize and slabs data members rather than wait till the constructor runs. Make them final while you are at it. Similar comment for SingleSizeCache, etc. What is this limit? int blocksPerSlab = Integer.MAX_VALUE / blockSize; Max of 4G per slab? Should slabsize just be hardcoded as Integer.MAX_VALUE? The kung fu going on in the shutdown of metaslab needs a comment. I think I know whats going on. Explain what 'cleaner' is. Is there something up w/ the naming in MetaSlab or is it me misreading? We are iterating up to blocksPerSlab but we are creating slabs, not blocks. I'd think we'd be making blocks in a slab, not slabs. You think this is going to happen or how could it happen? + if (backingMap.containsKey(blockName)){ + throw new RuntimeException( "Cached an already cached block" ); + } Whats going on here (I see this in a few places): + } catch (Exception e) { + + } The get from backingStore will never return a null here? + contentBlock = backingStore.get(key).getBlock(key, caching); Is this a good name for the default blocksize config? "hbase.mapreduce.hfileoutputformat.blocksize" The 'outputformat' would seem to come from mapreduce world (Which seems to be where you found it if I grep src). SHould we be using DEFAULT_BLOCKSIZE from hfile here instead? There are some copy paste issues with the javadoc you have in your TestSingleSlabCache. Hey Li, you get an F in documentation (smile) but the code is great. Good stuff.
        Hide
        Li Pi added a comment -

        Thanks!

        Theres quite a few bugs I just discovered. I'm fixing them right now. I'm adding reasonable documentation too now.

        Show
        Li Pi added a comment - Thanks! Theres quite a few bugs I just discovered. I'm fixing them right now. I'm adding reasonable documentation too now.
        Hide
        Li Pi added a comment -

        @Ted Yu - thats also a bug. Apparently, I diffed against a testing branch of this patch. Whoops. This explains the commented out code.

        Show
        Li Pi added a comment - @Ted Yu - thats also a bug. Apparently, I diffed against a testing branch of this patch. Whoops. This explains the commented out code.
        Hide
        Ted Yu added a comment -

        The code Stack mentioned:

        + contentBlock = backingStore.get(key).getBlock(key, caching);
        

        would lead to frequent NPE.

        There were only 87 entries in ConcurrentHashMap (backingStore) after cache.cacheBlock() call at line 58 in TestSlabCache
        The assertion at line 64 would fail.

        Show
        Ted Yu added a comment - The code Stack mentioned: + contentBlock = backingStore.get(key).getBlock(key, caching); would lead to frequent NPE. There were only 87 entries in ConcurrentHashMap (backingStore) after cache.cacheBlock() call at line 58 in TestSlabCache The assertion at line 64 would fail.
        Hide
        Li Pi added a comment -

        fixed a couple of the bugs mentioned, will add documentation and do more extensive testing in another diff.

        Show
        Li Pi added a comment - fixed a couple of the bugs mentioned, will add documentation and do more extensive testing in another diff.
        Hide
        Li Pi added a comment -

        @Ted, if it NPE's, that means that the cache doesn't contain the required item, so it's a cache miss.

        Show
        Li Pi added a comment - @Ted, if it NPE's, that means that the cache doesn't contain the required item, so it's a cache miss.
        Hide
        Li Pi added a comment -

        fixed test.

        Show
        Li Pi added a comment - fixed test.
        Hide
        Li Pi added a comment -

        added some documentation, will add more.

        Show
        Li Pi added a comment - added some documentation, will add more.
        Hide
        Li Pi added a comment -

        -And when we cache a block, how we know where to put it? On or off heap? How you decide where to put it (Looks like you put it off heap always here).

        When we catch a block, we throw it into on-heap. Originally, I planned for it to go off heap, but as the cache partitions in LRUBlockCache gives it scan resistance already, this seemed unnecessary.

        -Any chance of some offheap stats (is this getStats used?)

        getStats now works for the offheap cache, at least for eviction count/cache hit rates.

        -We only return heapsize of onheap cache. You think we should not include offheap?

        I figured since offheap wasn't really in the heap, I didn't include it. Not sure if this was the best option.

        -Yeah, a bunch of these state methods go unimplemented. Can we do any of them? Or is it that they just don't make sense in offheap context?

        I took a look, implemented some of them. SlabCache implements everything except getFreeSize. Since all the space it's given is allocated using directbytebuffers immediately, theres never any free size. So I have it return 0.

        -What is this limit? int blocksPerSlab = Integer.MAX_VALUE / blockSize; Max of 4G per slab?

        ByteBuffer positions are addressed using integers. You can only declare one up to 2gb using ByteBuffer.allocateDirect().

        -Should slabsize just be hardcoded as Integer.MAX_VALUE?

        Sometimes we want a slab thats less than 2gb. Such as when the size for our entire cache is smaller than that.

        -The kung fu going on in the shutdown of metaslab needs a comment. I think I know whats going on. Explain what 'cleaner' is.

        Yeah, this is how you deallocate directbytebuffers. Added comments, but its still pretty voodoo. Basically cleaner is a deconstructor for a direct byte buffer.

        -Is there something up w/ the naming in MetaSlab or is it me misreading? We are iterating up to blocksPerSlab but we are creating slabs, not blocks. I'd think we'd be making blocks in a slab, not slabs.

        If we want to create N blocks of size X, and a slab, can, at max, contain B blocks, we decrement N by B until N is below B. We create a slab each time and divide that slab into blocks. If N < B and N > 0, then we create our final slab that we use to get space for the remaining blocks.

        -You think this is going to happen or how could it happen?

        This shouldn't happen unless we have a race condition. But we might, in which case we should throw an exception. I'll add some comments to this portion.

        -Whats going on here (I see this in a few places):

        If we get a null pointer exception, that means the cache missed. Therefore we return null, increment the missed counter.

        -The get from backingStore will never return a null here?

        As an invariant, it should never. Because if we are running out of buffers, one should be evicted by the ConcurrentLinkedHashMap when we do a read. On closer inspection, this can happen in a multithreaded environment, I'll figure out a way to fix this. (Probably by synchronizing it.)

        -Is this a good name for the default blocksize config? "hbase.mapreduce.hfileoutputformat.blocksize" The 'outputformat' would seem to come from mapreduce world (Which seems to be where you found it if I grep src). SHould we be using DEFAULT_BLOCKSIZE from hfile here instead?

        Switched it to DEFAULT_BLOCKSIZE.

        Apologies for the first few patches. Apparently I diffed against a different branch than I had intended. Thus the pom.xml edits and commented out code. That was for speed testing and benchmarking. I fixed a few more things, and will continue to search for bugs and add documentation. I'm also implementing better metrics.

        Show
        Li Pi added a comment - -And when we cache a block, how we know where to put it? On or off heap? How you decide where to put it (Looks like you put it off heap always here). When we catch a block, we throw it into on-heap. Originally, I planned for it to go off heap, but as the cache partitions in LRUBlockCache gives it scan resistance already, this seemed unnecessary. -Any chance of some offheap stats (is this getStats used?) getStats now works for the offheap cache, at least for eviction count/cache hit rates. -We only return heapsize of onheap cache. You think we should not include offheap? I figured since offheap wasn't really in the heap, I didn't include it. Not sure if this was the best option. -Yeah, a bunch of these state methods go unimplemented. Can we do any of them? Or is it that they just don't make sense in offheap context? I took a look, implemented some of them. SlabCache implements everything except getFreeSize. Since all the space it's given is allocated using directbytebuffers immediately, theres never any free size. So I have it return 0. -What is this limit? int blocksPerSlab = Integer.MAX_VALUE / blockSize; Max of 4G per slab? ByteBuffer positions are addressed using integers. You can only declare one up to 2gb using ByteBuffer.allocateDirect(). -Should slabsize just be hardcoded as Integer.MAX_VALUE? Sometimes we want a slab thats less than 2gb. Such as when the size for our entire cache is smaller than that. -The kung fu going on in the shutdown of metaslab needs a comment. I think I know whats going on. Explain what 'cleaner' is. Yeah, this is how you deallocate directbytebuffers. Added comments, but its still pretty voodoo. Basically cleaner is a deconstructor for a direct byte buffer. -Is there something up w/ the naming in MetaSlab or is it me misreading? We are iterating up to blocksPerSlab but we are creating slabs, not blocks. I'd think we'd be making blocks in a slab, not slabs. If we want to create N blocks of size X, and a slab, can, at max, contain B blocks, we decrement N by B until N is below B. We create a slab each time and divide that slab into blocks. If N < B and N > 0, then we create our final slab that we use to get space for the remaining blocks. -You think this is going to happen or how could it happen? This shouldn't happen unless we have a race condition. But we might, in which case we should throw an exception. I'll add some comments to this portion. -Whats going on here (I see this in a few places): If we get a null pointer exception, that means the cache missed. Therefore we return null, increment the missed counter. -The get from backingStore will never return a null here? As an invariant, it should never. Because if we are running out of buffers, one should be evicted by the ConcurrentLinkedHashMap when we do a read. On closer inspection, this can happen in a multithreaded environment, I'll figure out a way to fix this. (Probably by synchronizing it.) -Is this a good name for the default blocksize config? "hbase.mapreduce.hfileoutputformat.blocksize" The 'outputformat' would seem to come from mapreduce world (Which seems to be where you found it if I grep src). SHould we be using DEFAULT_BLOCKSIZE from hfile here instead? Switched it to DEFAULT_BLOCKSIZE. Apologies for the first few patches. Apparently I diffed against a different branch than I had intended. Thus the pom.xml edits and commented out code. That was for speed testing and benchmarking. I fixed a few more things, and will continue to search for bugs and add documentation. I'm also implementing better metrics.
        Hide
        stack added a comment -

        When we catch a block, we throw it into on-heap.

        At one point does stuff go into the offheap cache?

        I figured since offheap wasn't really in the heap, I didn't include it. Not sure if this was the best option.

        How would an operator monitor offheap if no stats. What should they do? Its kinda tough because the cache interface only allows return of one 'size'

        Apologies for the first few patches.

        Don't apologize. This back and forth is healthy.

        Looks like the bulk of my comments above can be addressed by comment in code.

        Let us know when you think it ready for new review.

        Show
        stack added a comment - When we catch a block, we throw it into on-heap. At one point does stuff go into the offheap cache? I figured since offheap wasn't really in the heap, I didn't include it. Not sure if this was the best option. How would an operator monitor offheap if no stats. What should they do? Its kinda tough because the cache interface only allows return of one 'size' Apologies for the first few patches. Don't apologize. This back and forth is healthy. Looks like the bulk of my comments above can be addressed by comment in code. Let us know when you think it ready for new review.
        Hide
        Li Pi added a comment -
        At one point does stuff go into the offheap cache?

        Err, when we catch a block, it gets thrown into both on-heap and off-heap.

        How would an operator monitor offheap if no stats. What should they do? Its kinda tough because the cache interface only allows return of one 'size'

        Thats an unfortunate problem, seeing as the off heap cache is technically made of many differently sized caches. I'll have them log info the same was as the on heap cache does every once in a while.

        Show
        Li Pi added a comment - At one point does stuff go into the offheap cache? Err, when we catch a block, it gets thrown into both on-heap and off-heap. How would an operator monitor offheap if no stats. What should they do? Its kinda tough because the cache interface only allows return of one 'size' Thats an unfortunate problem, seeing as the off heap cache is technically made of many differently sized caches. I'll have them log info the same was as the on heap cache does every once in a while.
        Hide
        Todd Lipcon added a comment -

        In my opinion we need a servlet or jamon page which can dump the stats of the various slab classes as well as their hit/miss rates.

        Alternatively, a new MetricsContext for the slab info.

        But, somehow, we need to expose information similar to memcached's "stats items" and "stats slabs"

        Show
        Todd Lipcon added a comment - In my opinion we need a servlet or jamon page which can dump the stats of the various slab classes as well as their hit/miss rates. Alternatively, a new MetricsContext for the slab info. But, somehow, we need to expose information similar to memcached's "stats items" and "stats slabs"
        Hide
        Li Pi added a comment -

        Yeah, thats what I'm working on as well. Some easy visual way to expose stats.

        Time to learn webdev!

        Show
        Li Pi added a comment - Yeah, thats what I'm working on as well. Some easy visual way to expose stats. Time to learn webdev!
        Hide
        stack added a comment -

        @Li Related is HBASE-4057?

        Show
        stack added a comment - @Li Related is HBASE-4057 ?
        Hide
        Ted Yu added a comment -

        In TestSlabCache:

        +    Block[] blocks = generateRandomBlocks(87, blockSize);
        

        We'd better cover more cases where cache miss is exercised.

        Show
        Ted Yu added a comment - In TestSlabCache: + Block[] blocks = generateRandomBlocks(87, blockSize); We'd better cover more cases where cache miss is exercised.
        Hide
        Li Pi added a comment -

        Added tests for eviction, now logs finely grained stats to file.

        Added a bunch of documentation. A bunch - this should take care of most of the documentation concerns.

        Show
        Li Pi added a comment - Added tests for eviction, now logs finely grained stats to file. Added a bunch of documentation. A bunch - this should take care of most of the documentation concerns.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1191/
        -----------------------------------------------------------

        Review request for hbase and Todd Lipcon.

        Summary
        -------

        Uploading slabcachepatchv4 to review for Li Pi.

        This addresses bug HBASE-4027.
        https://issues.apache.org/jira/browse/HBASE-4027

        Diffs


        conf/hbase-env.sh 2d55d27
        pom.xml 729dc37
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552
        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1191/diff

        Testing
        -------

        Thanks,

        Todd

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1191/ ----------------------------------------------------------- Review request for hbase and Todd Lipcon. Summary ------- Uploading slabcachepatchv4 to review for Li Pi. This addresses bug HBASE-4027 . https://issues.apache.org/jira/browse/HBASE-4027 Diffs conf/hbase-env.sh 2d55d27 pom.xml 729dc37 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552 src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1191/diff Testing ------- Thanks, Todd
        Hide
        stack added a comment -

        Doc is great.

        These could be final:

        + private LruBlockCache onHeapCache;
        + private SlabCache offHeapCache;

        Says 'Metrics are the combined size and hits and misses of both caches' but down in getStats we seem to be getting onheap stats only. Intentional? Same for heapSize.

        Do you want to leave this line in hfile? + LOG.debug("decompressedSize = " + decompressedSize);

        Whats it mean when you say 'An exception will be thrown if the cached data is larger than the size of the allocated block'?

        More notes later.

        Show
        stack added a comment - Doc is great. These could be final: + private LruBlockCache onHeapCache; + private SlabCache offHeapCache; Says 'Metrics are the combined size and hits and misses of both caches' but down in getStats we seem to be getting onheap stats only. Intentional? Same for heapSize. Do you want to leave this line in hfile? + LOG.debug("decompressedSize = " + decompressedSize); Whats it mean when you say 'An exception will be thrown if the cached data is larger than the size of the allocated block'? More notes later.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1191/#review1182
        -----------------------------------------------------------

        could do with some tests for MetaSlab. also some multi-threaded tests - see MultithreadedTestUtil, example usage in TestMemStoreLAB

        pom.xml
        <https://reviews.apache.org/r/1191/#comment2484>

        did you determine that this ConcurrentLinkedHashMap was different than the one in Guava? I thought it got incorporated into Guava, which we already depend on.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2485>

        punctuation wise, I think it would be easier to read if you hyphenated on-heap and off-heap. This applies to log messages below as well.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2486>

        No need to line-break here

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2487>

        consider using StringUtils.humanReadableInt for these sizes.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2488>

        @Override

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2489>

        when you're just overriding something from the superclass, no need for javadoc unless it says something new and exciting. If you feel like you want to put something there, you can use /**

        {@inheritDoc}

        */ to be explicit that you're inheriting from the superclass.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2490>

        I think you should only put-back into the on-heap cache in the case that the 'caching' parameter is true.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2491>

        hrm, the class javadoc says that the statistics should be cumulative, but this seems to just forward

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2492>

        TODOs

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1191/#comment2493>

        is this code used? seems like dead copy-paste code to me.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        <https://reviews.apache.org/r/1191/#comment2497>

        extraneous debugging left in

        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
        <https://reviews.apache.org/r/1191/#comment2498>

        I think this is usually called a "slab class" - I think that name would be less confusing, since "Meta" is already used in HBase to refer to .META.

        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
        <https://reviews.apache.org/r/1191/#comment2499>

        unclear what the difference is between the two.

        Is "slabs" the list of 2GB buffers, and "buffers" is the list of actual items that will be allocated? I think the traditional names here are "slabs" and "items". where each slab holds some number of allocatable items

        Also, rather than // comments, use /** javadoc comments */ before the vars

        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
        <https://reviews.apache.org/r/1191/#comment2500>

        these vars probably better called maxBlocksPerSlab and maxSlabSize, since they're upper bounds.

        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
        <https://reviews.apache.org/r/1191/#comment2501>

        I think this code would be a little easier to understand if you split it into one loop for the full slabs, and an if statement for the partially full one. Something like:

        int numFullSlabs = numBlocks / maxBlocksPerSlab;
        boolean hasPartialSlab = (numBlocks % maxBlocksPerSlab) > 0;

        for (int i = 0; i < numFullSlabs; i++)

        { alloc one of maxSlabSize; addBuffersForSlab(slab); }

        if (hasPartialSlab)

        { alloc the partial one addBuffersForSlab(slab); }

        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
        <https://reviews.apache.org/r/1191/#comment2502>

        should be a LOG.warn

        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
        <https://reviews.apache.org/r/1191/#comment2503>

        shouldn't this class have an alloc() and free() method?

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2511>

        shouldn't this implement BlockCache?

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2504>

        no need to line-break

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2505>

        It seems dirty to reach back upwards to "master" here. Cyclical dependencies are a code smell...

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2508>

        I don't understand what purpose returnMap is serving here

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2506>

        add '@Override', then you don't need to copy-paste javadoc

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2507>

        this is where you would call backingStore.alloc() – then buffers can be made into a private member and keep encapsulation

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2510>

        @Override, remove javadoc

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2509>

        this is super ugly, plus bad performance - throwing an exception is very expensive, since it has to construct a whole stack trace object, etc.

        Instead, check backingMap.get(key) against null

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2512>

        why does master need to know about this?

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2495>

        numBlocks isn't the size, is it?

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2496>

        missing space before "blocks"

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
        <https://reviews.apache.org/r/1191/#comment2494>

        there are several identical copies of this floating around. Can they all use the same class?

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2513>

        overall note: I think we could move all of this to a new io.hfile.slab package

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2520>

        should this implement BlockCache?

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2515>

        since this is only modified during initialization, we can use an unsynchronized TreeMap which is probably more efficient

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2516>

        rename to statThreadPeriodSecs to indicate time unit.

        Though, since it's a constant, it probably should be STAT_THREAD_PERIOD. This should be configurable but we can address in a follow-up

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2525>

        you can set the thread factory to make daemon threads, then you don't need setDaemon(true) below

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2514>

        line breaks unnecessary

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2517>

        this stuff will have to be configurable too. but this is fine for now

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2518>

        should also note that it returns 0 for the case of an uncacheable object

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2519>

        this function is never used – you should make it private, and then call it from cacheBlock below.

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2521>

        again this is no good. check for null instead of catching an exception

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2522>

        when would this be true? evicting something that doesn't have an entry in backingStore?

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2523>

        it seems like there's a loop in flow control here.

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2524>

        move this nice logging output to something like: slabstats.logUsage()

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2526>

        since you're using scheduleAtFixedRate, this should implement Runnable, not extend Thread

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2527>

        need to be public?

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2528>

        should be ALL_CAPS

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2529>

        ALSO_ALL_CAPS

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
        <https://reviews.apache.org/r/1191/#comment2530>

        why boxed longs?

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        <https://reviews.apache.org/r/1191/#comment2531>

        for consistency, I think we should name these something like:
        hfile.block.cache.offheap.size
        hfile.block.cache.offheap.minblocksize

        also, is min block size really a min? isn't it more like expectedblocksize?

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1191/#comment2532>

        update this javadoc

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1191/#comment2533>

        should use JUnit 4 style tests in new code - i.e not inherit TestCase.

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1191/#comment2535>

        nowhere in this test does it actually verify that the data is valid

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1191/#comment2534>

        some javadoc would be nice... it seems this code is copy-pasted from TestLruBlockCache, but really doesn't make sense when applied to the SingleSlabCache

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java
        <https://reviews.apache.org/r/1191/#comment2536>

        see above.

        If you can refactor some of this code into something like BlockCacheTestUtils.testBasicCacheContract(BlockCache cache) that would be good... copy paste evil

        • Todd

        On 2011-07-25 22:55:56, Todd Lipcon wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1191/

        -----------------------------------------------------------

        (Updated 2011-07-25 22:55:56)

        Review request for hbase and Todd Lipcon.

        Summary

        -------

        Uploading slabcachepatchv4 to review for Li Pi.

        This addresses bug HBASE-4027.

        https://issues.apache.org/jira/browse/HBASE-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        pom.xml 729dc37

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552

        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1191/diff

        Testing

        -------

        Thanks,

        Todd

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1191/#review1182 ----------------------------------------------------------- could do with some tests for MetaSlab. also some multi-threaded tests - see MultithreadedTestUtil, example usage in TestMemStoreLAB pom.xml < https://reviews.apache.org/r/1191/#comment2484 > did you determine that this ConcurrentLinkedHashMap was different than the one in Guava? I thought it got incorporated into Guava, which we already depend on. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2485 > punctuation wise, I think it would be easier to read if you hyphenated on-heap and off-heap. This applies to log messages below as well. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2486 > No need to line-break here src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2487 > consider using StringUtils.humanReadableInt for these sizes. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2488 > @Override src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2489 > when you're just overriding something from the superclass, no need for javadoc unless it says something new and exciting. If you feel like you want to put something there, you can use /** {@inheritDoc} */ to be explicit that you're inheriting from the superclass. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2490 > I think you should only put-back into the on-heap cache in the case that the 'caching' parameter is true. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2491 > hrm, the class javadoc says that the statistics should be cumulative, but this seems to just forward src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2492 > TODOs src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1191/#comment2493 > is this code used? seems like dead copy-paste code to me. src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java < https://reviews.apache.org/r/1191/#comment2497 > extraneous debugging left in src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java < https://reviews.apache.org/r/1191/#comment2498 > I think this is usually called a "slab class" - I think that name would be less confusing, since "Meta" is already used in HBase to refer to .META. src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java < https://reviews.apache.org/r/1191/#comment2499 > unclear what the difference is between the two. Is "slabs" the list of 2GB buffers, and "buffers" is the list of actual items that will be allocated? I think the traditional names here are "slabs" and "items". where each slab holds some number of allocatable items Also, rather than // comments, use /** javadoc comments */ before the vars src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java < https://reviews.apache.org/r/1191/#comment2500 > these vars probably better called maxBlocksPerSlab and maxSlabSize, since they're upper bounds. src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java < https://reviews.apache.org/r/1191/#comment2501 > I think this code would be a little easier to understand if you split it into one loop for the full slabs, and an if statement for the partially full one. Something like: int numFullSlabs = numBlocks / maxBlocksPerSlab; boolean hasPartialSlab = (numBlocks % maxBlocksPerSlab) > 0; for (int i = 0; i < numFullSlabs; i++) { alloc one of maxSlabSize; addBuffersForSlab(slab); } if (hasPartialSlab) { alloc the partial one addBuffersForSlab(slab); } src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java < https://reviews.apache.org/r/1191/#comment2502 > should be a LOG.warn src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java < https://reviews.apache.org/r/1191/#comment2503 > shouldn't this class have an alloc() and free() method? src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2511 > shouldn't this implement BlockCache? src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2504 > no need to line-break src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2505 > It seems dirty to reach back upwards to "master" here. Cyclical dependencies are a code smell... src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2508 > I don't understand what purpose returnMap is serving here src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2506 > add '@Override', then you don't need to copy-paste javadoc src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2507 > this is where you would call backingStore.alloc() – then buffers can be made into a private member and keep encapsulation src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2510 > @Override, remove javadoc src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2509 > this is super ugly, plus bad performance - throwing an exception is very expensive, since it has to construct a whole stack trace object, etc. Instead, check backingMap.get(key) against null src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2512 > why does master need to know about this? src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2495 > numBlocks isn't the size, is it? src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2496 > missing space before "blocks" src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java < https://reviews.apache.org/r/1191/#comment2494 > there are several identical copies of this floating around. Can they all use the same class? src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2513 > overall note: I think we could move all of this to a new io.hfile.slab package src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2520 > should this implement BlockCache? src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2515 > since this is only modified during initialization, we can use an unsynchronized TreeMap which is probably more efficient src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2516 > rename to statThreadPeriodSecs to indicate time unit. Though, since it's a constant, it probably should be STAT_THREAD_PERIOD. This should be configurable but we can address in a follow-up src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2525 > you can set the thread factory to make daemon threads, then you don't need setDaemon(true) below src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2514 > line breaks unnecessary src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2517 > this stuff will have to be configurable too. but this is fine for now src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2518 > should also note that it returns 0 for the case of an uncacheable object src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2519 > this function is never used – you should make it private, and then call it from cacheBlock below. src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2521 > again this is no good. check for null instead of catching an exception src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2522 > when would this be true? evicting something that doesn't have an entry in backingStore? src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2523 > it seems like there's a loop in flow control here. src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2524 > move this nice logging output to something like: slabstats.logUsage() src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2526 > since you're using scheduleAtFixedRate, this should implement Runnable, not extend Thread src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2527 > need to be public? src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2528 > should be ALL_CAPS src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2529 > ALSO_ALL_CAPS src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java < https://reviews.apache.org/r/1191/#comment2530 > why boxed longs? src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java < https://reviews.apache.org/r/1191/#comment2531 > for consistency, I think we should name these something like: hfile.block.cache.offheap.size hfile.block.cache.offheap.minblocksize also, is min block size really a min? isn't it more like expectedblocksize? src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java < https://reviews.apache.org/r/1191/#comment2532 > update this javadoc src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java < https://reviews.apache.org/r/1191/#comment2533 > should use JUnit 4 style tests in new code - i.e not inherit TestCase. src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java < https://reviews.apache.org/r/1191/#comment2535 > nowhere in this test does it actually verify that the data is valid src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java < https://reviews.apache.org/r/1191/#comment2534 > some javadoc would be nice... it seems this code is copy-pasted from TestLruBlockCache, but really doesn't make sense when applied to the SingleSlabCache src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java < https://reviews.apache.org/r/1191/#comment2536 > see above. If you can refactor some of this code into something like BlockCacheTestUtils.testBasicCacheContract(BlockCache cache) that would be good... copy paste evil Todd On 2011-07-25 22:55:56, Todd Lipcon wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1191/ ----------------------------------------------------------- (Updated 2011-07-25 22:55:56) Review request for hbase and Todd Lipcon. Summary ------- Uploading slabcachepatchv4 to review for Li Pi. This addresses bug HBASE-4027 . https://issues.apache.org/jira/browse/HBASE-4027 Diffs ----- conf/hbase-env.sh 2d55d27 pom.xml 729dc37 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552 src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1191/diff Testing ------- Thanks, Todd
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > pom.xml, line 712

        > <https://reviews.apache.org/r/1191/diff/1/?file=26914#file26914line712>

        >

        > did you determine that this ConcurrentLinkedHashMap was different than the one in Guava? I thought it got incorporated into Guava, which we already depend on.

        Changed to MapMaker. Guava has deprecated some of the functions I'm using in trunk, but hasn't provided a release with a replacement. But we'll deal with that when the time comes.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 33

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line33>

        >

        > punctuation wise, I think it would be easier to read if you hyphenated on-heap and off-heap. This applies to log messages below as well.

        Done

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 52-55

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line52>

        >

        > No need to line-break here

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 64-65

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line64>

        >

        > consider using StringUtils.humanReadableInt for these sizes.

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 72

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line72>

        >

        > @Override

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-85

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line77>

        >

        > when you're just overriding something from the superclass, no need for javadoc unless it says something new and exciting. If you feel like you want to put something there, you can use /** {@inheritDoc} */ to be explicit that you're inheriting from the superclass.

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 102

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line102>

        >

        > I think you should only put-back into the on-heap cache in the case that the 'caching' parameter is true.

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 128

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line128>

        >

        > hrm, the class javadoc says that the statistics should be cumulative, but this seems to just forward

        Fixed. Also refactored CacheStats into a separate class.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 148-166

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line148>

        >

        > TODOs

        Implemented these.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 168

        > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line168>

        >

        > is this code used? seems like dead copy-paste code to me.

        It is now.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java, line 1125

        > <https://reviews.apache.org/r/1191/diff/1/?file=26916#file26916line1125>

        >

        > extraneous debugging left in

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 31

        > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line31>

        >

        > I think this is usually called a "slab class" - I think that name would be less confusing, since "Meta" is already used in HBase to refer to .META.

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 32-34

        > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line32>

        >

        > unclear what the difference is between the two.

        >

        > Is "slabs" the list of 2GB buffers, and "buffers" is the list of actual items that will be allocated? I think the traditional names here are "slabs" and "items". where each slab holds some number of allocatable items

        >

        > Also, rather than // comments, use /** javadoc comments */ before the vars

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 46

        > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line46>

        >

        > these vars probably better called maxBlocksPerSlab and maxSlabSize, since they're upper bounds.

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 49-55

        > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line49>

        >

        > I think this code would be a little easier to understand if you split it into one loop for the full slabs, and an if statement for the partially full one. Something like:

        >

        > int numFullSlabs = numBlocks / maxBlocksPerSlab;

        > boolean hasPartialSlab = (numBlocks % maxBlocksPerSlab) > 0;

        >

        > for (int i = 0; i < numFullSlabs; i++) { bq. > alloc one of maxSlabSize; bq. > addBuffersForSlab(slab); bq. > }

        >

        > if (hasPartialSlab) { bq. > alloc the partial one bq. > addBuffersForSlab(slab); bq. > }

        >

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 77-78

        > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line77>

        >

        > should be a LOG.warn

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 43

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line43>

        >

        > shouldn't this implement BlockCache?

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 58-65

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line58>

        >

        > no need to line-break

        Eclipse automatically does this. Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 67

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line67>

        >

        > It seems dirty to reach back upwards to "master" here. Cyclical dependencies are a code smell...

        If the SingleSizeSlab decides to evict, the master should evict as well.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 70

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line70>

        >

        > I don't understand what purpose returnMap is serving here

        ReturnMap is pretty much how free() is implemented. I probably should move this to MetaSlab - now Slab

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 94

        > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line94>

        >

        > shouldn't this class have an alloc() and free() method?

        Moving alloc() and free() functions from SingleSizeCache to MetaSlab. Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 106

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line106>

        >

        > this is where you would call backingStore.alloc() – then buffers can be made into a private member and keep encapsulation

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 123

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line123>

        >

        > @Override, remove javadoc

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 96-99

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line96>

        >

        > add '@Override', then you don't need to copy-paste javadoc

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 131-133

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line131>

        >

        > this is super ugly, plus bad performance - throwing an exception is very expensive, since it has to construct a whole stack trace object, etc.

        >

        > Instead, check backingMap.get(key) against null

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 155

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line155>

        >

        > why does master need to know about this?

        Master keeps its own records for which SingleSizeCache the key is assigned to.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 164

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line164>

        >

        > numBlocks isn't the size, is it?

        Good catch. Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 166

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line166>

        >

        > missing space before "blocks"

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 207

        > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line207>

        >

        > there are several identical copies of this floating around. Can they all use the same class?

        Refactored out and fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 21

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line21>

        >

        > overall note: I think we could move all of this to a new io.hfile.slab package

        Will do next revision.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 44

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line44>

        >

        > should this implement BlockCache?

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 47

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line47>

        >

        > since this is only modified during initialization, we can use an unsynchronized TreeMap which is probably more efficient

        Fixed. Although, in the future, we may want to modify it on the fly, which is why it was a ConcurrentSkipListMap.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 49

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line49>

        >

        > rename to statThreadPeriodSecs to indicate time unit.

        >

        > Though, since it's a constant, it probably should be STAT_THREAD_PERIOD. This should be configurable but we can address in a follow-up

        fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 53

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line53>

        >

        > you can set the thread factory to make daemon threads, then you don't need setDaemon(true) below

        changed to runnable.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 65-68

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line65>

        >

        > line breaks unnecessary

        eclipse autoformatting . fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 80-85

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line80>

        >

        > this stuff will have to be configurable too. but this is fine for now

        next revision!

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 95

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line95>

        >

        > should also note that it returns 0 for the case of an uncacheable object

        changed so it returns the entry instead of an int. returns null if unreachable.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 98

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line98>

        >

        > this function is never used – you should make it private, and then call it from cacheBlock below.

        fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 145-148

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line145>

        >

        > again this is no good. check for null instead of catching an exception

        fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 174

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line174>

        >

        > when would this be true? evicting something that doesn't have an entry in backingStore?

        It'll be true because theres a callback from SlabCache.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 177

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line177>

        >

        > it seems like there's a loop in flow control here.

        Nope, SingleSlabCache should return false, and not loop.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 214

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line214>

        >

        > move this nice logging output to something like: slabstats.logUsage()

        Done.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 229

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line229>

        >

        > since you're using scheduleAtFixedRate, this should implement Runnable, not extend Thread

        Done

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 250

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line250>

        >

        > need to be public?

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 253

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line253>

        >

        > should be ALL_CAPS

        Done

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 254

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line254>

        >

        > ALSO_ALL_CAPS

        Done.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 268

        > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line268>

        >

        > why boxed longs?

        Changed to AtomicLong[]

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 369

        > <https://reviews.apache.org/r/1191/diff/1/?file=26920#file26920line369>

        >

        > for consistency, I think we should name these something like:

        > hfile.block.cache.offheap.size

        > hfile.block.cache.offheap.minblocksize

        >

        > also, is min block size really a min? isn't it more like expectedblocksize?

        Will do next revision.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 84

        > <https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line84>

        >

        > nowhere in this test does it actually verify that the data is valid

        Fixed.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 38

        > <https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line38>

        >

        > should use JUnit 4 style tests in new code - i.e not inherit TestCase.

        I thought since everything else was JUnit 3 style, I should leave it as JUnit 3 style.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java, line 109

        > <https://reviews.apache.org/r/1191/diff/1/?file=26922#file26922line109>

        >

        > see above.

        >

        > If you can refactor some of this code into something like BlockCacheTestUtils.testBasicCacheContract(BlockCache cache) that would be good... copy paste evil

        Will fix this when I redo all tests tomorrow.

        On 2011-07-25 23:48:42, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 95

        > <https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line95>

        >

        > some javadoc would be nice... it seems this code is copy-pasted from TestLruBlockCache, but really doesn't make sense when applied to the SingleSlabCache

        Will fix this when I redo all tests tomorrow.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1191/#review1182
        -----------------------------------------------------------

        On 2011-07-25 22:55:56, Todd Lipcon wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1191/

        -----------------------------------------------------------

        (Updated 2011-07-25 22:55:56)

        Review request for hbase and Todd Lipcon.

        Summary

        -------

        Uploading slabcachepatchv4 to review for Li Pi.

        This addresses bug HBASE-4027.

        https://issues.apache.org/jira/browse/HBASE-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        pom.xml 729dc37

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552

        src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1191/diff

        Testing

        -------

        Thanks,

        Todd

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-25 23:48:42, Todd Lipcon wrote: > pom.xml, line 712 > < https://reviews.apache.org/r/1191/diff/1/?file=26914#file26914line712 > > > did you determine that this ConcurrentLinkedHashMap was different than the one in Guava? I thought it got incorporated into Guava, which we already depend on. Changed to MapMaker. Guava has deprecated some of the functions I'm using in trunk, but hasn't provided a release with a replacement. But we'll deal with that when the time comes. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 33 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line33 > > > punctuation wise, I think it would be easier to read if you hyphenated on-heap and off-heap. This applies to log messages below as well. Done On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 52-55 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line52 > > > No need to line-break here Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 64-65 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line64 > > > consider using StringUtils.humanReadableInt for these sizes. Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 72 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line72 > > > @Override Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-85 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line77 > > > when you're just overriding something from the superclass, no need for javadoc unless it says something new and exciting. If you feel like you want to put something there, you can use /** {@inheritDoc} */ to be explicit that you're inheriting from the superclass. Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 102 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line102 > > > I think you should only put-back into the on-heap cache in the case that the 'caching' parameter is true. Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 128 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line128 > > > hrm, the class javadoc says that the statistics should be cumulative, but this seems to just forward Fixed. Also refactored CacheStats into a separate class. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 148-166 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line148 > > > TODOs Implemented these. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 168 > < https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line168 > > > is this code used? seems like dead copy-paste code to me. It is now. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java, line 1125 > < https://reviews.apache.org/r/1191/diff/1/?file=26916#file26916line1125 > > > extraneous debugging left in Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 31 > < https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line31 > > > I think this is usually called a "slab class" - I think that name would be less confusing, since "Meta" is already used in HBase to refer to .META. Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 32-34 > < https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line32 > > > unclear what the difference is between the two. > > Is "slabs" the list of 2GB buffers, and "buffers" is the list of actual items that will be allocated? I think the traditional names here are "slabs" and "items". where each slab holds some number of allocatable items > > Also, rather than // comments, use /** javadoc comments */ before the vars Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 46 > < https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line46 > > > these vars probably better called maxBlocksPerSlab and maxSlabSize, since they're upper bounds. Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 49-55 > < https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line49 > > > I think this code would be a little easier to understand if you split it into one loop for the full slabs, and an if statement for the partially full one. Something like: > > int numFullSlabs = numBlocks / maxBlocksPerSlab; > boolean hasPartialSlab = (numBlocks % maxBlocksPerSlab) > 0; > > for (int i = 0; i < numFullSlabs; i++) { bq. > alloc one of maxSlabSize; bq. > addBuffersForSlab(slab); bq. > } > > if (hasPartialSlab) { bq. > alloc the partial one bq. > addBuffersForSlab(slab); bq. > } > Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 77-78 > < https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line77 > > > should be a LOG.warn Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 43 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line43 > > > shouldn't this implement BlockCache? Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 58-65 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line58 > > > no need to line-break Eclipse automatically does this. Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 67 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line67 > > > It seems dirty to reach back upwards to "master" here. Cyclical dependencies are a code smell... If the SingleSizeSlab decides to evict, the master should evict as well. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 70 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line70 > > > I don't understand what purpose returnMap is serving here ReturnMap is pretty much how free() is implemented. I probably should move this to MetaSlab - now Slab On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 94 > < https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line94 > > > shouldn't this class have an alloc() and free() method? Moving alloc() and free() functions from SingleSizeCache to MetaSlab. Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 106 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line106 > > > this is where you would call backingStore.alloc() – then buffers can be made into a private member and keep encapsulation Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 123 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line123 > > > @Override, remove javadoc Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 96-99 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line96 > > > add '@Override', then you don't need to copy-paste javadoc Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 131-133 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line131 > > > this is super ugly, plus bad performance - throwing an exception is very expensive, since it has to construct a whole stack trace object, etc. > > Instead, check backingMap.get(key) against null Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 155 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line155 > > > why does master need to know about this? Master keeps its own records for which SingleSizeCache the key is assigned to. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 164 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line164 > > > numBlocks isn't the size, is it? Good catch. Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 166 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line166 > > > missing space before "blocks" Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 207 > < https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line207 > > > there are several identical copies of this floating around. Can they all use the same class? Refactored out and fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 21 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line21 > > > overall note: I think we could move all of this to a new io.hfile.slab package Will do next revision. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 44 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line44 > > > should this implement BlockCache? Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 47 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line47 > > > since this is only modified during initialization, we can use an unsynchronized TreeMap which is probably more efficient Fixed. Although, in the future, we may want to modify it on the fly, which is why it was a ConcurrentSkipListMap. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 49 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line49 > > > rename to statThreadPeriodSecs to indicate time unit. > > Though, since it's a constant, it probably should be STAT_THREAD_PERIOD. This should be configurable but we can address in a follow-up fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 53 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line53 > > > you can set the thread factory to make daemon threads, then you don't need setDaemon(true) below changed to runnable. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 65-68 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line65 > > > line breaks unnecessary eclipse autoformatting . fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 80-85 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line80 > > > this stuff will have to be configurable too. but this is fine for now next revision! On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 95 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line95 > > > should also note that it returns 0 for the case of an uncacheable object changed so it returns the entry instead of an int. returns null if unreachable. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 98 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line98 > > > this function is never used – you should make it private, and then call it from cacheBlock below. fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 145-148 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line145 > > > again this is no good. check for null instead of catching an exception fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 174 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line174 > > > when would this be true? evicting something that doesn't have an entry in backingStore? It'll be true because theres a callback from SlabCache. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 177 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line177 > > > it seems like there's a loop in flow control here. Nope, SingleSlabCache should return false, and not loop. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 214 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line214 > > > move this nice logging output to something like: slabstats.logUsage() Done. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 229 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line229 > > > since you're using scheduleAtFixedRate, this should implement Runnable, not extend Thread Done On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 250 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line250 > > > need to be public? Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 253 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line253 > > > should be ALL_CAPS Done On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 254 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line254 > > > ALSO_ALL_CAPS Done. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 268 > < https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line268 > > > why boxed longs? Changed to AtomicLong[] On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 369 > < https://reviews.apache.org/r/1191/diff/1/?file=26920#file26920line369 > > > for consistency, I think we should name these something like: > hfile.block.cache.offheap.size > hfile.block.cache.offheap.minblocksize > > also, is min block size really a min? isn't it more like expectedblocksize? Will do next revision. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 84 > < https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line84 > > > nowhere in this test does it actually verify that the data is valid Fixed. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 38 > < https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line38 > > > should use JUnit 4 style tests in new code - i.e not inherit TestCase. I thought since everything else was JUnit 3 style, I should leave it as JUnit 3 style. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java, line 109 > < https://reviews.apache.org/r/1191/diff/1/?file=26922#file26922line109 > > > see above. > > If you can refactor some of this code into something like BlockCacheTestUtils.testBasicCacheContract(BlockCache cache) that would be good... copy paste evil Will fix this when I redo all tests tomorrow. On 2011-07-25 23:48:42, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 95 > < https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line95 > > > some javadoc would be nice... it seems this code is copy-pasted from TestLruBlockCache, but really doesn't make sense when applied to the SingleSlabCache Will fix this when I redo all tests tomorrow. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1191/#review1182 ----------------------------------------------------------- On 2011-07-25 22:55:56, Todd Lipcon wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1191/ ----------------------------------------------------------- (Updated 2011-07-25 22:55:56) Review request for hbase and Todd Lipcon. Summary ------- Uploading slabcachepatchv4 to review for Li Pi. This addresses bug HBASE-4027 . https://issues.apache.org/jira/browse/HBASE-4027 Diffs ----- conf/hbase-env.sh 2d55d27 pom.xml 729dc37 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552 src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1191/diff Testing ------- Thanks, Todd
        Hide
        Li Pi added a comment -

        Fixed as per Todd's reviewboard comments.

        Show
        Li Pi added a comment - Fixed as per Todd's reviewboard comments.
        Hide
        Li Pi added a comment -

        addressed all of Todd's comments. Added multi threaded tests, wrote test for Slab.java.

        To Stack/Ted Yu: now is a good time to review.

        Show
        Li Pi added a comment - addressed all of Todd's comments. Added multi threaded tests, wrote test for Slab.java. To Stack/Ted Yu: now is a good time to review.
        Hide
        Li Pi added a comment -

        PDF with benchmark results and some design notes.

        Show
        Li Pi added a comment - PDF with benchmark results and some design notes.
        Hide
        Todd Lipcon added a comment -

        Nice work and nice write-up. One interesting thing I noticed in the benchmarks is that the scalability of the FS cache performance is poor - 10 threads looks like it's only twice as fast as 1 thread on the FS implementations, whereas it scales much better on the various in-process caches. How large is the dataset in this test? Is it small enough that all of the threads are contending to read the same HFile block, and thus hitting the locks in HFile reader?

        Show
        Todd Lipcon added a comment - Nice work and nice write-up. One interesting thing I noticed in the benchmarks is that the scalability of the FS cache performance is poor - 10 threads looks like it's only twice as fast as 1 thread on the FS implementations, whereas it scales much better on the various in-process caches. How large is the dataset in this test? Is it small enough that all of the threads are contending to read the same HFile block, and thus hitting the locks in HFile reader?
        Hide
        Li Pi added a comment -

        Dataset is small - I wanted to guarantee it remained in FS cache. I'll make it larger.

        Show
        Li Pi added a comment - Dataset is small - I wanted to guarantee it remained in FS cache. I'll make it larger.
        Hide
        Jonathan Gray added a comment -

        Has the diff on reviewboard been updated? Trying to review but looks like it's still the old one?

        Show
        Jonathan Gray added a comment - Has the diff on reviewboard been updated? Trying to review but looks like it's still the old one?
        Hide
        Li Pi added a comment -

        gimme a sec. will do it.

        Show
        Li Pi added a comment - gimme a sec. will do it.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-07-28 18:29:54.907234)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Added reviewers. Feel free to review anyways.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 18:29:54.907234) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Added reviewers. Feel free to review anyways. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        Review request for hbase, Todd Lipcon and Li Pi.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- Review request for hbase, Todd Lipcon and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-07-28 18:29:02.934543)

        Review request for hbase and Li Pi.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 18:29:02.934543) Review request for hbase and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-07-28 21:39:13.311916)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing (updated)
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 21:39:13.311916) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing (updated) ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        small bugfix.

        Show
        Li Pi added a comment - small bugfix.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-07-28 23:02:50.734090)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        small bugfix, off by one error in slab.java. declaring things to maximum size would fail.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 23:02:50.734090) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- small bugfix, off by one error in slab.java. declaring things to maximum size would fail. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1220
        -----------------------------------------------------------

        conf/hbase-env.sh
        <https://reviews.apache.org/r/1214/#comment2739>

        Extra whitespace is obvious on review board.

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment2738>

        Year of copyright.

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment2740>

        Long line should wrap.

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment2741>

        Indentation should be corrected for the if block.

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment2743>

        Copyright missing.

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment2744>

        This implies that an eviction may involve fewer than one block.
        If so, this count can be named evictedBlockCount.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2745>

        Should we add hitCount for offHeapCache ?
        This can be addressed in a follow-on JIRA.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2746>

        In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2747>

        Is this a typo ?

        • Ted

        On 2011-07-28 23:02:50, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-28 23:02:50)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1220 ----------------------------------------------------------- conf/hbase-env.sh < https://reviews.apache.org/r/1214/#comment2739 > Extra whitespace is obvious on review board. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment2738 > Year of copyright. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment2740 > Long line should wrap. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment2741 > Indentation should be corrected for the if block. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment2743 > Copyright missing. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment2744 > This implies that an eviction may involve fewer than one block. If so, this count can be named evictedBlockCount. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2745 > Should we add hitCount for offHeapCache ? This can be addressed in a follow-on JIRA. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2746 > In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2747 > Is this a typo ? Ted On 2011-07-28 23:02:50, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 23:02:50) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1221
        -----------------------------------------------------------

        I see you fixed a bug, but you didn't add a new test case! Bug fixes should include regression tests. Also, the new diff changed some formatting - looks like some >80char lines now. Will do a full review tomorrow.

        • Todd

        On 2011-07-28 23:02:50, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-28 23:02:50)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1221 ----------------------------------------------------------- I see you fixed a bug, but you didn't add a new test case! Bug fixes should include regression tests. Also, the new diff changed some formatting - looks like some >80char lines now. Will do a full review tomorrow. Todd On 2011-07-28 23:02:50, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 23:02:50) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        The bug actually broke an existing test case. I'm not sure why I didn't notice - I did after trying to rebuild it this morning.

        Show
        Li Pi added a comment - The bug actually broke an existing test case. I'm not sure why I didn't notice - I did after trying to rebuild it this morning.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1211
        -----------------------------------------------------------

        conf/hbase-env.sh
        <https://reviews.apache.org/r/1214/#comment2675>

        Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is)

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment2676>

        2011

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment2677>

        whitespace

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment2678>

        license

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment2679>

        class comment

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2680>

        whitespace here and throughout this file

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2681>

        Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc).

        If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks)

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2682>

        longer than 80 chars

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2683>

        This seems like a behavior that we may not always want.

        If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?)

        we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap)

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment2684>

        this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment2685>

        huh?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment2686>

        line > 80 chars

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment2687>

        getTotalNumBlocks() and getRemainingNumBlocks() or something? i find the method names a little unclear (or just add some javadoc)

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment2688>

        javadoc on these

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2689>

        I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2690>

        Why these ratios? At the least, this should all be configurable (even if just in code and undocumented).

        Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive.

        I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2691>

        seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2692>

        is this really a cache miss? this cache will never take this block.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2693>

        but here we need to check for null and increment cache miss rather than hit if it is null

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2694>

        this non-javadoc comment is not necessary, just the @Override has been normal format in hbase

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2762>

        should this really be a bunch of INFO logs? Could we have these exported as metrics instead?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment2763>

        license

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        <https://reviews.apache.org/r/1214/#comment2764>

        formatting looks a little strange here (missing spaces and such)

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        <https://reviews.apache.org/r/1214/#comment2765>

        line > 80 chars

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1214/#comment2766>

        2011

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1214/#comment2767>

        This test looks more specific than testing "the concurrent LruBlockCache"

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
        <https://reviews.apache.org/r/1214/#comment2768>

        you are missing a space here and throughout this file and some others... our format is to always have a space before a {

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
        <https://reviews.apache.org/r/1214/#comment2769>

        license

        • Jonathan

        On 2011-07-28 23:02:50, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-28 23:02:50)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1211 ----------------------------------------------------------- conf/hbase-env.sh < https://reviews.apache.org/r/1214/#comment2675 > Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is) src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment2676 > 2011 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment2677 > whitespace src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment2678 > license src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment2679 > class comment src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2680 > whitespace here and throughout this file src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2681 > Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc). If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks) src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2682 > longer than 80 chars src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2683 > This seems like a behavior that we may not always want. If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?) we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap) src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment2684 > this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment2685 > huh? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment2686 > line > 80 chars src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment2687 > getTotalNumBlocks() and getRemainingNumBlocks() or something? i find the method names a little unclear (or just add some javadoc) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment2688 > javadoc on these src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2689 > I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2690 > Why these ratios? At the least, this should all be configurable (even if just in code and undocumented). Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive. I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2691 > seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2692 > is this really a cache miss? this cache will never take this block. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2693 > but here we need to check for null and increment cache miss rather than hit if it is null src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2694 > this non-javadoc comment is not necessary, just the @Override has been normal format in hbase src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2762 > should this really be a bunch of INFO logs? Could we have these exported as metrics instead? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment2763 > license src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java < https://reviews.apache.org/r/1214/#comment2764 > formatting looks a little strange here (missing spaces and such) src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java < https://reviews.apache.org/r/1214/#comment2765 > line > 80 chars src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java < https://reviews.apache.org/r/1214/#comment2766 > 2011 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java < https://reviews.apache.org/r/1214/#comment2767 > This test looks more specific than testing "the concurrent LruBlockCache" src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java < https://reviews.apache.org/r/1214/#comment2768 > you are missing a space here and throughout this file and some others... our format is to always have a space before a { src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java < https://reviews.apache.org/r/1214/#comment2769 > license Jonathan On 2011-07-28 23:02:50, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 23:02:50) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > conf/hbase-env.sh, line 42

        > <https://reviews.apache.org/r/1214/diff/2/?file=28356#file28356line42>

        >

        > Extra whitespace is obvious on review board.

        Fixed.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2

        > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line2>

        >

        > Year of copyright.

        Fixed.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 48

        > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line48>

        >

        > Long line should wrap.

        Fixed.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 64

        > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line64>

        >

        > Indentation should be corrected for the if block.

        Fixed

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 2

        > <https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line2>

        >

        > Copyright missing.

        Fixed

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 25

        > <https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line25>

        >

        > This implies that an eviction may involve fewer than one block.

        > If so, this count can be named evictedBlockCount.

        Fixed.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100

        > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100>

        >

        > Should we add hitCount for offHeapCache ?

        > This can be addressed in a follow-on JIRA.

        I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115

        > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line115>

        >

        > In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts.

        Agreed. I may be redoing this substantially in a follow on Jira.

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 145

        > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line145>

        >

        > Is this a typo ?

        Yup, and fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1220
        -----------------------------------------------------------

        On 2011-07-28 23:02:50, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-28 23:02:50)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-29 05:14:48, Ted Yu wrote: > conf/hbase-env.sh, line 42 > < https://reviews.apache.org/r/1214/diff/2/?file=28356#file28356line42 > > > Extra whitespace is obvious on review board. Fixed. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2 > < https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line2 > > > Year of copyright. Fixed. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 48 > < https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line48 > > > Long line should wrap. Fixed. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 64 > < https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line64 > > > Indentation should be corrected for the if block. Fixed On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 2 > < https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line2 > > > Copyright missing. Fixed On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 25 > < https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line25 > > > This implies that an eviction may involve fewer than one block. > If so, this count can be named evictedBlockCount. Fixed. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100 > < https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100 > > > Should we add hitCount for offHeapCache ? > This can be addressed in a follow-on JIRA. I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115 > < https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line115 > > > In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts. Agreed. I may be redoing this substantially in a follow on Jira. On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 145 > < https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line145 > > > Is this a typo ? Yup, and fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1220 ----------------------------------------------------------- On 2011-07-28 23:02:50, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-28 23:02:50) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-07-30 00:39:48.920025)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Addressed TedYu/JGray's comments.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-30 00:39:48.920025) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Addressed TedYu/JGray's comments. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        New updated benchmark results. Tested with 1gb dataset, as per todd's suggestion. the FS cache does better, but still can't match the off heap cache.

        Show
        Li Pi added a comment - New updated benchmark results. Tested with 1gb dataset, as per todd's suggestion. the FS cache does better, but still can't match the off heap cache.
        Hide
        Li Pi added a comment -

        addressed some of Ted Yu/JGrays comments on reviewboard.

        Show
        Li Pi added a comment - addressed some of Ted Yu/JGrays comments on reviewboard.
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > conf/hbase-env.sh, lines 44-45

        > <https://reviews.apache.org/r/1214/diff/1/?file=28125#file28125line44>

        >

        > Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is)

        I haven't figured out a way. Just asked StackOverflow - hopefully they'll have an answer.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2

        > <https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line2>

        >

        > 2011

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 1

        > <https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line1>

        >

        > license

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 41

        > <https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line41>

        >

        > whitespace

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 5

        > <https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line5>

        >

        > class comment

        Added

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 39

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line39>

        >

        > Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc).

        >

        > If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks)

        It can be made more generic easily. I just haven't done it. Will do though - just create a constructor that takes two BlockCaches?

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 66

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line66>

        >

        > longer than 80 chars

        fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-79

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line77>

        >

        > This seems like a behavior that we may not always want.

        >

        > If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?)

        >

        > we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap)

        I was thinking of sending all initial caches to the off heap cache, giving it an inbuilt scan resistance, but LRUBlockCache already does partitioning to deal with scans.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line115>

        >

        > this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both.

        The idea was the combined evicted stats of both, but yeah, CacheStats are implemented for both LruBlockCache and SlabCache, so exposing both might be a good idea.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 74

        > <https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line74>

        >

        > huh?

        Just a typo. Fixed. Don't know why that line is there.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 84

        > <https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line84>

        >

        > line > 80 chars

        fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 122-125

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line122>

        >

        > seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed

        If the size of the item being cached is too big to fit into any slab, we can't cache it. Again, I'm moving away from MemCached's model, so this should work better.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 151

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line151>

        >

        > is this really a cache miss? this cache will never take this block.

        Good point. Taking that part out.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 155

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line155>

        >

        > but here we need to check for null and increment cache miss rather than hit if it is null

        Yup, doing so as well.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 170-173

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line170>

        >

        > this non-javadoc comment is not necessary, just the @Override has been normal format in hbase

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 269

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line269>

        >

        > should this really be a bunch of INFO logs? Could we have these exported as metrics instead?

        Will take care of this next pass.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 1

        > <https://reviews.apache.org/r/1214/diff/1/?file=28135#file28135line1>

        >

        > license

        added.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 374-375

        > <https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line374>

        >

        > formatting looks a little strange here (missing spaces and such)

        fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 28

        > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line28>

        >

        > This test looks more specific than testing "the concurrent LruBlockCache"

        Woah, typo, fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 34

        > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line34>

        >

        > you are missing a space here and throughout this file and some others... our format is to always have a space before a {

        fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java, line 1

        > <https://reviews.apache.org/r/1214/diff/1/?file=28139#file28139line1>

        >

        > license

        added.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 2

        > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line2>

        >

        > 2011

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 376

        > <https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line376>

        >

        > line > 80 chars

        Fixed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 63-65

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line63>

        >

        > Why these ratios? At the least, this should all be configurable (even if just in code and undocumented).

        >

        > Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive.

        >

        > I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated.

        Its sort of expensive to change cache sizes with DirectByteBuffer after instantiation. I have a plan to restructure this away from MemCached's model though.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 37

        > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line37>

        >

        > whitespace here and throughout this file

        Removed.

        On 2011-07-29 17:41:50, Jonathan Gray wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 48

        > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line48>

        >

        > I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab?

        I'm using MemCached's allocation model - although I have one written that does it in a wholly different manner. But MemCached gets around fragmentation by allocating a certain range of sizes to a Single Slab.

        If something is smaller than the max size of a particular slab, the extra space is just wasted. Each SingleSizeCache can cache things up to a certain size. Things above that size cannot be cached, and things below that size will waste space.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1211
        -----------------------------------------------------------

        On 2011-07-30 00:39:48, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-30 00:39:48)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-29 17:41:50, Jonathan Gray wrote: > conf/hbase-env.sh, lines 44-45 > < https://reviews.apache.org/r/1214/diff/1/?file=28125#file28125line44 > > > Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is) I haven't figured out a way. Just asked StackOverflow - hopefully they'll have an answer. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2 > < https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line2 > > > 2011 Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 1 > < https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line1 > > > license Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 41 > < https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line41 > > > whitespace Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 5 > < https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line5 > > > class comment Added On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 39 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line39 > > > Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc). > > If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks) It can be made more generic easily. I just haven't done it. Will do though - just create a constructor that takes two BlockCaches? On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 66 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line66 > > > longer than 80 chars fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-79 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line77 > > > This seems like a behavior that we may not always want. > > If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?) > > we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap) I was thinking of sending all initial caches to the off heap cache, giving it an inbuilt scan resistance, but LRUBlockCache already does partitioning to deal with scans. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line115 > > > this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both. The idea was the combined evicted stats of both, but yeah, CacheStats are implemented for both LruBlockCache and SlabCache, so exposing both might be a good idea. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 74 > < https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line74 > > > huh? Just a typo. Fixed. Don't know why that line is there. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 84 > < https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line84 > > > line > 80 chars fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 122-125 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line122 > > > seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed If the size of the item being cached is too big to fit into any slab, we can't cache it. Again, I'm moving away from MemCached's model, so this should work better. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 151 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line151 > > > is this really a cache miss? this cache will never take this block. Good point. Taking that part out. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 155 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line155 > > > but here we need to check for null and increment cache miss rather than hit if it is null Yup, doing so as well. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 170-173 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line170 > > > this non-javadoc comment is not necessary, just the @Override has been normal format in hbase Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 269 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line269 > > > should this really be a bunch of INFO logs? Could we have these exported as metrics instead? Will take care of this next pass. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 1 > < https://reviews.apache.org/r/1214/diff/1/?file=28135#file28135line1 > > > license added. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 374-375 > < https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line374 > > > formatting looks a little strange here (missing spaces and such) fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 28 > < https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line28 > > > This test looks more specific than testing "the concurrent LruBlockCache" Woah, typo, fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 34 > < https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line34 > > > you are missing a space here and throughout this file and some others... our format is to always have a space before a { fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java, line 1 > < https://reviews.apache.org/r/1214/diff/1/?file=28139#file28139line1 > > > license added. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 2 > < https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line2 > > > 2011 Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 376 > < https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line376 > > > line > 80 chars Fixed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 63-65 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line63 > > > Why these ratios? At the least, this should all be configurable (even if just in code and undocumented). > > Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive. > > I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated. Its sort of expensive to change cache sizes with DirectByteBuffer after instantiation. I have a plan to restructure this away from MemCached's model though. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 37 > < https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line37 > > > whitespace here and throughout this file Removed. On 2011-07-29 17:41:50, Jonathan Gray wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 48 > < https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line48 > > > I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab? I'm using MemCached's allocation model - although I have one written that does it in a wholly different manner. But MemCached gets around fragmentation by allocating a certain range of sizes to a Single Slab. If something is smaller than the max size of a particular slab, the extra space is just wasted. Each SingleSizeCache can cache things up to a certain size. Things above that size cannot be cached, and things below that size will waste space. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1211 ----------------------------------------------------------- On 2011-07-30 00:39:48, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-30 00:39:48) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-29 05:14:48, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100

        > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100>

        >

        > Should we add hitCount for offHeapCache ?

        > This can be addressed in a follow-on JIRA.

        Li Pi wrote:

        I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics.

        What I meant to say, is, that it does have a hitcount. It just isn't displayed anywhere.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1220
        -----------------------------------------------------------

        On 2011-07-30 00:39:48, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-30 00:39:48)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-29 05:14:48, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100 > < https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100 > > > Should we add hitCount for offHeapCache ? > This can be addressed in a follow-on JIRA. Li Pi wrote: I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics. What I meant to say, is, that it does have a hitcount. It just isn't displayed anywhere. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1220 ----------------------------------------------------------- On 2011-07-30 00:39:48, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-30 00:39:48) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1238
        -----------------------------------------------------------

        conf/hbase-env.sh
        <https://reviews.apache.org/r/1214/#comment2812>

        Still some whitespaces.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment2813>

        Doesn't match actual name of class.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment2814>

        I don't see SkipList being used in this class.

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
        <https://reviews.apache.org/r/1214/#comment2815>

        This doesn't match the actual test.

        • Ted

        On 2011-07-30 00:39:48, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-07-30 00:39:48)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1238 ----------------------------------------------------------- conf/hbase-env.sh < https://reviews.apache.org/r/1214/#comment2812 > Still some whitespaces. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment2813 > Doesn't match actual name of class. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment2814 > I don't see SkipList being used in this class. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java < https://reviews.apache.org/r/1214/#comment2815 > This doesn't match the actual test. Ted On 2011-07-30 00:39:48, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-07-30 00:39:48) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        Fixed build error, addressed most of Ted Yu/JGray's review.

        Show
        Li Pi added a comment - Fixed build error, addressed most of Ted Yu/JGray's review.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-01 23:54:08.232314)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Addressed most of Ted Yu/JGray's comments. Will do another patch later. Fixed build errors. Tested patch/build on fresh pull from trunk.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-01 23:54:08.232314) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Addressed most of Ted Yu/JGray's comments. Will do another patch later. Fixed build errors. Tested patch/build on fresh pull from trunk. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        configuration changed to percentage of MaxDirectMemorySize.

        Show
        Li Pi added a comment - configuration changed to percentage of MaxDirectMemorySize.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-02 00:46:57.567666)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        configuration changed to percentage of MaxDirectMemorySize rather than specifying size of offHeapCache.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-02 00:46:57.567666) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- configuration changed to percentage of MaxDirectMemorySize rather than specifying size of offHeapCache. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-09 07:52:09.666709)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Rebased onto 3857. Configuration added as well.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        CHANGES.txt 7c6f592
        conf/hbase-env.sh 2d55d27
        pom.xml 0f24681
        src/docbkx/book.xml 2c19cef
        src/docbkx/configuration.xml 3595e76
        src/docbkx/developer.xml a3e22ea
        src/docbkx/performance.xml d8e104f
        src/docbkx/troubleshooting.xml 9f93cd9
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/ipc/Delayable.java 56f0650
        src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f223e46
        src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 6fb1da7
        src/main/java/org/apache/hadoop/hbase/mapred/Driver.java d38956c
        src/main/java/org/apache/hadoop/hbase/mapreduce/Driver.java dda4241
        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff05df8
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
        src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 8e87c83
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 887f736
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java 9d87fa5
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 151c90b
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 8e1f6e0
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937
        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java edca7f4
        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 0729410
        src/main/java/org/apache/hadoop/hbase/util/PoolMap.java ebf3078
        src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java a0bef34
        src/main/ruby/hbase/admin.rb 4460d6e
        src/site/xdoc/index.xml 8e1b531
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java 0b21c6c
        src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6d83b00
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java b4c407b
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java dc43eb2
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 381ac90
        src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java 2c565d7

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 07:52:09.666709) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Rebased onto 3857. Configuration added as well. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) CHANGES.txt 7c6f592 conf/hbase-env.sh 2d55d27 pom.xml 0f24681 src/docbkx/book.xml 2c19cef src/docbkx/configuration.xml 3595e76 src/docbkx/developer.xml a3e22ea src/docbkx/performance.xml d8e104f src/docbkx/troubleshooting.xml 9f93cd9 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/Delayable.java 56f0650 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f223e46 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 6fb1da7 src/main/java/org/apache/hadoop/hbase/mapred/Driver.java d38956c src/main/java/org/apache/hadoop/hbase/mapreduce/Driver.java dda4241 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff05df8 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 8e87c83 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 887f736 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java 9d87fa5 src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 151c90b src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 8e1f6e0 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java edca7f4 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 0729410 src/main/java/org/apache/hadoop/hbase/util/PoolMap.java ebf3078 src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java a0bef34 src/main/ruby/hbase/admin.rb 4460d6e src/site/xdoc/index.xml 8e1b531 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java 0b21c6c src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6d83b00 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java b4c407b src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java dc43eb2 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 381ac90 src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java 2c565d7 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-09 08:36:02.175711)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Fixed diff. Rebase on 3857 works!

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 08:36:02.175711) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Fixed diff. Rebase on 3857 works! Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        ignore v10 diff, that is just broken. this is the fixed one.

        Show
        Li Pi added a comment - ignore v10 diff, that is just broken. this is the fixed one.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55.719673)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Fixed compilation error. Corrected package location of cachetestutils.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        CHANGES.txt e9c0478
        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55.719673) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Fixed compilation error. Corrected package location of cachetestutils. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        Moved HFileBlockCacheTestUtils/SingleSizeCacheTestUtils to the tests folder, so it builds under mvn.

        Show
        Li Pi added a comment - Moved HFileBlockCacheTestUtils/SingleSizeCacheTestUtils to the tests folder, so it builds under mvn.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        CHANGES.txt
        <https://reviews.apache.org/r/1214/#comment3063>

        your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it.

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
        <https://reviews.apache.org/r/1214/#comment3064>

        style:
        /**

        • ...
          */
          public class CacheStates {

        (comment formatting and space before '{')

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3068>

        hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no?

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3069>

        does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3065>

        hyphenate 'on-heap' and 'off-heap' for clarity

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3066>

        missing space - " bytes ..."

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3067>

        same as above

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3070>

        we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3071>

        vertically collapse this - one line per param

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3072>

        when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:

        ByteBuffer storedBlock = ...allloc
        ... fill it in...
        ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);
        if (alreadyCached != null)

        { // we didn't insert the new one, so free it and throw an exception backingStore.free(storedBlock); throw new RuntimeException("already cached xxxxx"); }

        make sense?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3073>

        I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3074>

        this.size() is in units of bytes, not blocks

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3075>

        maybe rename to getOccupiedSize?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3076>

        wrong Log class - should use org.apache.commons.logging.Log

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3077>

        remove extra whitespace

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3078>

        hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3079>

        LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message – eg "Unable to deallocate direct memory during shutdown".

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3080>

        getBlock*s*Remaining, right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3081>

        incomplete comment here

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3083>

        this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3085>

        this isn't calling addSlabByConf here, so its javadoc seems out of date?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3087>

        spelling: proportions

        Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3088>

        "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3094>

        just to be thorough, check that they aren't negative?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3089>

        > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3091>

        should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1."

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3095>

        should we also check that they sum up to at least 0.99 or something?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3092>

        wrap line

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3093>

        you should .trim() the strings so whitespace is ignored

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3096>

        is that true? I thought it threw an RTE

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3097>

        collapse vertical space

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3098>

        this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock.

        In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3100>

        javadoc to indicate where this is called from.

        Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class:
        new SlabItemEvictionWatcher() {
        public void onEviction(String key)

        { internalBlockEvicted(key); }

        }

        that way it doesn't show up as a public API to anyone else

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3102>

        I think that the assignedCache's evict() counter is going to be double incremented here... call stack like:

        SingleSizeCache.ConcurrentMap.eviction
        -> SingleSizeCache's eviction watcher
        ---> calls stats.evict()
        ---> removes from backingMap
        ---> calls SlabCache.onEviction()
        -----> calls assignedCache.evict(key)
        -------> calls stats.evict() again
        -------> removes from map which is always a no-op

        right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3103>

        may as well just iterate over sizer.values()

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3104>

        again can iterate over values()

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3105>

        move this function up in the file so it's with other functions. Also needs @Override right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3106>

        also move this function up above the inner classes

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        <https://reviews.apache.org/r/1214/#comment3099>

        does this make a copy? it shouldn't need to at this point, right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment3108>

        this can be package-private right?

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment3107>

        you don't need abstract inside an interface

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
        <https://reviews.apache.org/r/1214/#comment3109>

        see note about consistency of conf params above

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3110>

        Apache license

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3111>

        this is unused?

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3112>

        remove empty javadocs

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java
        <https://reviews.apache.org/r/1214/#comment3113>

        patch accidentally reverting another recent commit

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3114>

        add license

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3115>

        long line

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3117>

        using a queue here ensures that two threads never request the same block in this test - that's one good test, but there should be another that just caches a small number of blocks and pounds on get only

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3116>

        I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below:
        long total = totalQueries.incrementAndGet();
        if (total % 10000 == 0)

        { LOG.debug("Ran " + total + " queries"); }

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3118>

        rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests – otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3120>

        this class looks like mostly dup code

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
        <https://reviews.apache.org/r/1214/#comment3119>

        hrm, this is identical to the other method?

        • Todd

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- CHANGES.txt < https://reviews.apache.org/r/1214/#comment3063 > your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java < https://reviews.apache.org/r/1214/#comment3064 > style: /** ... */ public class CacheStates { (comment formatting and space before '{') src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3068 > hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no? src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3069 > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here? src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3065 > hyphenate 'on-heap' and 'off-heap' for clarity src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3066 > missing space - " bytes ..." src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3067 > same as above src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3070 > we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3071 > vertically collapse this - one line per param src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3072 > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like: ByteBuffer storedBlock = ...allloc ... fill it in... ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock); if (alreadyCached != null) { // we didn't insert the new one, so free it and throw an exception backingStore.free(storedBlock); throw new RuntimeException("already cached xxxxx"); } make sense? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3073 > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3074 > this.size() is in units of bytes, not blocks src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3075 > maybe rename to getOccupiedSize? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3076 > wrong Log class - should use org.apache.commons.logging.Log src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3077 > remove extra whitespace src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3078 > hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3079 > LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message – eg "Unable to deallocate direct memory during shutdown". src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3080 > getBlock*s*Remaining, right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3081 > incomplete comment here src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3083 > this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3085 > this isn't calling addSlabByConf here, so its javadoc seems out of date? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3087 > spelling: proportions Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3088 > "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3094 > just to be thorough, check that they aren't negative? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3089 > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3091 > should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1." src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3095 > should we also check that they sum up to at least 0.99 or something? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3092 > wrap line src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3093 > you should .trim() the strings so whitespace is ignored src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3096 > is that true? I thought it threw an RTE src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3097 > collapse vertical space src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3098 > this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock. In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3100 > javadoc to indicate where this is called from. Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class: new SlabItemEvictionWatcher() { public void onEviction(String key) { internalBlockEvicted(key); } } that way it doesn't show up as a public API to anyone else src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3102 > I think that the assignedCache's evict() counter is going to be double incremented here... call stack like: SingleSizeCache.ConcurrentMap.eviction -> SingleSizeCache's eviction watcher ---> calls stats.evict() ---> removes from backingMap ---> calls SlabCache.onEviction() -----> calls assignedCache.evict(key) -------> calls stats.evict() again -------> removes from map which is always a no-op right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3103 > may as well just iterate over sizer.values() src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3104 > again can iterate over values() src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3105 > move this function up in the file so it's with other functions. Also needs @Override right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3106 > also move this function up above the inner classes src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java < https://reviews.apache.org/r/1214/#comment3099 > does this make a copy? it shouldn't need to at this point, right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment3108 > this can be package-private right? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment3107 > you don't need abstract inside an interface src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java < https://reviews.apache.org/r/1214/#comment3109 > see note about consistency of conf params above src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3110 > Apache license src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3111 > this is unused? src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3112 > remove empty javadocs src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java < https://reviews.apache.org/r/1214/#comment3113 > patch accidentally reverting another recent commit src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3114 > add license src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3115 > long line src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3117 > using a queue here ensures that two threads never request the same block in this test - that's one good test, but there should be another that just caches a small number of blocks and pounds on get only src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3116 > I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below: long total = totalQueries.incrementAndGet(); if (total % 10000 == 0) { LOG.debug("Ran " + total + " queries"); } src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3118 > rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests – otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3120 > this class looks like mostly dup code src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java < https://reviews.apache.org/r/1214/#comment3119 > hrm, this is identical to the other method? Todd On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1371
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3155>

        I think this was intended for use by hbase-env.sh

        • Ted

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1371 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3155 > I think this was intended for use by hbase-env.sh Ted On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-10 01:44:22, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18>

        >

        > I think this was intended for use by hbase-env.sh

        Yup, it's used by StoreFile.java now. We now specify the size of the off heap cache as a proportion of total direct memory size.

        long offHeapCacheSize = (long) (conf.getFloat("hbase.offheapcachepercentage", (float) 0.95) * DirectMemoryUtils.getDirectMemorySize());

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1371
        -----------------------------------------------------------

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-10 01:44:22, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18 > > > I think this was intended for use by hbase-env.sh Yup, it's used by StoreFile.java now. We now specify the size of the off heap cache as a proportion of total direct memory size. long offHeapCacheSize = (long) (conf.getFloat("hbase.offheapcachepercentage", (float) 0.95) * DirectMemoryUtils.getDirectMemorySize()); Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1371 ----------------------------------------------------------- On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1392
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
        <https://reviews.apache.org/r/1214/#comment3179>

        I believe this was a typo.
        You meant 1 (for bytes), right ?

        • Ted

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1392 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java < https://reviews.apache.org/r/1214/#comment3179 > I believe this was a typo. You meant 1 (for bytes), right ? Ted On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-10 21:28:51, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 21

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line21>

        >

        > I believe this was a typo.

        > You meant 1 (for bytes), right ?

        Good catch. Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1392
        -----------------------------------------------------------

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-10 21:28:51, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 21 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line21 > > > I believe this was a typo. > You meant 1 (for bytes), right ? Good catch. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1392 ----------------------------------------------------------- On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-07-30 02:57:26, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 41

        > <https://reviews.apache.org/r/1214/diff/3/?file=28543#file28543line41>

        >

        > I don't see SkipList being used in this class.

        Fixed. It got changed to an unsynchronized TreeMap, as we aren't supporting online cache resizing.

        On 2011-07-30 02:57:26, Ted Yu wrote:

        > conf/hbase-env.sh, line 40

        > <https://reviews.apache.org/r/1214/diff/3/?file=28534#file28534line40>

        >

        > Still some whitespaces.

        Fixed.

        On 2011-07-30 02:57:26, Ted Yu wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java, line 31

        > <https://reviews.apache.org/r/1214/diff/3/?file=28549#file28549line31>

        >

        > This doesn't match the actual test.

        Fixed.

        On 2011-07-30 02:57:26, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 31

        > <https://reviews.apache.org/r/1214/diff/3/?file=28542#file28542line31>

        >

        > Doesn't match actual name of class.

        Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1238
        -----------------------------------------------------------

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-07-30 02:57:26, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 41 > < https://reviews.apache.org/r/1214/diff/3/?file=28543#file28543line41 > > > I don't see SkipList being used in this class. Fixed. It got changed to an unsynchronized TreeMap, as we aren't supporting online cache resizing. On 2011-07-30 02:57:26, Ted Yu wrote: > conf/hbase-env.sh, line 40 > < https://reviews.apache.org/r/1214/diff/3/?file=28534#file28534line40 > > > Still some whitespaces. Fixed. On 2011-07-30 02:57:26, Ted Yu wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java, line 31 > < https://reviews.apache.org/r/1214/diff/3/?file=28549#file28549line31 > > > This doesn't match the actual test. Fixed. On 2011-07-30 02:57:26, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 31 > < https://reviews.apache.org/r/1214/diff/3/?file=28542#file28542line31 > > > Doesn't match actual name of class. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1238 ----------------------------------------------------------- On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > CHANGES.txt, line 197

        > <https://reviews.apache.org/r/1214/diff/8/?file=31762#file31762line197>

        >

        > your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it.

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, lines 24-26

        > <https://reviews.apache.org/r/1214/diff/8/?file=31766#file31766line24>

        >

        > style:

        > /**

        > * ...

        > */

        > public class CacheStates {

        >

        > (comment formatting and space before '{')

        Fixed

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 58

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line58>

        >

        > hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no?

        This is currently taken care of in StoreFile.java, but, good idea, I'll change that.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>

        >

        > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?

        Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 65

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line65>

        >

        > hyphenate 'on-heap' and 'off-heap' for clarity

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67>

        >

        > missing space - " bytes ..."

        The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different?

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 137

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line137>

        >

        > we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well.

        Will take care of next revision.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>

        >

        > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000

        changed it to returnBlock.put(contentBlock.duplicate()) instead.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 176

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line176>

        >

        > this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock.

        >

        > In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now?

        Definitely. Will do. At that point, might also change DoubleBlockCache to not cache things that aren't CacheableBlocks.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 190

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line190>

        >

        > maybe rename to getOccupiedSize?

        Done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 148

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line148>

        >

        > this.size() is in units of bytes, not blocks

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45

        > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>

        >

        > hrm, this is identical to the other method?

        Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60>

        >

        > vertically collapse this - one line per param

        80 char limit?. I don't know why eclipse keeps doing this. Will fix!

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>

        >

        > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:

        >

        > ByteBuffer storedBlock = ...allloc

        > ... fill it in...

        > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);

        > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > }

        >

        > make sense?

        Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."

        Good change, still though, I changed it to free alreadyCached instead.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 26

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line26>

        >

        > wrong Log class - should use org.apache.commons.logging.Log

        fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 35

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line35>

        >

        > remove extra whitespace

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 37-38

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line37>

        >

        > hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout?

        Okay. Will figure out names.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 90-91

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line90>

        >

        > LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message – eg "Unable to deallocate direct memory during shutdown".

        done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 104

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line104>

        >

        > getBlock*s*Remaining, right?

        Done

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 111

        > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line111>

        >

        > incomplete comment here

        I have no idea what that comment is supposed to be, removed it.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 88

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line88>

        >

        > this isn't calling addSlabByConf here, so its javadoc seems out of date?

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 93

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line93>

        >

        > spelling: proportions

        >

        > Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix

        Fixed. All prefixes changed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 43

        > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line43>

        >

        > this class looks like mostly dup code

        It is, but SingleSizeCache still caches bytebuffers rather than HFileBlocks.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 42

        > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line42>

        >

        > rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests – otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals

        equals was never implemented beforehand. I added it. I think default equals is just ==? I don't think anyone was using .equals beforehand.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 1

        > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line1>

        >

        > add license

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java, line 30

        > <https://reviews.apache.org/r/1214/diff/8/?file=31778#file31778line30>

        >

        > patch accidentally reverting another recent commit

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 38

        > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line38>

        >

        > I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below:

        > long total = totalQueries.incrementAndGet();

        > if (total % 10000 == 0) { bq. > LOG.debug("Ran " + total + " queries"); bq. > }

        I just removed the printout - I don't think its needed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 29

        > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line29>

        >

        > long line

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 100

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line100>

        >

        > "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is.

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 110

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line110>

        >

        > just to be thorough, check that they aren't negative?

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>

        >

        > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1

        http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)

        Returns:
        -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 116

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line116>

        >

        > should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1."

        Done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 117

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line117>

        >

        > should we also check that they sum up to at least 0.99 or something?

        added a LOG.warn if it doesn't add up to 0.99

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 121

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line121>

        >

        > wrap line

        Done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 143

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line143>

        >

        > you should .trim() the strings so whitespace is ignored

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 159

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line159>

        >

        > is that true? I thought it threw an RTE

        Yes, it does throw a RTE. Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 161-164

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line161>

        >

        > collapse vertical space

        Done.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 375-376

        > <https://reviews.apache.org/r/1214/diff/8/?file=31776#file31776line375>

        >

        > see note about consistency of conf params above

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 30

        > <https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line30>

        >

        > you don't need abstract inside an interface

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 23

        > <https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line23>

        >

        > this can be package-private right?

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 231

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line231>

        >

        > javadoc to indicate where this is called from.

        >

        > Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class:

        > new SlabItemEvictionWatcher() {

        > public void onEviction(String key) { bq. > internalBlockEvicted(key); bq. > }

        > }

        >

        >

        > that way it doesn't show up as a public API to anyone else

        this is now package private.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 60

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line60>

        >

        > this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 237

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line237>

        >

        > I think that the assignedCache's evict() counter is going to be double incremented here... call stack like:

        >

        > SingleSizeCache.ConcurrentMap.eviction

        > -> SingleSizeCache's eviction watcher

        > ---> calls stats.evict()

        > ---> removes from backingMap

        > ---> calls SlabCache.onEviction()

        > -----> calls assignedCache.evict(key)

        > -------> calls stats.evict() again

        > -------> removes from map which is always a no-op

        >

        > right?

        Whoops. Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 245

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line245>

        >

        > may as well just iterate over sizer.values()

        fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 315

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line315>

        >

        > again can iterate over values()

        fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 1

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line1>

        >

        > Apache license

        Fixed.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18>

        >

        > this is unused?

        Not anymore

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, lines 59-63

        > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line59>

        >

        > remove empty javadocs

        Done.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        On 2011-08-09 20:39:55, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-09 20:39:55)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt e9c0478

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-09 22:05:13, Todd Lipcon wrote: > CHANGES.txt, line 197 > < https://reviews.apache.org/r/1214/diff/8/?file=31762#file31762line197 > > > your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it. Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, lines 24-26 > < https://reviews.apache.org/r/1214/diff/8/?file=31766#file31766line24 > > > style: > /** > * ... > */ > public class CacheStates { > > (comment formatting and space before '{') Fixed On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 58 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line58 > > > hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no? This is currently taken care of in StoreFile.java, but, good idea, I'll change that. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64 > > > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here? Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 65 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line65 > > > hyphenate 'on-heap' and 'off-heap' for clarity Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67 > > > missing space - " bytes ..." The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different? On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 137 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line137 > > > we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well. Will take care of next revision. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118 > > > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000 changed it to returnBlock.put(contentBlock.duplicate()) instead. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 176 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line176 > > > this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock. > > In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now? Definitely. Will do. At that point, might also change DoubleBlockCache to not cache things that aren't CacheableBlocks. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 190 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line190 > > > maybe rename to getOccupiedSize? Done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 148 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line148 > > > this.size() is in units of bytes, not blocks Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45 > < https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45 > > > hrm, this is identical to the other method? Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60 > > > vertically collapse this - one line per param 80 char limit?. I don't know why eclipse keeps doing this. Will fix! On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101 > > > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like: > > ByteBuffer storedBlock = ...allloc > ... fill it in... > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock); > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > } > > make sense? Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value." Good change, still though, I changed it to free alreadyCached instead. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 26 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line26 > > > wrong Log class - should use org.apache.commons.logging.Log fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 35 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line35 > > > remove extra whitespace Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 37-38 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line37 > > > hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout? Okay. Will figure out names. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 90-91 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line90 > > > LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message – eg "Unable to deallocate direct memory during shutdown". done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 104 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line104 > > > getBlock*s*Remaining, right? Done On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 111 > < https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line111 > > > incomplete comment here I have no idea what that comment is supposed to be, removed it. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 88 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line88 > > > this isn't calling addSlabByConf here, so its javadoc seems out of date? Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 93 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line93 > > > spelling: proportions > > Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix Fixed. All prefixes changed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 43 > < https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line43 > > > this class looks like mostly dup code It is, but SingleSizeCache still caches bytebuffers rather than HFileBlocks. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 42 > < https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line42 > > > rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests – otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals equals was never implemented beforehand. I added it. I think default equals is just ==? I don't think anyone was using .equals beforehand. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 1 > < https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line1 > > > add license Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java, line 30 > < https://reviews.apache.org/r/1214/diff/8/?file=31778#file31778line30 > > > patch accidentally reverting another recent commit Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 38 > < https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line38 > > > I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below: > long total = totalQueries.incrementAndGet(); > if (total % 10000 == 0) { bq. > LOG.debug("Ran " + total + " queries"); bq. > } I just removed the printout - I don't think its needed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 29 > < https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line29 > > > long line Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 100 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line100 > > > "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is. Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 110 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line110 > > > just to be thorough, check that they aren't negative? Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114 > > > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1 http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal ) Returns: -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 116 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line116 > > > should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1." Done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 117 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line117 > > > should we also check that they sum up to at least 0.99 or something? added a LOG.warn if it doesn't add up to 0.99 On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 121 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line121 > > > wrap line Done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 143 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line143 > > > you should .trim() the strings so whitespace is ignored Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 159 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line159 > > > is that true? I thought it threw an RTE Yes, it does throw a RTE. Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 161-164 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line161 > > > collapse vertical space Done. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 375-376 > < https://reviews.apache.org/r/1214/diff/8/?file=31776#file31776line375 > > > see note about consistency of conf params above Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 30 > < https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line30 > > > you don't need abstract inside an interface Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 23 > < https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line23 > > > this can be package-private right? Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 231 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line231 > > > javadoc to indicate where this is called from. > > Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class: > new SlabItemEvictionWatcher() { > public void onEviction(String key) { bq. > internalBlockEvicted(key); bq. > } > } > > > that way it doesn't show up as a public API to anyone else this is now package private. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 60 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line60 > > > this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 237 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line237 > > > I think that the assignedCache's evict() counter is going to be double incremented here... call stack like: > > SingleSizeCache.ConcurrentMap.eviction > -> SingleSizeCache's eviction watcher > ---> calls stats.evict() > ---> removes from backingMap > ---> calls SlabCache.onEviction() > -----> calls assignedCache.evict(key) > -------> calls stats.evict() again > -------> removes from map which is always a no-op > > right? Whoops. Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 245 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line245 > > > may as well just iterate over sizer.values() fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 315 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line315 > > > again can iterate over values() fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 1 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line1 > > > Apache license Fixed. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18 > > > this is unused? Not anymore On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, lines 59-63 > < https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line59 > > > remove empty javadocs Done. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- On 2011-08-09 20:39:55, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-09 20:39:55) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt e9c0478 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        fixed things as per ted/todd's reviews

        Show
        Li Pi added a comment - fixed things as per ted/todd's reviews
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-11 07:52:55.019423)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Fixed as per ted/todd's reviews.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f22fb6e
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-11 07:52:55.019423) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Fixed as per ted/todd's reviews. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f22fb6e src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        fixed test failure, as reported by ted yu.

        Show
        Li Pi added a comment - fixed test failure, as reported by ted yu.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37.017483)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed test failure, as ted yu reported.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37.017483) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed test failure, as ted yu reported. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>

        >

        > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?

        Li Pi wrote:

        Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.

        is that true? wouldn't the 2G offheap always have a subset of what's in heap?

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67>

        >

        > missing space - " bytes ..."

        Li Pi wrote:

        The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different?

        oh, I see - you want it to say "1.9mbytes" instead of "1.9m bytes". That's fine

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60>

        >

        > vertically collapse this - one line per param

        Li Pi wrote:

        80 char limit?. I don't know why eclipse keeps doing this. Will fix!

        i think it'll still fit in 80 chars without all that vertical space

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>

        >

        > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:

        >

        > ByteBuffer storedBlock = ...allloc

        > ... fill it in...

        > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);

        > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > }

        >

        > make sense?

        Li Pi wrote:

        Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."

        Good change, still though, I changed it to free alreadyCached instead.

        ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right?

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>

        >

        > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000

        Li Pi wrote:

        changed it to returnBlock.put(contentBlock.duplicate()) instead.

        don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot?

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>

        >

        > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1

        Li Pi wrote:

        http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)

        Returns:

        -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.

        http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object)

        "Returns:
        a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object."

        so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45

        > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>

        >

        > hrm, this is identical to the other method?

        Li Pi wrote:

        Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.

        but you should be able to extract a method, even if the method has to take a type parameter. too much dup code

        • Todd

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        On 2011-08-12 08:41:37, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64 > > > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here? Li Pi wrote: Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap. is that true? wouldn't the 2G offheap always have a subset of what's in heap? On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67 > > > missing space - " bytes ..." Li Pi wrote: The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different? oh, I see - you want it to say "1.9mbytes" instead of "1.9m bytes". That's fine On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60 > > > vertically collapse this - one line per param Li Pi wrote: 80 char limit?. I don't know why eclipse keeps doing this. Will fix! i think it'll still fit in 80 chars without all that vertical space On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101 > > > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like: > > ByteBuffer storedBlock = ...allloc > ... fill it in... > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock); > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > } > > make sense? Li Pi wrote: Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value." Good change, still though, I changed it to free alreadyCached instead. ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right? On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118 > > > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000 Li Pi wrote: changed it to returnBlock.put(contentBlock.duplicate()) instead. don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot? On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114 > > > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1 Li Pi wrote: http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal ) Returns: -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val. http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object ) "Returns: a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object." so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1 On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45 > < https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45 > > > hrm, this is identical to the other method? Li Pi wrote: Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize. but you should be able to extract a method, even if the method has to take a type parameter. too much dup code Todd ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- On 2011-08-12 08:41:37, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114

        > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>

        >

        > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1

        Li Pi wrote:

        http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)

        Returns:

        -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.

        Todd Lipcon wrote:

        http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object)

        "Returns:

        a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object."

        so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1

        Okay. Will fix. Thanks!

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>

        >

        > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:

        >

        > ByteBuffer storedBlock = ...allloc

        > ... fill it in...

        > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);

        > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > }

        >

        > make sense?

        Li Pi wrote:

        Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."

        Good change, still though, I changed it to free alreadyCached instead.

        Todd Lipcon wrote:

        ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right?

        Yes, but that someone else stores a reference to the singlesizecache and a copy of the key. They'll look it up by key again, and will hit the new item.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64

        > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>

        >

        > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?

        Li Pi wrote:

        Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.

        Todd Lipcon wrote:

        is that true? wouldn't the 2G offheap always have a subset of what's in heap?

        ...ah yes, good point. i'll add this check in.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        On 2011-08-12 08:41:37, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114 > < https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114 > > > > 0, not == 1 – the contract of compareTo is just that it returns positive, not that it returns exactly 1 Li Pi wrote: http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal ) Returns: -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val. Todd Lipcon wrote: http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object ) "Returns: a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object." so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1 Okay. Will fix. Thanks! On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101 > > > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like: > > ByteBuffer storedBlock = ...allloc > ... fill it in... > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock); > if (alreadyCached != null) { bq. > // we didn't insert the new one, so free it and throw an exception bq. > backingStore.free(storedBlock); bq. > throw new RuntimeException("already cached xxxxx"); bq. > } > > make sense? Li Pi wrote: Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value." Good change, still though, I changed it to free alreadyCached instead. Todd Lipcon wrote: ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right? Yes, but that someone else stores a reference to the singlesizecache and a copy of the key. They'll look it up by key again, and will hit the new item. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64 > < https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64 > > > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here? Li Pi wrote: Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap. Todd Lipcon wrote: is that true? wouldn't the 2G offheap always have a subset of what's in heap? ...ah yes, good point. i'll add this check in. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- On 2011-08-12 08:41:37, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1423
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3285>

        this can race against getBlock() though:

        Thread A: backingMap.get(key) returns object
        Thread B: put() returns same object
        Thread B: free(object)
        Thread A: use object. boom?

        putIfAbsent shouldn't be any slower than put, may as well make use of it

        • Todd

        On 2011-08-12 08:41:37, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1423 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3285 > this can race against getBlock() though: Thread A: backingMap.get(key) returns object Thread B: put() returns same object Thread B: free(object) Thread A: use object. boom? putIfAbsent shouldn't be any slower than put, may as well make use of it Todd On 2011-08-12 08:41:37, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-12 17:26:40, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>

        >

        > this can race against getBlock() though:

        >

        > Thread A: backingMap.get(key) returns object

        > Thread B: put() returns same object

        > Thread B: free(object)

        > Thread A: use object. boom?

        >

        > putIfAbsent shouldn't be any slower than put, may as well make use of it

        Ah, gotcha! I see it now. Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1423
        -----------------------------------------------------------

        On 2011-08-12 08:41:37, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 08:41:37)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-12 17:26:40, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101 > > > this can race against getBlock() though: > > Thread A: backingMap.get(key) returns object > Thread B: put() returns same object > Thread B: free(object) > Thread A: use object. boom? > > putIfAbsent shouldn't be any slower than put, may as well make use of it Ah, gotcha! I see it now. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1423 ----------------------------------------------------------- On 2011-08-12 08:41:37, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 08:41:37) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-12 20:26:21.230751)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Fixed another broken test case. (Didn't reset buffer position before doing compare) and fixed race.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 20:26:21.230751) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Fixed another broken test case. (Didn't reset buffer position before doing compare) and fixed race. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        fixed broken test and race condition.

        Show
        Li Pi added a comment - fixed broken test and race condition.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1430
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3294>

        Still some white spaces to remove.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3295>

        Incorrect class name.

        • Ted

        On 2011-08-12 20:26:21, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 20:26:21)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1430 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3294 > Still some white spaces to remove. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3295 > Incorrect class name. Ted On 2011-08-12 20:26:21, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 20:26:21) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-12 21:52:43, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 38

        > <https://reviews.apache.org/r/1214/diff/11/?file=32400#file32400line38>

        >

        > Still some white spaces to remove.

        Got it.

        On 2011-08-12 21:52:43, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 37

        > <https://reviews.apache.org/r/1214/diff/11/?file=32401#file32401line37>

        >

        > Incorrect class name.

        Doh. Fixed.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1430
        -----------------------------------------------------------

        On 2011-08-12 20:26:21, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-12 20:26:21)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-12 21:52:43, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 38 > < https://reviews.apache.org/r/1214/diff/11/?file=32400#file32400line38 > > > Still some white spaces to remove. Got it. On 2011-08-12 21:52:43, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 37 > < https://reviews.apache.org/r/1214/diff/11/?file=32401#file32401line37 > > > Incorrect class name. Doh. Fixed. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1430 ----------------------------------------------------------- On 2011-08-12 20:26:21, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 20:26:21) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        fixed two typos.

        Show
        Li Pi added a comment - fixed two typos.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-12 22:30:10.310736)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed two bugs as per ted yu's reviews.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-12 22:30:10.310736) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed two bugs as per ted yu's reviews. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-17 04:16:06.047294)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        refactored, no more casting, cacheable added. multithreaded test that hammers single key added. general code cleanup stuff as well. heapsize not implemented completely yet.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-17 04:16:06.047294) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- refactored, no more casting, cacheable added. multithreaded test that hammers single key added. general code cleanup stuff as well. heapsize not implemented completely yet. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        a few more changes, substantially re-factored.

        Show
        Li Pi added a comment - a few more changes, substantially re-factored.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1489
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3402>

        Please add javadoc for these methods.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3403>

        This shows that Cacheable can reside in on-heap cache.
        The description for Cacheable should be refined.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3404>

        HeapSize is covered by Cacheable so is not needed here.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3405>

        Indentation is incorrect here.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3406>

        Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means.
        Please add javadoc.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3407>

        Please add javadoc for what this method does.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3408>

        Whitespace.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3409>

        Indentation.

        • Ted

        On 2011-08-17 04:16:06, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-17 04:16:06)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1489 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3402 > Please add javadoc for these methods. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3403 > This shows that Cacheable can reside in on-heap cache. The description for Cacheable should be refined. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3404 > HeapSize is covered by Cacheable so is not needed here. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3405 > Indentation is incorrect here. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3406 > Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means. Please add javadoc. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3407 > Please add javadoc for what this method does. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3408 > Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3409 > Indentation. Ted On 2011-08-17 04:16:06, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-17 04:16:06) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        Li Pi added a comment -

        added heapsize, fixed formatting, addressed ted yu's reviews.

        Show
        Li Pi added a comment - added heapsize, fixed formatting, addressed ted yu's reviews.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-18 08:31:59.585101)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        heapsize added. also addressed ted's review above. fixed formatting in hfileblock. (was inconsistent).

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-18 08:31:59.585101) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- heapsize added. also addressed ted's review above. fixed formatting in hfileblock. (was inconsistent). Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119

        > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>

        >

        > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000

        Li Pi wrote:

        changed it to returnBlock.put(contentBlock.duplicate()) instead.

        Todd Lipcon wrote:

        don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot?

        Since we duplicate it, we don't need to rewind after using it. The initial copy is already rewound, so we just duplicate it, do our put, and then discard it. TestHammerSingleKey has been written.

        On 2011-08-09 22:05:13, Todd Lipcon wrote:

        > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45

        > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>

        >

        > hrm, this is identical to the other method?

        Li Pi wrote:

        Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.

        Todd Lipcon wrote:

        but you should be able to extract a method, even if the method has to take a type parameter. too much dup code

        We have a Cacheable interface now. No more dup code.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1363
        -----------------------------------------------------------

        On 2011-08-18 08:31:59, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-18 08:31:59)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119 > < https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118 > > > I think there's a bug here if you have multiple users hammering the same contentBlock – two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000 Li Pi wrote: changed it to returnBlock.put(contentBlock.duplicate()) instead. Todd Lipcon wrote: don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot? Since we duplicate it, we don't need to rewind after using it. The initial copy is already rewound, so we just duplicate it, do our put, and then discard it. TestHammerSingleKey has been written. On 2011-08-09 22:05:13, Todd Lipcon wrote: > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45 > < https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45 > > > hrm, this is identical to the other method? Li Pi wrote: Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize. Todd Lipcon wrote: but you should be able to extract a method, even if the method has to take a type parameter. too much dup code We have a Cacheable interface now. No more dup code. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1363 ----------------------------------------------------------- On 2011-08-18 08:31:59, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-18 08:31:59) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 12

        > <https://reviews.apache.org/r/1214/diff/13/?file=32972#file32972line12>

        >

        > Please add javadoc for these methods.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 92

        > <https://reviews.apache.org/r/1214/diff/13/?file=32974#file32974line92>

        >

        > This shows that Cacheable can reside in on-heap cache.

        > The description for Cacheable should be refined.

        Done. Cacheable can reside in either.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 79

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line79>

        >

        > HeapSize is covered by Cacheable so is not needed here.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1528

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1528>

        >

        > Indentation is incorrect here.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1535

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1535>

        >

        > Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means.

        > Please add javadoc.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1540

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1540>

        >

        > Please add javadoc for what this method does.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1541

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1541>

        >

        > Whitespace.

        Done.

        On 2011-08-17 04:44:54, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1554

        > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1554>

        >

        > Indentation.

        Done.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1489
        -----------------------------------------------------------

        On 2011-08-18 08:31:59, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-18 08:31:59)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 12 > < https://reviews.apache.org/r/1214/diff/13/?file=32972#file32972line12 > > > Please add javadoc for these methods. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 92 > < https://reviews.apache.org/r/1214/diff/13/?file=32974#file32974line92 > > > This shows that Cacheable can reside in on-heap cache. > The description for Cacheable should be refined. Done. Cacheable can reside in either. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 79 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line79 > > > HeapSize is covered by Cacheable so is not needed here. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1528 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1528 > > > Indentation is incorrect here. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1535 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1535 > > > Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means. > Please add javadoc. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1540 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1540 > > > Please add javadoc for what this method does. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1541 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1541 > > > Whitespace. Done. On 2011-08-17 04:44:54, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1554 > < https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1554 > > > Indentation. Done. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1489 ----------------------------------------------------------- On 2011-08-18 08:31:59, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-18 08:31:59) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1525
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3476>

        Whitespace.

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        <https://reviews.apache.org/r/1214/#comment3477>

        Whitespace.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3478>

        Auto-formatter weirdness.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3480>

        this and all other formatting fixes should probably be in a seperate jira. filed.

        • Li

        On 2011-08-18 08:31:59, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-18 08:31:59)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1525 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3476 > Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java < https://reviews.apache.org/r/1214/#comment3477 > Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3478 > Auto-formatter weirdness. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java < https://reviews.apache.org/r/1214/#comment3480 > this and all other formatting fixes should probably be in a seperate jira. filed. Li On 2011-08-18 08:31:59, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-18 08:31:59) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-19 03:05:29.951253)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Cacheable interface now far less confusing. HFileBlock has reformatting reverted.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        CHANGES.txt 763ddbc
        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313
        src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 03:05:29.951253) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Cacheable interface now far less confusing. HFileBlock has reformatting reverted. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d CHANGES.txt 763ddbc conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313 src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39 src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1543
        -----------------------------------------------------------

        Todd: I'll get the fully serialized model out by tonight. This still leaves around 60 bytes of stuff on the heap.

        • Li

        On 2011-08-19 03:05:29, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 03:05:29)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        CHANGES.txt 763ddbc

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313

        src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1543 ----------------------------------------------------------- Todd: I'll get the fully serialized model out by tonight. This still leaves around 60 bytes of stuff on the heap. Li On 2011-08-19 03:05:29, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 03:05:29) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d CHANGES.txt 763ddbc conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313 src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39 src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1544
        -----------------------------------------------------------

        Cacheable interface is much more intuitive now.

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3516>

        Change an to the.

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3515>

        Do I see an incomplete sentence here ?

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3514>

        If self is always returned, why do we need the return value here ?

        • Ted

        On 2011-08-19 03:05:29, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 03:05:29)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        CHANGES.txt 763ddbc

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313

        src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1544 ----------------------------------------------------------- Cacheable interface is much more intuitive now. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3516 > Change an to the. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3515 > Do I see an incomplete sentence here ? src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java < https://reviews.apache.org/r/1214/#comment3514 > If self is always returned, why do we need the return value here ? Ted On 2011-08-19 03:05:29, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 03:05:29) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d CHANGES.txt 763ddbc conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313 src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39 src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-19 04:16:44.682053)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        fixed as per ted yu's comments above.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        CHANGES.txt 0478003
        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 04:16:44.682053) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- fixed as per ted yu's comments above. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) CHANGES.txt 0478003 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-19 04:10:41, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 49

        > <https://reviews.apache.org/r/1214/diff/15/?file=33671#file33671line49>

        >

        > If self is always returned, why do we need the return value here ?

        it returns a copy of itself, unless it doesn't need deserialization.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1544
        -----------------------------------------------------------

        On 2011-08-19 04:16:44, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 04:16:44)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt 0478003

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-19 04:10:41, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 49 > < https://reviews.apache.org/r/1214/diff/15/?file=33671#file33671line49 > > > If self is always returned, why do we need the return value here ? it returns a copy of itself, unless it doesn't need deserialization. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1544 ----------------------------------------------------------- On 2011-08-19 04:16:44, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 04:16:44) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt 0478003 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1546
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3522>

        Still some white spaces.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3518>

        This is an important class related to Cacheable. Should have added javadoc for class and methods.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        <https://reviews.apache.org/r/1214/#comment3519>

        REFERENCE refers to serializedData, right ?
        I think it would be clearer if the order of second and third parameters gets exchanged.

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
        <https://reviews.apache.org/r/1214/#comment3521>

        Can you explain the meaning of callAssignedCache ?
        It is not obvious by name.

        • Ted

        On 2011-08-19 04:16:44, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 04:16:44)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt 0478003

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1546 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3522 > Still some white spaces. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3518 > This is an important class related to Cacheable. Should have added javadoc for class and methods. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java < https://reviews.apache.org/r/1214/#comment3519 > REFERENCE refers to serializedData, right ? I think it would be clearer if the order of second and third parameters gets exchanged. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java < https://reviews.apache.org/r/1214/#comment3521 > Can you explain the meaning of callAssignedCache ? It is not obvious by name. Ted On 2011-08-19 04:16:44, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 04:16:44) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt 0478003 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1549
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
        <https://reviews.apache.org/r/1214/#comment3524>

        The following 10 lines of code is very similar to those starting line 65.
        Should be able to write another method so that both of them can call.

        • Ted

        On 2011-08-19 04:16:44, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 04:16:44)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        CHANGES.txt 0478003

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1549 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java < https://reviews.apache.org/r/1214/#comment3524 > The following 10 lines of code is very similar to those starting line 65. Should be able to write another method so that both of them can call. Ted On 2011-08-19 04:16:44, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 04:16:44) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- CHANGES.txt 0478003 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/
        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35.572967)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Changes
        -------

        Forgot to override .equals when redoing HFileBlock. Fixed, tests now work.

        Summary
        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.
        https://issues.apache.org/jira/browse/HBase-4027

        Diffs (updated)


        conf/hbase-env.sh 2d55d27
        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing
        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35.572967) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes ------- Forgot to override .equals when redoing HFileBlock. Fixed, tests now work. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-19 13:30:39, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 67

        > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line67>

        >

        > Still some white spaces.

        fixed.

        On 2011-08-19 13:30:39, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 244

        > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line244>

        >

        > This is an important class related to Cacheable. Should have added javadoc for class and methods.

        it's just a pair, added a comment though.

        On 2011-08-19 13:30:39, Ted Yu wrote:

        > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 255

        > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line255>

        >

        > REFERENCE refers to serializedData, right ?

        > I think it would be clearer if the order of second and third parameters gets exchanged.

        fixed. added comment.

        • Li

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1546
        -----------------------------------------------------------

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-19 13:30:39, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 67 > < https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line67 > > > Still some white spaces. fixed. On 2011-08-19 13:30:39, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 244 > < https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line244 > > > This is an important class related to Cacheable. Should have added javadoc for class and methods. it's just a pair, added a comment though. On 2011-08-19 13:30:39, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 255 > < https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line255 > > > REFERENCE refers to serializedData, right ? > I think it would be clearer if the order of second and third parameters gets exchanged. fixed. added comment. Li ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1546 ----------------------------------------------------------- On 2011-08-19 20:21:35, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1573
        -----------------------------------------------------------

        Ship it!

        All tests passed based on hbase-4027v12.7.diff

        • Ted

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50

        src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453

        src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93

        src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73

        src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece

        src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION

        src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0

        src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170

        Diff: https://reviews.apache.org/r/1214/diff

        Testing

        -------

        Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.

        Thanks,

        Li

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1573 ----------------------------------------------------------- Ship it! All tests passed based on hbase-4027v12.7.diff Ted On 2011-08-19 20:21:35, Li Pi wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ ----------------------------------------------------------- (Updated 2011-08-19 20:21:35) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Summary ------- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs ----- conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing ------- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1214/#review1575
        -----------------------------------------------------------

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
        <https://reviews.apache.org/r/1214/#comment3551>

        does this always return itself? I still think that once something has been serialized, we should throw away the original object, and when we re-instantiate it, it should create a new object.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3553>

        this object confuses me. once we've serialized something, all we should need is the ByteBuffer.

        src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
        <https://reviews.apache.org/r/1214/#comment3555>

        this isn't right - you need to take into account arrayOffset also. I think this .equals method should move to the tests, or a static method like blocksAndContentsEqual()

        • Todd

        On 2011-08-19 20:21:35, Li Pi wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1214/

        -----------------------------------------------------------

        (Updated 2011-08-19 20:21:35)

        Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.

        Summary

        -------

        Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.

        This addresses bug HBase-4027.

        https://issues.apache.org/jira/browse/HBase-4027

        Diffs

        -----

        conf/hbase-env.sh 2d55d27

        src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c

        src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION

        src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8

        src/main/j