Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11331

[blockcache] lazy block decompression

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 2.0.0, 0.98.7
    • Component/s: regionserver
    • Labels:
      None
    • Release Note:
      Hide
      When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the BlockCache in their on-disk format. This is different from the default behavior, which decompresses and decrypts a block before caching.

      For a region server hosting more data than fits into cache, enabling this feature with SNAPPY compression results in 50% increase in throughput and 30% improvement in mean latency while increasing GC by 80% and increasing overall CPU load by 2%.
      Show
      When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the BlockCache in their on-disk format. This is different from the default behavior, which decompresses and decrypts a block before caching. For a region server hosting more data than fits into cache, enabling this feature with SNAPPY compression results in 50% increase in throughput and 30% improvement in mean latency while increasing GC by 80% and increasing overall CPU load by 2%.

      Description

      Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk.

      This is related to but less invasive than HBASE-8894.

      1. HBASE-11331.00.patch
        44 kB
        Nick Dimiduk
      2. HBASE-11331.01.patch
        52 kB
        Nick Dimiduk
      3. HBASE-11331.02.patch
        78 kB
        Nick Dimiduk
      4. HBASE-11331.03.patch
        101 kB
        Nick Dimiduk
      5. HBASE-11331.04.patch
        101 kB
        Nick Dimiduk
      6. HBASE-11331.05.patch
        101 kB
        Nick Dimiduk
      7. HBASE-11331.05-0.98.patch
        100 kB
        Nick Dimiduk
      8. HBASE-11331.06.patch
        110 kB
        Nick Dimiduk
      9. HBASE-11331.06-0.98.patch
        110 kB
        Nick Dimiduk
      10. HBASE-11331.07.patch
        110 kB
        Nick Dimiduk
      11. HBASE-11331.07-0.98.patch
        110 kB
        Nick Dimiduk
      12. HBASE-11331LazyBlockDecompressperfcompare.pdf
        354 kB
        stack
      13. hbase-hbase-master-hor17n36.gq1.ygridcore.net.log
        610 kB
        Nick Dimiduk
      14. lazy-decompress.02.0.pdf
        430 kB
        Nick Dimiduk
      15. lazy-decompress.02.1.json
        27 kB
        Nick Dimiduk
      16. lazy-decompress.02.1.pdf
        512 kB
        Nick Dimiduk
      17. v03-20g-045g-false.pdf
        189 kB
        Nick Dimiduk
      18. v03-20g-045g-true.pdf
        183 kB
        Nick Dimiduk
      19. v03-20g-045g-true-16h.pdf
        241 kB
        Nick Dimiduk

        Issue Links

          Activity

          Hide
          ndimiduk Nick Dimiduk added a comment -

          Here's a patch with small tests passing. Will start playing with PerfEval and report back.

          Show
          ndimiduk Nick Dimiduk added a comment - Here's a patch with small tests passing. Will start playing with PerfEval and report back.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12649880/HBASE-11331.00.patch
          against trunk revision .
          ATTACHMENT ID: 12649880

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk
          org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite
          org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner
          org.apache.hadoop.hbase.io.hfile.TestHFileBlock

          -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:3499)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12649880/HBASE-11331.00.patch against trunk revision . ATTACHMENT ID: 12649880 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner org.apache.hadoop.hbase.io.hfile.TestHFileBlock -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:3499) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9747//console This message is automatically generated.
          Hide
          stack stack added a comment -

          Looks great to me Nick Dimiduk I am decompressing every time I read from a block rather than as we have now where we decompress once and that is what is in the blockcache? Will be interesting to see how this does compared. May show that this should be optional behavior (we give up CPU here... we need to start winning it back elsewhere).

          Show
          stack stack added a comment - Looks great to me Nick Dimiduk I am decompressing every time I read from a block rather than as we have now where we decompress once and that is what is in the blockcache? Will be interesting to see how this does compared. May show that this should be optional behavior (we give up CPU here... we need to start winning it back elsewhere).
          Hide
          anoop.hbase Anoop Sam John added a comment -

          Not decrypting gives (so much) size save?

          decompressing every time I read from a block rather than as we have now where we decompress once

          Ya a concerning factor and +1 for making it configurable.

          Show
          anoop.hbase Anoop Sam John added a comment - Not decrypting gives (so much) size save? decompressing every time I read from a block rather than as we have now where we decompress once Ya a concerning factor and +1 for making it configurable.
          Hide
          vrodionov Vladimir Rodionov added a comment -

          These two statements contradict each other:

          Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications.

          The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk.

          You either keep blocks compressed in a cache and decompress them on demand (1) or you decompress them lazily and keep them decompressed after that (2). What does "lazy decompression" means in this case? If you cache blocks on reads only (most of the time and default behavior) - there is no much sense in a lazy decompression, because your block will be accessed immediately after it got into the cache. Lazy decompression makes sense only if you cache blocks on writes, but in this case (2) contradicts (1) as I mentioned already.

          Show
          vrodionov Vladimir Rodionov added a comment - These two statements contradict each other: Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. You either keep blocks compressed in a cache and decompress them on demand (1) or you decompress them lazily and keep them decompressed after that (2). What does "lazy decompression" means in this case? If you cache blocks on reads only (most of the time and default behavior) - there is no much sense in a lazy decompression, because your block will be accessed immediately after it got into the cache. Lazy decompression makes sense only if you cache blocks on writes, but in this case (2) contradicts (1) as I mentioned already.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          In this implementation, the decompressed block does not replace the compressed block in the cache. Decompression cost is paid on block access, every time. I need to profile the scanner path to ensure a single request is not decompressing the same block multiple times. For hot blocks, I expect this to result in increased CPU load vs decompressing it only once. For a more evenly distributed access pattern, this should greatly reduce the amount of disk seeks because more data is cached. I believe the latter use-case is more common.

          Show
          ndimiduk Nick Dimiduk added a comment - In this implementation, the decompressed block does not replace the compressed block in the cache. Decompression cost is paid on block access, every time. I need to profile the scanner path to ensure a single request is not decompressing the same block multiple times. For hot blocks, I expect this to result in increased CPU load vs decompressing it only once. For a more evenly distributed access pattern, this should greatly reduce the amount of disk seeks because more data is cached. I believe the latter use-case is more common.
          Hide
          vrodionov Vladimir Rodionov added a comment -

          Nick Dimiduk, where do you keep decompressed blocks? In a fast on heap cache? You do not have to decompress block every time you access it - only once and all subsequent scanner.next will read from decompressed block. Sorry, I am not following you here.

          Show
          vrodionov Vladimir Rodionov added a comment - Nick Dimiduk , where do you keep decompressed blocks? In a fast on heap cache? You do not have to decompress block every time you access it - only once and all subsequent scanner.next will read from decompressed block. Sorry, I am not following you here.
          Hide
          vrodionov Vladimir Rodionov added a comment -

          In my own tests I have seen performance drop from ~ 100K to 75K ops with compression on. This is with a custom LZ4 codec and YMMV. I think 20-25% penalty is not that big to justify uncompressed mode of operation. In many cases, having more data in a cache is much more important than peak performance.

          Show
          vrodionov Vladimir Rodionov added a comment - In my own tests I have seen performance drop from ~ 100K to 75K ops with compression on. This is with a custom LZ4 codec and YMMV. I think 20-25% penalty is not that big to justify uncompressed mode of operation. In many cases, having more data in a cache is much more important than peak performance.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Decompressed blocks aren't stored anywhere in the patch posted. This patch, just as current master code, decompressed an HFileBlock into an on-heap ByteBuffer. There's no additional cache layer for decompressed blocks as it stands currently; they're decompressed, consumed, and thrown away. HFileReaderV2$AbstractScannerV2 keeps a reference to the current block, so a single GET shouldn't pay the cost of decompression multiple times, but I need to confirm this is true.

          Show
          ndimiduk Nick Dimiduk added a comment - Decompressed blocks aren't stored anywhere in the patch posted. This patch, just as current master code, decompressed an HFileBlock into an on-heap ByteBuffer. There's no additional cache layer for decompressed blocks as it stands currently; they're decompressed, consumed, and thrown away. HFileReaderV2$AbstractScannerV2 keeps a reference to the current block, so a single GET shouldn't pay the cost of decompression multiple times, but I need to confirm this is true.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          In many cases, having more data in a cache is much more important than peak performance.

          We are in complete agreement on this point.

          Show
          ndimiduk Nick Dimiduk added a comment - In many cases, having more data in a cache is much more important than peak performance. We are in complete agreement on this point.
          Hide
          stack stack added a comment -

          How feasible keeping count of how many times a block has been decompressed and if over a configurable threshold, instead shove the decompressed block back into the block cache in place of the compressed one? We already count if been accessed more than once? Could we leverage this fact?

          This is related to but less invasive than HBASE-8894.

          Would a better characterization be that this is a core piece of HBASE-8894 only done more in line w/ how hbase master branch works now (HBASE-8894 interjects a special-case handling of its L2 cache when reading blocks from HDFS... This makes do without special interjection).

          Show
          stack stack added a comment - How feasible keeping count of how many times a block has been decompressed and if over a configurable threshold, instead shove the decompressed block back into the block cache in place of the compressed one? We already count if been accessed more than once? Could we leverage this fact? This is related to but less invasive than HBASE-8894 . Would a better characterization be that this is a core piece of HBASE-8894 only done more in line w/ how hbase master branch works now ( HBASE-8894 interjects a special-case handling of its L2 cache when reading blocks from HDFS... This makes do without special interjection).
          Hide
          stack stack added a comment -

          How feasible keeping count of how many times a block has been decompressed and if over a configurable threshold, instead shove the decompressed block back into the block cache in place of the compressed one?

          Or, if hot, move the decompressed and decoded block up into L1?

          Show
          stack stack added a comment - How feasible keeping count of how many times a block has been decompressed and if over a configurable threshold, instead shove the decompressed block back into the block cache in place of the compressed one? Or, if hot, move the decompressed and decoded block up into L1?
          Hide
          stack stack added a comment -

          Report comparing compressed to uncompressed bucket cache. Finding is slightly less i/o but at the cost of lots more GC and CPU. Get rate goes way down too.

          Show
          stack stack added a comment - Report comparing compressed to uncompressed bucket cache. Finding is slightly less i/o but at the cost of lots more GC and CPU. Get rate goes way down too.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          How feasible keeping count of how many times a block has been decompressed and if over a configurable threshold, instead shove the decompressed block back into the block cache in place of the compressed one? We already count if been accessed more than once? Could we leverage this fact?

          I like it. Do you think that's necessary for this feature, or an acceptable follow-on JIRA?

          Show
          ndimiduk Nick Dimiduk added a comment - How feasible keeping count of how many times a block has been decompressed and if over a configurable threshold, instead shove the decompressed block back into the block cache in place of the compressed one? We already count if been accessed more than once? Could we leverage this fact? I like it. Do you think that's necessary for this feature, or an acceptable follow-on JIRA?
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Or, if hot, move the decompressed and decoded block up into L1?

          This sounds like a feature to add to CombinedCache. Can blocks become "less hot" and be demoted back down to a compressed state in L2, or is promotion a one-way street? I guess regular block eviction will take care of this naturally.

          Show
          ndimiduk Nick Dimiduk added a comment - Or, if hot, move the decompressed and decoded block up into L1? This sounds like a feature to add to CombinedCache. Can blocks become "less hot" and be demoted back down to a compressed state in L2, or is promotion a one-way street? I guess regular block eviction will take care of this naturally.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Rebased to master, fixed most tests in io.hfile.*

          Show
          ndimiduk Nick Dimiduk added a comment - Rebased to master, fixed most tests in io.hfile.*
          Hide
          stack stack added a comment -

          Do you think that's necessary for this feature, or an acceptable follow-on JIRA?

          Follow-on.

          Trying your latest patch. Will make new report with more variety to it.

          Show
          stack stack added a comment - Do you think that's necessary for this feature, or an acceptable follow-on JIRA? Follow-on. Trying your latest patch. Will make new report with more variety to it.
          Hide
          stack stack added a comment -

          Nick Dimiduk IMO, this can't be on by default given the report previous. Benefit is not enough. Will post a new report in next few days but think the benefit vs cost will be about same; tilted against.

          Show
          stack stack added a comment - Nick Dimiduk IMO, this can't be on by default given the report previous. Benefit is not enough. Will post a new report in next few days but think the benefit vs cost will be about same; tilted against.
          Hide
          stack stack added a comment -

          bq ....tilted against.

          ... it being on by default. For some use cases enabling it will make sense but not general case.

          Show
          stack stack added a comment - bq ....tilted against. ... it being on by default. For some use cases enabling it will make sense but not general case.
          Hide
          stack stack added a comment -

          Seeing this... trying to figure it in client:

          2014-07-26 00:38:46,924 DEBUG [IPC Client (1094864774) connection to c2021.halxg.cloudera.com/10.20.84.27:16020 from stack] ipc.RpcClient: IPC Client (1094864774) connection to c2021.halxg.cloudera.com/10.20.84.27:16020 from stack: got response header call_id: 1102 exception { exception_class_name: "java.io.IOException" stack_trace: "java.io.IOException\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2047)\n\tat org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)\n\tat java.lang.Thread.run(Thread.java:744)\nCaused by: java.lang.ArrayIndexOutOfBoundsException\n\tat org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:498)\n\tat org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:270)\n\tat org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:424)\n\tat org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:260)\n\tat org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:644)\n\tat org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:624)\n\tat org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:251)\n\tat org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:176)\n\tat org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)\n\tat org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:312)\n\tat org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:268)\n\tat org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:721)\n\tat org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:709)\n\tat org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:559)\n\tat org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:139)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:4002)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4082)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3951)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3933)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3920)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4865)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4838)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1619)\n\tat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29990)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2013)\n\t... 4 more\n" do_not_retry: false }, totalSize: 2735 bytes
          Show
          stack stack added a comment - Seeing this... trying to figure it in client: 2014-07-26 00:38:46,924 DEBUG [IPC Client (1094864774) connection to c2021.halxg.cloudera.com/10.20.84.27:16020 from stack] ipc.RpcClient: IPC Client (1094864774) connection to c2021.halxg.cloudera.com/10.20.84.27:16020 from stack: got response header call_id: 1102 exception { exception_class_name: "java.io.IOException" stack_trace: "java.io.IOException\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2047)\n\tat org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)\n\tat java.lang. Thread .run( Thread .java:744)\nCaused by: java.lang.ArrayIndexOutOfBoundsException\n\tat org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:498)\n\tat org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:270)\n\tat org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:424)\n\tat org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:260)\n\tat org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:644)\n\tat org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:624)\n\tat org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:251)\n\tat org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:176)\n\tat org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)\n\tat org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:312)\n\tat org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:268)\n\tat org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:721)\n\tat org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:709)\n\tat org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:559)\n\tat org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:139)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:4002)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4082)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3951)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3933)\n\tat org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3920)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4865)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4838)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1619)\n\tat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29990)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2013)\n\t... 4 more\n" do_not_retry: false }, totalSize: 2735 bytes
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Thanks stack. v02 addresses the exception you see here. It also:

          • makes the feature configurable. enable with hbase.block.data.cachecompressed=true
          • only retains data blocks in their "packed" format in memory. Everything else should be cached decompressed.

          One observation: when running the RS with a compression-enabled table, I see a lot of log messages says "got brand-new decompressor". I don't see any messages saying "got recycled compressor" (i've enabled debug logging for org.apache.hadoop.io.compress). I think we're not returning the objects to the pool. Running this small PE workload results in ~1600 said log lines. Run the same workload with hbase.block.data.cachecompressed=true, I see ~10k, so this will have a significant impact on the performance of this feature. Will investigate further.

          Show
          ndimiduk Nick Dimiduk added a comment - Thanks stack . v02 addresses the exception you see here. It also: makes the feature configurable. enable with hbase.block.data.cachecompressed=true only retains data blocks in their "packed" format in memory. Everything else should be cached decompressed. One observation: when running the RS with a compression-enabled table, I see a lot of log messages says "got brand-new decompressor". I don't see any messages saying "got recycled compressor" (i've enabled debug logging for org.apache.hadoop.io.compress). I think we're not returning the objects to the pool. Running this small PE workload results in ~1600 said log lines. Run the same workload with hbase.block.data.cachecompressed=true, I see ~10k, so this will have a significant impact on the performance of this feature. Will investigate further.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Looks like the lack of compressor reuse is a known issue, somewhere between HBASE-5881, HBASE-7435, and HADOOP-9171.

          Show
          ndimiduk Nick Dimiduk added a comment - Looks like the lack of compressor reuse is a known issue, somewhere between HBASE-5881 , HBASE-7435 , and HADOOP-9171 .
          Hide
          ndimiduk Nick Dimiduk added a comment - - edited

          re: lazy-decompress.02.0.pdf

          Attaching some metrics after running v02 with the new flag enabled/disabled.

          The test cluster is a single machine, configured with 12g heap and 0.6 for blockcache, so slighly over 7g mem available for cache. The TestTable is snappy compressed, written with PE --size=100. Shell status 'simple' reports compressionRatio=0.2437. RandomRead test is run with --size=25, so with feature enabled, we should be at ~6g of compressed blocks.

          client perceived perf

          With flag=false, I see slightly higher throughput (~7.5k vs ~8k) and slightly less gc (360ms vs 325ms).

          system load

          CPU load looks slightly higher with flag=true (user time diff is negligible, system time more noticeable, no difference in iowait).

          blockcache utilization

          There are definitely cache evictions happening with flag=false that do not happen with flag=true. The number of blocks cached is definitely higher with flag=true (~17.5k vs ~21.5k).

          IO

          No difference in reported IO requests. Probably it's the OS buffer cache to the rescue here.

          Show
          ndimiduk Nick Dimiduk added a comment - - edited re: lazy-decompress.02.0.pdf Attaching some metrics after running v02 with the new flag enabled/disabled. The test cluster is a single machine, configured with 12g heap and 0.6 for blockcache, so slighly over 7g mem available for cache. The TestTable is snappy compressed, written with PE --size=100. Shell status 'simple' reports compressionRatio=0.2437 . RandomRead test is run with --size=25, so with feature enabled, we should be at ~6g of compressed blocks. client perceived perf With flag=false, I see slightly higher throughput (~7.5k vs ~8k) and slightly less gc (360ms vs 325ms). system load CPU load looks slightly higher with flag=true (user time diff is negligible, system time more noticeable, no difference in iowait). blockcache utilization There are definitely cache evictions happening with flag=false that do not happen with flag=true. The number of blocks cached is definitely higher with flag=true (~17.5k vs ~21.5k). IO No difference in reported IO requests. Probably it's the OS buffer cache to the rescue here.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Looks like I have about 10g used as disk cache. Will run the workloads again with RS heap increased to 20g.

          Show
          ndimiduk Nick Dimiduk added a comment - Looks like I have about 10g used as disk cache. Will run the workloads again with RS heap increased to 20g.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12659287/HBASE-11331.02.patch
          against trunk revision .
          ATTACHMENT ID: 12659287

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 36 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestIOFencing
          org.apache.hadoop.hbase.regionserver.TestRegionReplicas
          org.apache.hadoop.hbase.client.TestReplicasClient
          org.apache.hadoop.hbase.master.TestRestartCluster
          org.apache.hadoop.hbase.TestRegionRebalancing

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12659287/HBASE-11331.02.patch against trunk revision . ATTACHMENT ID: 12659287 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 36 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.TestIOFencing org.apache.hadoop.hbase.regionserver.TestRegionReplicas org.apache.hadoop.hbase.client.TestReplicasClient org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.TestRegionRebalancing Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10308//console This message is automatically generated.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          re: lazy-decompress.02.1.pdf

          Attaching some metrics after running v02 with less os page cache and feature enabled/disabled.

          This time the total RS heap is increased to 20g, so the effective blockcache size is ~12g. The machine has 24g of RAM, so this squeezes the OS-available memory down to ~4g – when the blockcache is full. I've added blockcache size and OS page cache to the charts Same table, same compression ratio, same PE randomRead test, this time run with --size=45, so I expect ~11g of compressed blocks.

          I expected this to be an IO-bound workload with flag=false but to be cache-at-capacity for flag=true. The massive evictions look to me like the blockcache heapsize calculation is based on the decompressed size, not compressed size. Updating patch.

          Show
          ndimiduk Nick Dimiduk added a comment - re: lazy-decompress.02.1.pdf Attaching some metrics after running v02 with less os page cache and feature enabled/disabled. This time the total RS heap is increased to 20g, so the effective blockcache size is ~12g. The machine has 24g of RAM, so this squeezes the OS-available memory down to ~4g – when the blockcache is full. I've added blockcache size and OS page cache to the charts Same table, same compression ratio, same PE randomRead test, this time run with --size=45, so I expect ~11g of compressed blocks. I expected this to be an IO-bound workload with flag=false but to be cache-at-capacity for flag=true. The massive evictions look to me like the blockcache heapsize calculation is based on the decompressed size, not compressed size. Updating patch.
          Hide
          stack stack added a comment -

          Any chance of our fixing compressor reuse. As you say, I'd imagine it'd mess up any possibility of nice numbers when this feature enabled. I'm game to rerun test when you say its ready. That a fancy graph tool on front a tsdb? Nick Dimiduk? Nice graphs.

          Show
          stack stack added a comment - Any chance of our fixing compressor reuse. As you say, I'd imagine it'd mess up any possibility of nice numbers when this feature enabled. I'm game to rerun test when you say its ready. That a fancy graph tool on front a tsdb? Nick Dimiduk ? Nice graphs.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Any chance of our fixing compressor reuse.

          I'll look into it. If anything, it'll be a separate ticket.

          I'm game to rerun test when you say its ready.

          Hold off on rerunning until I verify/refute my hypothesis on the blockcache heapsize stuff.

          Nice graphs

          Awe shucks, thanks boss. This is Grafana running on top of OpenTSDB. No offense, but I find it nicer to use these dashboards than the gnuplot stuff we get otherwise. I'm still learning it, but I'll attach the dashboard json file I used to make the latest figures.

          Show
          ndimiduk Nick Dimiduk added a comment - Any chance of our fixing compressor reuse. I'll look into it. If anything, it'll be a separate ticket. I'm game to rerun test when you say its ready. Hold off on rerunning until I verify/refute my hypothesis on the blockcache heapsize stuff. Nice graphs Awe shucks, thanks boss. This is Grafana running on top of OpenTSDB. No offense, but I find it nicer to use these dashboards than the gnuplot stuff we get otherwise. I'm still learning it, but I'll attach the dashboard json file I used to make the latest figures.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Attach lazy-decompress.02.1.json, used with grafana 1.7rc for the legend aliasing.

          Show
          ndimiduk Nick Dimiduk added a comment - Attach lazy-decompress.02.1.json, used with grafana 1.7rc for the legend aliasing.
          Hide
          apurtell Andrew Purtell added a comment -

          Grafana running on top of OpenTSDB

          Very nice. Is there a way to autogen the JSON? Pardon for hijacking the issue.

          Show
          apurtell Andrew Purtell added a comment - Grafana running on top of OpenTSDB Very nice. Is there a way to autogen the JSON? Pardon for hijacking the issue.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Is there a way to autogen the JSON?

          The json file I attached was exported from Grafana; I did not build it in a text editor. It's UI starts with a black dashboard and you add to it the rows and column with metrics you want. Probably I'm not explaining it well; I'll leave it to grafana.org/docs

          Show
          ndimiduk Nick Dimiduk added a comment - Is there a way to autogen the JSON? The json file I attached was exported from Grafana; I did not build it in a text editor. It's UI starts with a black dashboard and you add to it the rows and column with metrics you want. Probably I'm not explaining it well; I'll leave it to grafana.org/docs
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Parking an updated patch. Rebased onto master, adds a bit of instrumentation to LruBlockCache when TRACE debugging is enabled, and adds an e2e test comparing directly the flag enabled and disabled. The only testing I've done with this one is mvn test -Dtest=org.apache.hadoop.hbase.io.hfile.*

          I've not been able to reproduce the jagged blockcache size spikes in recent runs. I'll take this patch through the ringer with my rig tomorrow, and report back with my findings.

          Show
          ndimiduk Nick Dimiduk added a comment - Parking an updated patch. Rebased onto master, adds a bit of instrumentation to LruBlockCache when TRACE debugging is enabled, and adds an e2e test comparing directly the flag enabled and disabled. The only testing I've done with this one is mvn test -Dtest=org.apache.hadoop.hbase.io.hfile.* I've not been able to reproduce the jagged blockcache size spikes in recent runs. I'll take this patch through the ringer with my rig tomorrow, and report back with my findings.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Adding a RB link since the patch has passed 100k. https://reviews.apache.org/r/24880/

          Show
          ndimiduk Nick Dimiduk added a comment - Adding a RB link since the patch has passed 100k. https://reviews.apache.org/r/24880/
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Sharing some more test results. These are running PerfEval randomRead test on a single machine cluster deployment. The RS gets a 20g heap out of 24g available on the box. Table is snappy compressed. The idea is to demonstrate a best-case scenario for this patch: you have enough ram for the whole compressed data set, but not after decompressed. The true run below shows some IO wait, so I may have the math on the compression ratio slightly off.

          Values in this table come from the "average" values reported on the attached screen shots. I've chosen some of the more critical metrics, but it's all there for reference. Let me know if there's a metric I'm missing, I can add it to the report (if OpenTSDB collects it, that is).

            hbase.block.data.cachecompressed=false hbase.block.data.cachecompressed=true delta
          hbase.regionserver.server.Get_num_ops 423 4.93k 1065%
          hbase.regionserver.server.Get_mean 19.14 ms 1.00 ms -94%
          hbase.regionserver.server.Get_99th_percentile 182.58 ms 33.17 ms -81%
          hbase.regionserver.jvmmetrics.GcTimeMillis 27.73 ms 401.16 ms 1346%
          proc.loadavg.1min 11.55 7.82 -32%
          proc.stat.cpu.percpu {type=iowait} 358.43 211.83 -40%
          hbase.regionserver.server.blockCacheCount 181.66k 722.55k 297%
          Show
          ndimiduk Nick Dimiduk added a comment - Sharing some more test results. These are running PerfEval randomRead test on a single machine cluster deployment. The RS gets a 20g heap out of 24g available on the box. Table is snappy compressed. The idea is to demonstrate a best-case scenario for this patch: you have enough ram for the whole compressed data set, but not after decompressed. The true run below shows some IO wait, so I may have the math on the compression ratio slightly off. Values in this table come from the "average" values reported on the attached screen shots. I've chosen some of the more critical metrics, but it's all there for reference. Let me know if there's a metric I'm missing, I can add it to the report (if OpenTSDB collects it, that is).   hbase.block.data.cachecompressed=false hbase.block.data.cachecompressed=true delta hbase.regionserver.server.Get_num_ops 423 4.93k 1065% hbase.regionserver.server.Get_mean 19.14 ms 1.00 ms -94% hbase.regionserver.server.Get_99th_percentile 182.58 ms 33.17 ms -81% hbase.regionserver.jvmmetrics.GcTimeMillis 27.73 ms 401.16 ms 1346% proc.loadavg.1min 11.55 7.82 -32% proc.stat.cpu.percpu {type=iowait} 358.43 211.83 -40% hbase.regionserver.server.blockCacheCount 181.66k 722.55k 297%
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Previous comment is re: v03-20g-045g-false.pdf, v03-20g-045g-true.pdf

          Show
          ndimiduk Nick Dimiduk added a comment - Previous comment is re: v03-20g-045g-false.pdf, v03-20g-045g-true.pdf
          Hide
          ndimiduk Nick Dimiduk added a comment -

          I'm now investigating those heart-beat looking GC spikes in the compressed=true graphs.

          Show
          ndimiduk Nick Dimiduk added a comment - I'm now investigating those heart-beat looking GC spikes in the compressed=true graphs.
          Hide
          vrodionov Vladimir Rodionov added a comment -

          The standard YCSB is very friendly to compressed block cache (especially, with zipfian data access pattern). Just to let you know, Nick.

          Show
          vrodionov Vladimir Rodionov added a comment - The standard YCSB is very friendly to compressed block cache (especially, with zipfian data access pattern). Just to let you know, Nick.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Vladimir Rodionov I think PE has zipfian now too (or maybe that's just the value size?). I'll take a look at YCSB, thanks.

          Show
          ndimiduk Nick Dimiduk added a comment - Vladimir Rodionov I think PE has zipfian now too (or maybe that's just the value size?). I'll take a look at YCSB, thanks.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          re: v03-20g-045g-true-16h.pdf

          I baked this patch overnight (16h + some warmup) with compress=true. Same --size=45. Things look stable.

          Show
          ndimiduk Nick Dimiduk added a comment - re: v03-20g-045g-true-16h.pdf I baked this patch overnight (16h + some warmup) with compress=true. Same --size=45. Things look stable.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Interesting that proc.loadavg.1min decreases even though we know there's more decompress operations happening and the gc activity increases. Maybe that measurement includes iowait?

          Show
          ndimiduk Nick Dimiduk added a comment - Interesting that proc.loadavg.1min decreases even though we know there's more decompress operations happening and the gc activity increases. Maybe that measurement includes iowait?
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Attaching patch addressing rb feedback.

          Show
          ndimiduk Nick Dimiduk added a comment - Attaching patch addressing rb feedback.
          Hide
          stack stack added a comment -

          You figure the compressor issue, our not reusing them?

          13x the GC because we are doing 10x the throughput is fair enough. All other numbers are very nice.

          This is best case (when =false, we are seeking? Or is it always inside fscache?)

          What is the 'cost' keeping stuff compressed? What if you do a run where all fits in cache, for both cases?

          Show
          stack stack added a comment - You figure the compressor issue, our not reusing them? 13x the GC because we are doing 10x the throughput is fair enough. All other numbers are very nice. This is best case (when =false, we are seeking? Or is it always inside fscache?) What is the 'cost' keeping stuff compressed? What if you do a run where all fits in cache, for both cases?
          Hide
          ndimiduk Nick Dimiduk added a comment -

          I've tripped my assertions around blockcache count metrics. Updating the patch to print more details. Will file a separate ticket once I understand what's happening here.

          Show
          ndimiduk Nick Dimiduk added a comment - I've tripped my assertions around blockcache count metrics. Updating the patch to print more details. Will file a separate ticket once I understand what's happening here.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Thanks for having a look stack.

          You figure the compressor issue, our not reusing them?

          I've punted on the compressor is a non-issue, but I haven't run with a profiler yet. I think it was related to the non-native gz impl while running locally. I'll re-enable tracing there as well with the next runs and see what I see.

          This is best case (when =false, we are seeking? Or is it always inside fscache?)

          Yes, this configuration is targeting a best case for this patch. The fscache is minimized with this config, seems to stay down around 3.5g (vs 11.5g blockcache). Compression ratio is reported as 0.2437, so --size=45 should be ~11g compressed – larger than the fscache. Because the PE test is random, I believe we'll be thrashing the fscache with both =true and =false. The iowait charts indicate both configs are doing io constantly, just more with =false (as expected).

          What is the 'cost' keeping stuff compressed? What if you do a run where all fits in cache, for both cases?

          I'm testing a couple more scenarios, this one was already on the list.

          Show
          ndimiduk Nick Dimiduk added a comment - Thanks for having a look stack . You figure the compressor issue, our not reusing them? I've punted on the compressor is a non-issue, but I haven't run with a profiler yet. I think it was related to the non-native gz impl while running locally. I'll re-enable tracing there as well with the next runs and see what I see. This is best case (when =false, we are seeking? Or is it always inside fscache?) Yes, this configuration is targeting a best case for this patch. The fscache is minimized with this config, seems to stay down around 3.5g (vs 11.5g blockcache). Compression ratio is reported as 0.2437, so --size=45 should be ~11g compressed – larger than the fscache. Because the PE test is random, I believe we'll be thrashing the fscache with both =true and =false. The iowait charts indicate both configs are doing io constantly, just more with =false (as expected). What is the 'cost' keeping stuff compressed? What if you do a run where all fits in cache, for both cases? I'm testing a couple more scenarios, this one was already on the list.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12663549/HBASE-11331.05.patch
          against trunk revision .
          ATTACHMENT ID: 12663549

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 44 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 7 warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + LOG.trace("freed " + StringUtils.byteDesc(bytesFreed) + " from single and multi buckets");
          + LOG.trace("freed " + StringUtils.byteDesc(bytesFreed) + " total from all three buckets ");

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.hbase.mapreduce.TestImportTsv.testMROnTable(TestImportTsv.java:113)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663549/HBASE-11331.05.patch against trunk revision . ATTACHMENT ID: 12663549 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 44 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 7 warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + LOG.trace("freed " + StringUtils.byteDesc(bytesFreed) + " from single and multi buckets"); + LOG.trace("freed " + StringUtils.byteDesc(bytesFreed) + " total from all three buckets "); +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 2 zombie test(s): at org.apache.hadoop.hbase.mapreduce.TestImportTsv.testMROnTable(TestImportTsv.java:113) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10521//console This message is automatically generated.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          As best as I can tell, both of these configurations stay entirely in BlockCache. Verified by looking at the RS BlockCache stats and confirmed by the low iowait stat being basically flat for them both. Looks like enabling this feature when it's not needed is quite expensive.

            =false, 11g =true, 40g delta
          hbase.regionserver.server.Get_num_ops 15.15 k 6.07 k -60%
          hbase.regionserver.server.Get_mean 0.00 ns 0.00 ns 0%
          hbase.regionserver.server.Get_99th_percentile 1.00 ms 22.65 ms 2165%
          hbase.regionserver.jvmmetrics.GcTimeMillis 48.89 ms 441.33 ms 802%
          proc.loadavg.1min 0.56 3.25 480%
          proc.stat.cpu.percpu {type=iowait} 3.55 3.47 -2%
          hbase.regionserver.server.blockCacheCount 181.75 k 666.44 k 266%
          Show
          ndimiduk Nick Dimiduk added a comment - As best as I can tell, both of these configurations stay entirely in BlockCache. Verified by looking at the RS BlockCache stats and confirmed by the low iowait stat being basically flat for them both. Looks like enabling this feature when it's not needed is quite expensive.   =false, 11g =true, 40g delta hbase.regionserver.server.Get_num_ops 15.15 k 6.07 k -60% hbase.regionserver.server.Get_mean 0.00 ns 0.00 ns 0% hbase.regionserver.server.Get_99th_percentile 1.00 ms 22.65 ms 2165% hbase.regionserver.jvmmetrics.GcTimeMillis 48.89 ms 441.33 ms 802% proc.loadavg.1min 0.56 3.25 480% proc.stat.cpu.percpu {type=iowait} 3.55 3.47 -2% hbase.regionserver.server.blockCacheCount 181.75 k 666.44 k 266%
          Hide
          ndimiduk Nick Dimiduk added a comment -

          re: hbase-hbase-master-hor17n36.gq1.ygridcore.net.log

          I've tripped my assertions around blockcache count metrics.

          False alarm. There is a case when the reported metric is greater than 5% different from the actual map size, but it happened while executing the shutdown hooks, not while the job was running. Attaching the log file in case anyone is curious.

          Show
          ndimiduk Nick Dimiduk added a comment - re: hbase-hbase-master-hor17n36.gq1.ygridcore.net.log I've tripped my assertions around blockcache count metrics. False alarm. There is a case when the reported metric is greater than 5% different from the actual map size, but it happened while executing the shutdown hooks, not while the job was running. Attaching the log file in case anyone is curious.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Running some numbers where dataset is larger than available blockcache in both cases, will report back. In the mean time, would be nice to get some more eyes on the patch. I'll also be creating patches for branch-1 and 0.98.

          ping Enis Soztutar, Andrew Purtell, Lars Hofhansl

          Show
          ndimiduk Nick Dimiduk added a comment - Running some numbers where dataset is larger than available blockcache in both cases, will report back. In the mean time, would be nice to get some more eyes on the patch. I'll also be creating patches for branch-1 and 0.98. ping Enis Soztutar , Andrew Purtell , Lars Hofhansl
          Hide
          ndimiduk Nick Dimiduk added a comment -

          This data is from increasing the query range to size=100; more data than fits in cache with either config. I had to run the warmup and test for longer periods because it took longer for the oscache and BlockCache to reach steady-state.

            =false, 100g =true, 100g delta
          hbase.regionserver.server.Get_num_ops 313.83 466.69 49%
          hbase.regionserver.server.Get_mean 28.91 ms 20.00 ms -31%
          hbase.regionserver.server.Get_99th_percentile 221.06 ms 197.30 ms -11%
          hbase.regionserver.jvmmetrics.GcTimeMillis 26.99 ms 48.84 ms 81%
          proc.loadavg.1min 11.71 12.00 2%
          proc.stat.cpu.percpu {type=iowait} 343.11 404.10 18%
          hbase.regionserver.server.blockCacheCount 181.85 k 716.79 k 294%

          Overall, I'd say that you want this feature enabled unless:

          1. your data decompressed fits in BlockCache
          2. your machines are not your own and CPU time is at a premium (ie, AWS).

          2nd point above is just a guess. It's likely other factors are at play in these environments.

          Show
          ndimiduk Nick Dimiduk added a comment - This data is from increasing the query range to size=100; more data than fits in cache with either config. I had to run the warmup and test for longer periods because it took longer for the oscache and BlockCache to reach steady-state.   =false, 100g =true, 100g delta hbase.regionserver.server.Get_num_ops 313.83 466.69 49% hbase.regionserver.server.Get_mean 28.91 ms 20.00 ms -31% hbase.regionserver.server.Get_99th_percentile 221.06 ms 197.30 ms -11% hbase.regionserver.jvmmetrics.GcTimeMillis 26.99 ms 48.84 ms 81% proc.loadavg.1min 11.71 12.00 2% proc.stat.cpu.percpu {type=iowait} 343.11 404.10 18% hbase.regionserver.server.blockCacheCount 181.85 k 716.79 k 294% Overall, I'd say that you want this feature enabled unless: your data decompressed fits in BlockCache your machines are not your own and CPU time is at a premium (ie, AWS). 2nd point above is just a guess. It's likely other factors are at play in these environments.
          Hide
          stack stack added a comment -

          50% more ops for 80% more GC (20% more CPU) sounds like reasonable trade off. Would be interesting to see how it does in a long running test? Does the extra GC bring on the dreaded FGC? Is GC steady? On the blockCacheCount, this is indication of our caching more blocks? 3x?

          Show
          stack stack added a comment - 50% more ops for 80% more GC (20% more CPU) sounds like reasonable trade off. Would be interesting to see how it does in a long running test? Does the extra GC bring on the dreaded FGC? Is GC steady? On the blockCacheCount, this is indication of our caching more blocks? 3x?
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Would be interesting to see how it does in a long running test?

          Today's results were from 90m warmup + 40m test. The 16h run I posted on Thursday [0] seemed stable. No slope in those graphs. This is just PE mind you, no concurrent writes.

          On the blockCacheCount, this is indication of our caching more blocks? 3x?

          Yes, exactly. It's purely compression ratio manifest in memory. All this is using SNAPPY and the reported compression ratio is 0.2473, thus the ~3x increase. Probably using GZ would have better BC utilization buy higher CPU load (should translate to a standard compression benchmark).

          [0]: https://issues.apache.org/jira/secure/attachment/12663479/v03-20g-045g-true-16h.pdf

          Show
          ndimiduk Nick Dimiduk added a comment - Would be interesting to see how it does in a long running test? Today's results were from 90m warmup + 40m test. The 16h run I posted on Thursday [0] seemed stable. No slope in those graphs. This is just PE mind you, no concurrent writes. On the blockCacheCount, this is indication of our caching more blocks? 3x? Yes, exactly. It's purely compression ratio manifest in memory. All this is using SNAPPY and the reported compression ratio is 0.2473, thus the ~3x increase. Probably using GZ would have better BC utilization buy higher CPU load (should translate to a standard compression benchmark). [0] : https://issues.apache.org/jira/secure/attachment/12663479/v03-20g-045g-true-16h.pdf
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Adding intended fix-versions. Will sort with RM's as they weigh in.

          Show
          ndimiduk Nick Dimiduk added a comment - Adding intended fix-versions. Will sort with RM's as they weigh in.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Commit applies cleanly onto branch-1. Posting a patch for 0.98, based on applying commit for v05 and resolving conflicts. Tested this one with mvn test -Dtest=org.apache.hadoop.hbase.io.hfile.* and a local sandbox run with PerfEval and the flag enabled.

          Show
          ndimiduk Nick Dimiduk added a comment - Commit applies cleanly onto branch-1. Posting a patch for 0.98, based on applying commit for v05 and resolving conflicts. Tested this one with mvn test -Dtest=org.apache.hadoop.hbase.io.hfile.* and a local sandbox run with PerfEval and the flag enabled.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12664245/HBASE-11331.05-0.98.patch
          against trunk revision .
          ATTACHMENT ID: 12664245

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 46 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10574//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664245/HBASE-11331.05-0.98.patch against trunk revision . ATTACHMENT ID: 12664245 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 46 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10574//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment - - edited

          +1 for 0.98.

          This is good work.

          Didn't relish going through HFile code again (hasn't been long enough since HBASE-7544). Nothing jumped out to me as wrong. I will try taking another look when not spread so thin.

          The schema option needs to be false by default. We can write a note in the release notes and/or manual that if using compression the user should try enabling the option and only disable if observing too high CPU usage or too much GC under their workload.

          Edit: I reviewed the 0.98 patch

          Show
          apurtell Andrew Purtell added a comment - - edited +1 for 0.98. This is good work. Didn't relish going through HFile code again (hasn't been long enough since HBASE-7544 ). Nothing jumped out to me as wrong. I will try taking another look when not spread so thin. The schema option needs to be false by default. We can write a note in the release notes and/or manual that if using compression the user should try enabling the option and only disable if observing too high CPU usage or too much GC under their workload. Edit: I reviewed the 0.98 patch
          Hide
          ndimiduk Nick Dimiduk added a comment -

          The schema option needs to be false by default

          This is the intention. If it's not, I've messed up. Will double-check before commit.

          Misty Stanley-Jones where would I put a section in the book on this feature? Maybe something under "9.6.4. Block Cache" ?

          Show
          ndimiduk Nick Dimiduk added a comment - The schema option needs to be false by default This is the intention. If it's not, I've messed up. Will double-check before commit. Misty Stanley-Jones where would I put a section in the book on this feature? Maybe something under "9.6.4. Block Cache" ?
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Any other comments from reviewers? Are we happy with the config name "hbase.block.data.cachecompressed" ?

          Show
          ndimiduk Nick Dimiduk added a comment - Any other comments from reviewers? Are we happy with the config name "hbase.block.data.cachecompressed" ?
          Hide
          enis Enis Soztutar added a comment -

          Left some comments at RB.

          Any other comments from reviewers? Are we happy with the config name "hbase.block.data.cachecompressed" ?

          Should this be hbase.block.cache.data.cachecompressed?

          Show
          enis Enis Soztutar added a comment - Left some comments at RB. Any other comments from reviewers? Are we happy with the config name "hbase.block.data.cachecompressed" ? Should this be hbase.block.cache.data.cachecompressed ?
          Hide
          enis Enis Soztutar added a comment -

          Also add the parameter to hbase-default.xml and release notes so that it is findable.

          Show
          enis Enis Soztutar added a comment - Also add the parameter to hbase-default.xml and release notes so that it is findable.
          Hide
          misty Misty Stanley-Jones added a comment -

          Nick Dimiduk wow there are a lot of comments here and I don't have bandwidth to parse it right now. From looking at the description, it looks like maybe this needs a mention in the blockcache section and also in the compression / codec appendix. WDYT?

          Show
          misty Misty Stanley-Jones added a comment - Nick Dimiduk wow there are a lot of comments here and I don't have bandwidth to parse it right now. From looking at the description, it looks like maybe this needs a mention in the blockcache section and also in the compression / codec appendix. WDYT?
          Hide
          apurtell Andrew Purtell added a comment -

          Moving to 0.98.7

          Show
          apurtell Andrew Purtell added a comment - Moving to 0.98.7
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Quick update: I've hit an ArrayIndexOutOfBoundsException while running a mixed workload via LTT. Investigating.

          Show
          ndimiduk Nick Dimiduk added a comment - Quick update: I've hit an ArrayIndexOutOfBoundsException while running a mixed workload via LTT. Investigating.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Patch v6 addresses Enis's comments from RB (not much) and resolves a bug in the interaction between this feature and serialization of blocks for L2 cache's. In addition to resolving the issue, I've added some testing for serialization (I didn't find any previously) and sured up the meaning of HFileBlock.equals().

          master patch applies cleanly to branch-1. 0.98 patch included as well. Please give it one more look so we can put this one to bed.

          Show
          ndimiduk Nick Dimiduk added a comment - Patch v6 addresses Enis's comments from RB (not much) and resolves a bug in the interaction between this feature and serialization of blocks for L2 cache's. In addition to resolving the issue, I've added some testing for serialization (I didn't find any previously) and sured up the meaning of HFileBlock.equals(). master patch applies cleanly to branch-1. 0.98 patch included as well. Please give it one more look so we can put this one to bed.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667477/HBASE-11331.06.patch
          against trunk revision .
          ATTACHMENT ID: 12667477

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 44 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + assertEquals((int) encodedSizes.get(blockId), blockFromHFile.getUncompressedSizeWithoutHeader());
          + LOG.info("packedHeapsize=" + packedHeapsize + ", unpackedHeadsize=" + blockUnpacked.heapSize());

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667477/HBASE-11331.06.patch against trunk revision . ATTACHMENT ID: 12667477 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 44 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + assertEquals((int) encodedSizes.get(blockId), blockFromHFile.getUncompressedSizeWithoutHeader()); + LOG.info("packedHeapsize=" + packedHeapsize + ", unpackedHeadsize=" + blockUnpacked.heapSize()); +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10796//console This message is automatically generated.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Addressing line length nits from buildbot.

          Show
          ndimiduk Nick Dimiduk added a comment - Addressing line length nits from buildbot.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          +1

          Show
          yuzhihong@gmail.com Ted Yu added a comment - +1
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667525/HBASE-11331.07.patch
          against trunk revision .
          ATTACHMENT ID: 12667525

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 44 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestMultiParallel
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting

          -1 core zombie tests. There are 6 zombie test(s): at org.apache.hadoop.hbase.util.TestBytes.testToStringBytesBinaryReversible(TestBytes.java:295)
          at org.apache.hadoop.hbase.backup.TestHFileArchiving.testCleaningRace(TestHFileArchiving.java:372)
          at org.apache.hadoop.hbase.ipc.TestDelayedRpc.testTooManyDelayedRpcs(TestDelayedRpc.java:202)
          at org.apache.hadoop.hbase.TestIOFencing.testFencingAroundCompaction(TestIOFencing.java:224)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667525/HBASE-11331.07.patch against trunk revision . ATTACHMENT ID: 12667525 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 44 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestMultiParallel org.apache.hadoop.hbase.master.TestDistributedLogSplitting -1 core zombie tests . There are 6 zombie test(s): at org.apache.hadoop.hbase.util.TestBytes.testToStringBytesBinaryReversible(TestBytes.java:295) at org.apache.hadoop.hbase.backup.TestHFileArchiving.testCleaningRace(TestHFileArchiving.java:372) at org.apache.hadoop.hbase.ipc.TestDelayedRpc.testTooManyDelayedRpcs(TestDelayedRpc.java:202) at org.apache.hadoop.hbase.TestIOFencing.testFencingAroundCompaction(TestIOFencing.java:224) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10800//console This message is automatically generated.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Left some comments at RB.

          Any other comments from reviewers? Are we happy with the config name "hbase.block.data.cachecompressed" ?

          Should this be hbase.block.cache.data.cachecompressed?

          There's not a lot of consistency around cache configurations. We also have:

          • hbase.rs.cacheblocksonwrite
          • hfile.block.index.cacheonwrite
          • hfile.block.bloom.cacheonwrite
          • hbase.rs.evictblocksonclose
          • hbase.bucketcache.*
          • hbase.rs.prefetchblocksonopen
          • hbase.offheapcache.minblocksize
          Show
          ndimiduk Nick Dimiduk added a comment - Left some comments at RB. Any other comments from reviewers? Are we happy with the config name "hbase.block.data.cachecompressed" ? Should this be hbase.block.cache.data.cachecompressed ? There's not a lot of consistency around cache configurations. We also have: hbase.rs.cacheblocksonwrite hfile.block.index.cacheonwrite hfile.block.bloom.cacheonwrite hbase.rs.evictblocksonclose hbase.bucketcache.* hbase.rs.prefetchblocksonopen hbase.offheapcache.minblocksize
          Hide
          stack stack added a comment -

          +1

          Add to the simplification of block cache config a note that we need to unify the configuration args?

          Needs fat release note and on commit, add something to the refguide in the block cache section else I'm afraid folks won't come across this feature.

          Show
          stack stack added a comment - +1 Add to the simplification of block cache config a note that we need to unify the configuration args? Needs fat release note and on commit, add something to the refguide in the block cache section else I'm afraid folks won't come across this feature.
          Hide
          enis Enis Soztutar added a comment -

          Is this right? Assuming HeapByteBuffer here.

                   >    @Override
                                                                        >    public void serialize(ByteBuffer destination) {
                                                                        > -    ByteBuffer dupBuf = this.buf.duplicate();
                                                                        > -    dupBuf.rewind();
                                                                        > -    destination.put(dupBuf);
                                                                        > +    // assumes HeapByteBuffer
                                                                        > +    destination.put(this.buf.array(), this.buf.arrayOffset()
                                                                        > +      getSerializedLength() - EXTRA_SERIALIZATION_SPACE);
                                                                        >      serializeExtraInfo(destination);
                                                                        >    }
          

          On naming conf, I'll go with whatever you think is better. Agreed with Stack. Fat release note would be good.

          Show
          enis Enis Soztutar added a comment - Is this right? Assuming HeapByteBuffer here. > @Override > public void serialize(ByteBuffer destination) { > - ByteBuffer dupBuf = this .buf.duplicate(); > - dupBuf.rewind(); > - destination.put(dupBuf); > + // assumes HeapByteBuffer > + destination.put( this .buf.array(), this .buf.arrayOffset() > + getSerializedLength() - EXTRA_SERIALIZATION_SPACE); > serializeExtraInfo(destination); > } On naming conf, I'll go with whatever you think is better. Agreed with Stack. Fat release note would be good.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Thanks Enis Soztutar, stack for having a look.

          Add to the simplification of block cache config a note that we need to unify the configuration args?

          Alright, I'll open a ticket.

          Needs fat release note and on commit, add something to the refguide in the block cache section else I'm afraid folks won't come across this feature.

          Release note I'd planned, just what's in the commit message; you want more than this?

          When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the BlockCache in their on-disk format. This is different from the default behavior, which decompresses and decrypts a block before caching.
          

          I'll open a docs ticket for updating the book.

          Is this right? Assuming HeapByteBuffer here.

          Alas, yes. There's a number of assumptions baked into HFileBlock about HeapByteArrays. The whole read-ahead of the next block's header feature is based on this supposition (look for reads beyond limit without checking capacity; this is how I ran into the serialization bug in the first place).

          Show
          ndimiduk Nick Dimiduk added a comment - Thanks Enis Soztutar , stack for having a look. Add to the simplification of block cache config a note that we need to unify the configuration args? Alright, I'll open a ticket. Needs fat release note and on commit, add something to the refguide in the block cache section else I'm afraid folks won't come across this feature. Release note I'd planned, just what's in the commit message; you want more than this? When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the BlockCache in their on-disk format. This is different from the default behavior, which decompresses and decrypts a block before caching. I'll open a docs ticket for updating the book. Is this right? Assuming HeapByteBuffer here. Alas, yes. There's a number of assumptions baked into HFileBlock about HeapByteArrays. The whole read-ahead of the next block's header feature is based on this supposition (look for reads beyond limit without checking capacity; this is how I ran into the serialization bug in the first place).
          Hide
          enis Enis Soztutar added a comment -

          Release note I'd planned, just what's in the commit message; you want more than this?

          Maybe add why you would want this (X times more block cache while trading more CPU usage).

          There's a number of assumptions baked into HFileBlock about HeapByteArrays

          Ok. Seems we may need to address that in the future.

          1 for 0.99.

          Show
          enis Enis Soztutar added a comment - Release note I'd planned, just what's in the commit message; you want more than this? Maybe add why you would want this (X times more block cache while trading more CPU usage). There's a number of assumptions baked into HFileBlock about HeapByteArrays Ok. Seems we may need to address that in the future. 1 for 0.99 .
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Opened HBASE-11938 for cache configuration and HBASE-11939 for updating the book.

          Show
          ndimiduk Nick Dimiduk added a comment - Opened HBASE-11938 for cache configuration and HBASE-11939 for updating the book.
          Hide
          stack stack added a comment -

          you want more than this?

          No. The note you have looks good.

          Show
          stack stack added a comment - you want more than this? No. The note you have looks good.
          Hide
          apurtell Andrew Purtell added a comment -

          +1 for 0.98.

          0.98.7 is open.

          Show
          apurtell Andrew Purtell added a comment - +1 for 0.98. 0.98.7 is open.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Pushed to 0.98+. FYI, I re-ran the suite over o.a.h.h.io.hfile.* on each branch before pushing. I also added stack's excellent summary of the resource utilization involved in this feature to the release note.

          Thanks for the patient reviews everyone!

          Show
          ndimiduk Nick Dimiduk added a comment - Pushed to 0.98+. FYI, I re-ran the suite over o.a.h.h.io.hfile.* on each branch before pushing. I also added stack 's excellent summary of the resource utilization involved in this feature to the release note. Thanks for the patient reviews everyone!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK #5489 (See https://builds.apache.org/job/HBase-TRUNK/5489/)
          HBASE-11331 [blockcache] lazy block decompression (ndimiduk: rev eec15bd17218a2543c9f4cbb9a78841a9fdec043)

          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
          • hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5489 (See https://builds.apache.org/job/HBase-TRUNK/5489/ ) HBASE-11331 [blockcache] lazy block decompression (ndimiduk: rev eec15bd17218a2543c9f4cbb9a78841a9fdec043) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-0.98 #510 (See https://builds.apache.org/job/HBase-0.98/510/)
          HBASE-11331 [blockcache] lazy block decompression (ndimiduk: rev b8851309e0ff1e04a3a0abd5fbad6d1868a151bc)

          • hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
          • hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-0.98 #510 (See https://builds.apache.org/job/HBase-0.98/510/ ) HBASE-11331 [blockcache] lazy block decompression (ndimiduk: rev b8851309e0ff1e04a3a0abd5fbad6d1868a151bc) hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-1.0 #171 (See https://builds.apache.org/job/HBase-1.0/171/)
          HBASE-11331 [blockcache] lazy block decompression (ndimiduk: rev 4d51cf0ee7d2a508569bd96630eb6d2d9afd6493)

          • hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-1.0 #171 (See https://builds.apache.org/job/HBase-1.0/171/ ) HBASE-11331 [blockcache] lazy block decompression (ndimiduk: rev 4d51cf0ee7d2a508569bd96630eb6d2d9afd6493) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #483 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/483/)
          HBASE-11331 [blockcache] lazy block decompression (ndimiduk: rev b8851309e0ff1e04a3a0abd5fbad6d1868a151bc)

          • hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #483 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/483/ ) HBASE-11331 [blockcache] lazy block decompression (ndimiduk: rev b8851309e0ff1e04a3a0abd5fbad6d1868a151bc) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
          Hide
          enis Enis Soztutar added a comment -

          Closing this issue after 0.99.0 release.

          Show
          enis Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

            People

            • Assignee:
              ndimiduk Nick Dimiduk
              Reporter:
              ndimiduk Nick Dimiduk
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development