[HBASE-15542] Minimize garbage memory allocations during compaction

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Component/s: Compaction
• Assignee: Vladimir Rodionov
• Reporter: Vladimir Rodionov
• Created: 26/Mar/16 20:47
• Updated: 30/Jun/17 05:46

Description

Compactions are CPU-bound, not I/O bound. We can mitigate this by increasing number of compaction threads for small and large compactions, but excessive object allocation limits maximum number of compaction threads we can use w/o bringing down RS due to very frequent GC cycles.

Issue Links

• is part of: HBASE-15492 Memory usage optimizations
• is related to: HBASE-15555 Memory Management
• requires: HBASE-15509 Avoid copy of block data in HFileBlock$Writer.finishBlock

Sub-Tasks

• HFileScannerImpl.getCell() allocates cell object on every call: Open
• HFileReaderImpl.readBlock allocates memory for HFileBlock on every call: Patch Available
• org.apache.hadoop.io.compress.DecompressorStream allocates too much memory: Open
• StoreFile$Writer.appendGeneralBloomFilter generates extra KV: Resolved
• Make DeleteBloomFilters work with BloomContext: Resolved
• Change read path Bloom check to work with Cells with out any copy: Resolved