[HBASE-15950] Fix memstore size estimates to be more tighter

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0, 2.0.0
• Hadoop Flags: Incompatible change

Release Note

The estimates of heap usage by the memstore objects (KeyValue, object and array header sizes, etc) have been made more accurate for heap sizes up to 32G (using CompressedOops), resulting in them dropping by 10-50% in practice. This also results in less number of flushes and compactions due to "fatter" flushes. YMMV. As a result, the actual heap usage of the memstore before being flushed may increase by up to 100%. If configured memory limits for the region server had been tuned based on observed usage, this change could result in worse GC behavior or even OutOfMemory errors. Set the environment property (not hbase-site.xml) "hbase.memorylayout.use.unsafe" to false to disable.

• Assignee: Enis Soztutar
• Reporter: Enis Soztutar
• Created: 03/Jun/16 03:56
• Updated: 20/Apr/18 02:52
• Resolved: 15/Jun/16 21:23

Description

While testing something else, I was loading a region with a lot of data. Writing 30M cells in 1M rows, with 1 byte values.

The memstore size turned out to be estimated as 4.5GB, while with the JFR profiling I can see that we are using 2.8GB for all the objects in the memstore (KV + KV byte[] + CSLM.Node + CSLM.Index).

This obviously means that there is room in the write cache that we are not effectively using.

Issue Links

• is related to: HBASE-17254 Do alignment in heapOverhead() for ExtendedCell implementations
• is related to: HBASE-16058 TestHRegion fails on 1.4 builds