[IMPALA-7424] Improve in-memory representation of incremental stats

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.13.0, Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Catalog
• Epic Color: ghx-label-1
• Assignee: Bharath Vissapragada
• Reporter: Bharath Vissapragada
• Created: 10/Aug/18 19:30
• Updated: 18/Oct/18 21:32
• Resolved: 31/Aug/18 22:32

Description

Incremental stats are stored in the HMS' parameters map as plain Java Strings. This is suboptimal since Java String class internally uses UTF-16 encoding for the underlying bytes. The idea here is to switch to a byte array representation so that we can reduce the memory usage by half (8 bytes).  We can also compress the byte array using gzip compression and lazily decompress them when needed (typically during the incremental stats computation's finalization phase).

A prototype of this patch on a real-world Catalog dump showed ~54% JVM heap usage reduction (end-to-end) and ~79% reduction in the heap footprint for the incremental stats.

Issue Links

• is related to: IMPALA-2648 catalogd crashes when serialized messages are over 2 GB