[IMPALA-7406] Flatbuffer wrappers use almost as much memory as underlying data

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.1.0
• Component/s: Catalog
• Epic Color: ghx-label-9
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 07/Aug/18 21:14
• Updated: 17/Aug/18 23:19
• Resolved: 17/Aug/18 18:42

Description

Currently the file descriptors stored in the catalogd memory for each partition use a FlatBuffer to reduce the number of separate objects on the Java heap. However, the FlatBuffer objects internally each store a ByteBuffer and int position, so each object takes 32 bytes on its own. The ByteBuffer takes 56 bytes since it stores various references, endianness, limit, mark, position, etc. This amounts to about 88 bytes overhead on top of the actual underlying flatbuf byte array which is typically around 100 bytes for a single-block file. So, we're have about a 1:1 ratio of memory overhead and a 2:1 ratio of object count overhead for each partition.

If we simply stored the byte[] array and constructed wrappers on demand, we'd save 88 bytes and 2 objects per partition. The downside is that we'd need to do short-lived ByteBuffer allocations at access time, and based on some benchmarking I did, they don't get escape-analyzed out. So, it's not a super clear win, but still worth considering.