[IMPALA-5500] Reduce catalog topic size when --compact_catalog_topic is enabled

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.10.0
• Component/s: Catalog
• Target Version: Impala 2.10.0
• Epic Color: ghx-label-5
• Assignee: Dimitris Tsirogiannis
• Reporter: Dimitris Tsirogiannis
• Created: 13/Jun/17 17:49
• Updated: 04/Apr/18 19:04
• Resolved: 29/Jun/17 03:33

Description

IMPALA-4029 introduced the use of flatbuffers for storing file and block metadata. The flatbuffers are stored as binary fields in the THdfsPartition thrift struct. However, by doing so, we limit the ability of Thrift to compact catalog update messages sent by the statestore to the impalad nodes when --compact_catalog_topic is set to true. In particular, if that flag is set to true upgrading to Impala 2.9 will result in the size of catalog topic updates to increase by 2X, thereby increasing network and CPU utilization both at the statestore and the impalads. We should consider compressing the catalog update messages sent from the statestore to the impalad nodes when that flag is set to true.