[HIVE-20165] Enable ZLIB for streaming ingest

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 4.0.0
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Component/s: Streaming, Transactions
• Target Version/s: 3.2.0, 4.0.0
• Assignee: Prasanth Jayachandran
• Reporter: Prasanth Jayachandran
• Created: 13/Jul/18 06:56
• Updated: 17/Nov/22 08:55
• Resolved: 18/Jul/18 18:29

Description

Per gopalv's recommendation tried running streaming ingest with and without zlib. Following are the numbers

 
Compression: NONE
Total rows committed: 93800000
Throughput: 1563333 rows/second
$ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
14.1 G  /apps/hive/warehouse/prasanth.db/culvert
 
Compression: ZLIB
Total rows committed: 92100000
Throughput: 1535000 rows/second
$ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
7.4 G  /apps/hive/warehouse/prasanth.db/culvert
 
ZLIB is getting us 2x compression and only 2% lesser throughput. We should enable ZLIB by default for streaming ingest.