[HIVE-21037] Replicate column statistics for Hive tables

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: HiveServer2
• Target Version/s: 4.0.0
• Assignee: Ashutosh Bapat
• Reporter: Ashutosh Bapat
• Created: 13/Dec/18 09:33
• Updated: 09/Jan/19 02:38

Description

Statistics is important for query optimizations and thus keeping those up-to-date on replica is important from query performance perspective. The statistics are collected by scanning a table entirely. Thus when the data is replicated a. we could update the statistics by scanning it on replica or b. we could just replicate the statistics also. For following reasons we desire to go by the second approach instead of the first.

1. Scanning the data on replica isn’t a good option since it wastes CPU cycles and puts load during replication, which can be significant.
2. Storages like S3 may not have compute capabilities and thus when we are replicating from on-prem to cloud, we can not rely on the target to gather statistics.
3. For ACID tables, the statistics should be associated with the snapshot. This means the statistics collection on target should sync with the write-id on the source since target doesn't generate target ids of its own.

Issue Links

• incorporates: HIVE-21108 Assign writeId for stats update for a converted transactional table

Sub-Tasks

• Replicate column and table level statistics for unpartitioned Hive tables: Closed
• Replicate column statistics for partitions of partitioned table.: Closed
• Support stats replication for ACID tables.: Closed
• Stats replication for materialized views: Open