[IMPALA-9857] Batch ALTER_PARTITION events

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Catalog
• Epic Color: ghx-label-4
• Assignee: Vihang Karajgaonkar
• Reporter: Vihang Karajgaonkar
• Created: 15/Jun/20 05:52
• Updated: 20/Jun/22 00:45
• Resolved: 02/Dec/21 16:54

Description

When Hive inserts data into partitioned tables, it generates a lot of ALTER_PARTITION (and possibly INSERT_EVENT) in quick succession. Currently, such events are processed one by one by EventsProcessor which is can be slow and can cause EventsProcessor to lag behind. This JIRA proposes to use batching for such ALTER_PARTITION events such that all the successive ALTER_PARTITION events for the same table are batched together into one ALTER_PARTITIONS event and then are processed together to refresh all the partitions from the events. This can significantly speed up the event processing in such cases.

Issue Links

• is part of: IMPALA-7954 Support automatic invalidates using metastore notification events
• is related to: IMPALA-10949 Improve batching logic of events