Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9857

Batch ALTER_PARTITION events

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 4.1.0
    • Catalog
    • None
    • ghx-label-4

    Description

      When Hive inserts data into partitioned tables, it generates a lot of ALTER_PARTITION (and possibly INSERT_EVENT) in quick succession. Currently, such events are processed one by one by EventsProcessor which is can be slow and can cause EventsProcessor to lag behind. This JIRA proposes to use batching for such ALTER_PARTITION events such that all the successive ALTER_PARTITION events for the same table are batched together into one ALTER_PARTITIONS event and then are processed together to refresh all the partitions from the events. This can significantly speed up the event processing in such cases.

      Attachments

        Issue Links

          Activity

            People

              vihangk1 Vihang Karajgaonkar
              vihangk1 Vihang Karajgaonkar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: