Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10949

Improve batching logic of events

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Catalog
    • None

    Description

      This is a followup based on the review comment https://gerrit.cloudera.org/#/c/17848/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1641

      Current approach of batching batches together the events from a single operation so that self-event check is done per-batch. However, it looks like there is a considerable scope of improving the batching logic by clubbing together accross the various sources of the events on a table when IMPALA-10926 is merged. After IMPALA-10926 each table will track the last_synced_event and then the events processor can simply ignore a event which <= the last_synced_event. This simplification of self-events logic will enable easier batching for all the events of a type on a table.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              vihangk1 Vihang Karajgaonkar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: