Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10926

Sync db/table to latest event id for ddl operations in catalog metastore server

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

      Whenever DDL operations like add/remove partitions, alter table etc are performed from catalog HMS endpoints, we will do the following: 

      1. Acquire lock on db/table 
      2. Perform HMS operation
      3. For db/table in step 1, fetch all events from HMS
      4. Apply those events on the db/table in catalogd cache 
      5. Set the last processed event id for db/table 

      In addition to the above, the following needs to be handled as well: 

      1. Event processor should skip processing an event on a db/table if the db/table is already synced till that event id. 
      2. If last synced event id in table/db is less than the event being processed by event processor, it should set the last synced db/table event id after successfully processing the event. 
      3. Full table refresh should set the last event processed. 

      cc - kishendas vihangk1

      Attachments

        Issue Links

          Activity

            People

              sourabh912 Sourabh Goyal
              sourabh912 Sourabh Goyal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: