For ensuring the transactional tables are consistent, we currently take whole table refreshing even a change is just for a partition only. That is too expensive and possibly make event processing has a longer delay.
To enable fine-grained table refreshing, there are three main changes in this proposal.
- maintain validWriteIdList in Catalogd for transactional tables. We will track write id changes by AllocWriteIdEvents, CommitTxnEvents, and AbortTxnEvents.
- trigger partition level refreshing for addPartitionEvents, dropPartitionEvents, and AlterPartitionEvents.
- Introduce a config incremental_refresh_acid, which can switch on/off the fine-grained table refreshing