[PHOENIX-7382] Eliminating index building and treating max lookback as TTL for CDC Index

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.3.0
• Assignee: Kadir Ozdemir
• Reporter: Kadir Ozdemir
• Created: 13/Aug/24 19:10
• Updated: 26/Sep/24 18:30
• Resolved: 26/Sep/24 18:30

Description

PHOENIX-7001 introduced a Change Data Capture (CDC) feature by leveraging Phoenix Max Lookback and Uncovered Global Index on PHOENIX_ROW_TIMESTAMP(). The max lookback feature retains recent changes to a table and the uncovered index allows efficient retrieval of the changes in the order of their arrival.

Since the changes are retained only within the max lookback window, a CDC index does not need to have rows beyond the max lookback window. This means we can treat the max lookback age as the TTL for CDC indexes. 

When a CDC feature is enabled on a table, a CDC index is created with the building state, and like any other index, it is built from the data table. This means for every mutation on the data table, an index row mutation is built. However, the mutations beyond max lookback are not required for CDC. Since index built is an expensive background operation (it can take days for very large tables), eliminating index built is desirable when the CDC is enabled on a table. This means that the CDC feature will track changes that happen after it is enabled, which is the expected behavior. 

The code changes for this improvement will be for

• Creating a CDC index with the active state 
• Treating the max lookback age as the TTL for CDC indexes during compaction, and index verification, repair and rebuild.

Issue Links

• links to: GitHub Pull Request #1964