[PHOENIX-7006] Configure maxLookbackAge at table level

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.3.0
• Assignee: Sanjeet Malhotra
• Reporter: Viraj Jasani
• Created: 27/Jul/23 02:16
• Updated: 14/Mar/24 16:21
• Resolved: 11/Mar/24 20:09

Description

Phoenix max lookback age feature preserves live or deleted row versions that are only visible through the max lookback window, it does not preserve any unwanted row versions that should not be visible through the max lookback window. More details on the max lookback redesign: PHOENIX-6888

As of today, maxlookback age is only configurable at the cluster level (config key: phoenix.max.lookback.age.seconds), meaning the same value is used by all tables. This does not allow individual table level compaction scanner to be able to retain data based on the table level maxlookback age. Setting max lookback age at the table level can serve multiple purposes e.g. change-data-capture (PHOENIX-7001) for individual table should have it's own latest data retention period.

The purpose of this Jira is to allow maxlookback age as a table level property:

• New column in SYSTEM.CATALOG to preserve table level maxlookback age
• PTable object to read the value of maxlookback from SYSTEM.CATALOG
• Allow CREATE/ALTER TABLE DDLs to provide maxlookback attribute
• CompactionScanner should use table level maxlookbackAge, if available, else use cluster level config

Issue Links

• is depended upon by: PHOENIX-7001 Change Data Capture leveraging Max Lookback and Uncovered Indexes
• links to: Design doc
• links to: GitHub Pull Request #1750
• links to: GitHub Pull Request #1751