[PHOENIX-6092] Optionally queue DDL requests issued while metadata upgrade is in progress and replay on upgrade failure

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 5.0.0, 4.15.0
• Fix Version/s: 4.17.0, 4.16.2
• Assignee: Unassigned
• Reporter: Chinmay Kulkarni
• Created: 20/Aug/20 21:27
• Updated: 21/May/21 16:26

Description

Currently, if a metadata upgrade is in progress (either triggered by an explicit "EXECUTE UPGRADE" command or by a new client with autoUpgrade enabled), in-flight DDLs will generally go through and work as expected.

However, if the upgrade happens to fail, we restore the snapshot of SYSTEM.CATALOG (and with PHOENIX-6086 even other SYSTEM tables) to represent its state before the upgrade started. Due to this, any DDLs issued after the upgrade began are lost.

There are upgrade steps that need to iterate over each table/index/view in the cluster and multiple steps that need full table scans on SYSTEM.CATALOG and so this time window where we could potentially lose client DDLs is not negligible (could be to the order of minutes).

This Jira is to discuss ways to tackle this problem. Perhaps we can introduce a configuration which when enabled could use some sort of write-ahead log to store DDLs issued while the upgrade is in progress and replay those DDLs in case we need to restore SYSTEM tables from their snapshot.