[HUDI-7527] Include instants outside commits and compaction for generating the latest instant and timeline hash for timeline server requests

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Component/s: table-service, timeline-server
• Story Points: 8
• Assignee: Ethan Guo (this is the old account; please use "yihua")
• Reporter: Ethan Guo (this is the old account; please use "yihua")
• Created: 22/Mar/24 06:21
• Updated: 01/Oct/24 00:31

Description

When the write client sends the request to the timeline server with the latest instant and timeline hash, the latest instant and timeline hash are based on the filtered timeline that does not contain pending log compaction or clustering (replacecommit).  In such a case, consecutive async scheduling of clustering, even with a newly instantiated write client, does not trigger FSV refresh at the timeline server; but FSV is affected by pending clustering and log compaction (e.g., fgIdToPendingLogCompaction, fgIdToReplaceInstants, fgIdToPendingClustering).