[HUDI-1045] Support updates during clustering

• Type: Task
• Status: In Progress
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Component/s: clustering, table-service
• Epic Link: Non-blocking concurrency control
• Assignee: Vinoth Chandar
• Reporter: leesf
• Created: 24/Jun/20 02:01
• Updated: 21/Nov/24 21:09

Description

We need to allow a writer w writing to file groups f1, f2, f3, concurrently while a clustering service C  reclusters them into  f4, f5. 

Goals

• Writes can be either updates, deletes or inserts. 
• Either clustering C or the writer W can finish first
• Both W and C need to be able to complete their actions without much redoing of work. 
• The number of output file groups for C can be higher or lower than input file groups. 
• Need to work across and be oblivious to whether the writers are operating in OCC or NBCC modes
• Needs to interplay well with cleaning and compaction services.

Non-goals 

• Strictly the sort order achieved by clustering, in face of updates (e.g updates change clustering field values, causing output clustering file groups to be not fully sorted by those fields)