[HUDI-3872] Write tests for async clustering job to run concurrently w/ deltastreamer w/o overlap

• Type: Test
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: clustering, deltastreamer, table-service
• Assignee: Unassigned
• Reporter: sivabalan narayanan
• Created: 13/Apr/22 14:06
• Updated: 28/Mar/24 01:14

Description

We want to test HoodieClusteringJob along w/ deltastreamer continuous job. But to test it reliably, we need to ensure regular writers don't overlap w/ clustering file groups. If not, one of them will fail. So, we need to come up w/ test data generation differently so that regular writers keep writing to new partition in every batch and so clustering can take care of older partitions and there won't be any over-stepping among these.