[HUDI-6232] Support skip archive for glue sync

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.14.0
• Component/s: meta-sync
• Labels: pull-request-available
• Assignee: kazdy
• Reporter: kazdy
• Created: 17/May/23 21:07
• Updated: 24/May/23 18:21
• Resolved: 24/May/23 18:21

Description

Glue has soft limit of 100k table versions. With every table update new version is created. In streaming scenarios we can hit 100k table versions quickly.
Users should be able to skip table archival when using glue meta sync. 
It's been asked many times in Hudi Slack, how to reduce no of glue table versions created. There is conditional hive sync, but it does not solve the problem fully. With skip_table_archive config, users can reduce no. of table versions created even more.

Issue Links

• links to: GitHub Pull Request #8744
• links to: PR 8744