[HUDI-2151] Make performant out-of-box configs

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Component/s: code-quality, writer-core
• Labels: pull-request-available
• Story Points: 2
• Epic Link: Performance Improvements
• Assignee: sivabalan narayanan
• Reporter: Vinoth Chandar
• Created: 08/Jul/21 21:00
• Updated: 28/Mar/24 00:42
• Resolved: 28/Mar/24 00:42

Description

We have quite a few configs which deliver better performance or usability, but guarded by flags. 

 This is to identify them, change them, test (functionally, perf) and make them default

 

Need to ensure we also capture all the backwards compatibility issues that can arise

Issue Links

• is a parent of: HUDI-2928 Evaluate rebasing Hudi's default compression from Gzip to Zstd
• is a parent of: HUDI-2276 Enable Metadata Table by default for both writers and readers
• relates to: HUDI-4105 Identify out of the box performance config flips for spark-ds
• relates to: HUDI-4106 Identify out of the box default performance flips for spark-sql
• relates to: HUDI-4070 Better Spark SQL default configs
• relates to: HUDI-4071 Better Spark Datasource default configs
• links to: GitHub Pull Request #3383
• links to: GitHub Pull Request #3452
• links to: GitHub Pull Request #3938
• links to: GitHub Pull Request #3948
• links to: GitHub Pull Request #3950
• links to: GitHub Pull Request #3967