[HUDI-4070] Better Spark SQL default configs

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.12.0
• Epic Link: Config clean up
• Assignee: Sagar Sumit
• Reporter: Sagar Sumit
• Created: 09/May/22 16:31
• Updated: 29/Jul/22 07:02
• Resolved: 29/Jul/22 07:02

Description

Default configs should be:

1. Optimized for insert/bulk_insert e.g. by default if we have NONE sort mode then it's as good as parquet writes with some additional work for meta columns. An extension of this is to keep a map of minimal optimized configs per operation type. This is partly related to better performant configs HUDI-2151
2. Make reasonable assumptions, e.g. for index type, bloom filter does not rely on any external system, so it can be a better default candidate than let's say HBase index.
3. Scout all configs with noDefaultValue and assign a default if necessary.
4. Keep spark-sql and spark datasource config keys same as much as possible, otherwise it's difficult operationally for the user. Rename/reuse existing datasource keys that are meant for same purpose. This is related to HUDI-4071 as well.

Issue Links

• is related to: HUDI-2151 Make performant out-of-box configs
• relates to: HUDI-4071 Better Spark Datasource default configs