[HUDI-4584] SQLConf is not propagated correctly into RDDs

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.12.1
• Labels: pull-request-available
• Story Points: 6
• Epic Link: Hudi Spark Datasource
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 09/Aug/22 22:12
• Updated: 04/Mar/24 23:26
• Resolved: 25/Aug/22 15:38

Description

There were a few reports and slack as well as in GI, related to Spark SQL configs not being respected by DeltaStreamer while working perfectly fine when leveraging DataSource API:

https://github.com/apache/hudi/issues/6278

 

I was able to trace these down to

1. `HoodieSparkUtils.createRDD` instantiating `AvroSerializer` which uses SQLConf that isn't propagated by Spark properly.

Issue Links

• relates to: HUDI-7479 SQL confs don't propagate to spark row writer deltastreamer
• links to: GitHub Pull Request #6351
• links to: GitHub Pull Request #6352