[HUDI-7523] Add HOODIE_SPARK_DATASOURCE_OPTIONS to be used in HoodieIncrSource

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.15.0, 1.0.0
• Component/s: deltastreamer
• Labels: pull-request-available
• Story Points: 4
• Assignee: Vinish Reddy
• Reporter: Vinish Reddy
• Created: 21/Mar/24 06:43
• Updated: 15/May/24 14:20
• Resolved: 15/May/24 14:20

Description

Add a new config HOODIE_SPARK_DATASOURCE_OPTIONS which is used by the spark dataframe reader for HoodieIncrSource, options like using metadataTable, dataSkipping present inDataSourceOptions.scala can be passed for efficient pruning of files. 

https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala 

Issue Links

• links to: GitHub Pull Request #10900