[SPARK-29163] Provide a mixin to simplify HadoopConf access patterns in DataSource V2

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Holden Karau
• Reporter: Holden Karau
• Created: 19/Sep/19 00:17
• Updated: 16/Mar/20 22:53

Description

Since many data sources need the hadoop config we should make an easy way for them to get access to it with minimal overhead (e.g. broadcasting + mixin).

 

TODO after SPARK-29158. Also look at DSV1 and see if there were any interesting hacks we did before to make this fast.

Issue Links

• is blocked by: SPARK-29158 Expose SerializableConfiguration for DSv2
• links to: GitHub Pull Request #26005