[PHOENIX-5197] Use extraOptions to set the configuration for Spark Workers

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: connectors-6.0.0
• Fix Version/s: connectors-6.0.0
• Assignee: Chinmay Kulkarni
• Reporter: Chinmay Kulkarni
• Created: 14/Mar/19 21:06
• Updated: 05/Oct/21 08:59
• Resolved: 09/Apr/19 18:44

Description

We need a standardized way to set configurations for Spark driver and workers when reading and writing data.

Quoting from my offline discussion with [e-mail]:

"if we use phoenixTableAsRDD to create the rdd then we can pass in a config
if we use the standard df.write.format() we can't pass in a conf
maybe the best was to do this is to put config in the option map when we call read/write .format and then set those in the config"

Issue Links

• causes: PHOENIX-6566 shaded phoenix connectors include restricitve log4j config files
• links to: GitHub Pull Request #2