[SPARK-36925] Overriding parameters

• Type: Question
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: 3.1.1, 3.1.2
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Abhinav Kumar
• Created: 04/Oct/21 10:54
• Updated: 12/Dec/22 18:10
• Resolved: 05/Oct/21 01:56

Description

Hello team, One quick question.

I read somewhere in the doc - Order of priority

Value set up in spark-submit overrides value set up in code overrides value set up in .conf

Is this correct understanding? If so, we may have problem.

I was trying to set spark.kubernetes.namespace. I had used default in code but wanted to override in submit-submit. Application kept failing. Finally I had to remove from code to make it work.

Same for some other kubernetes setups like PersistentVolume set up.