[SPARK-25880] user set some hadoop configurations can not work

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.2.0
• Component/s: Spark Core, SQL
• Assignee: Unassigned
• Reporter: guojh
• Created: 30/Oct/18 06:50
• Updated: 06/Nov/19 15:06
• Resolved: 06/Nov/19 15:06

Description

When user set some hadoop configuration in spark-defaults.conf, for instance: 

spark.hadoop.mapreduce.input.fileinputformat.split.maxsize   100000

and then user use the spark-sql and use set command to overwrite this configuration, but it can not cover the value which set in the file of spark-defaults.conf. 

 

Issue Links

• links to: [Github] Pull Request #22887 (gjhkael)
• links to: [Github] Pull Request #22887 (gjhkael)
• links to: GitHub Pull Request #22887