[SPARK-4229] Create hadoop configuration in a consistent way

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Auto Closed
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Cody Koeninger
• Created: 04/Nov/14 22:33
• Updated: 06/Jun/19 13:57
• Resolved: 06/Jun/19 13:57

Description

Some places use SparkHadoopUtil.get.conf, some create a new hadoop config. Prefer SparkHadoopUtil so that spark.hadoop.* properties are pulled in.

http://apache-spark-developers-list.1001551.n3.nabble.com/Hadoop-configuration-for-checkpointing-td9084.html

Issue Links

• is related to: SPARK-4629 Spark SQL uses Hadoop Configuration in a thread-unsafe manner when writing Parquet files
• links to: [Github] Pull Request #3543 (koeninger)
• links to: [Github] Pull Request #7772 (koeninger)

Sub-Tasks

• Consistent hadoop config for streaming: Resolved
• Consistent hadoop config for SQL: Resolved
• Consistent hadoop config for core: Resolved
• Consistent hadoop config for external/*: Resolved