[SPARK-3979] Yarn backend's default file replication should match HDFS's default one

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: YARN
• Target Version/s: 1.2.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 16/Oct/14 19:59
• Updated: 17/Oct/14 18:45
• Resolved: 17/Oct/14 18:45

Description

This code in ClientBase.scala sets the replication used for files uploaded to HDFS:

    val replication = sparkConf.getInt("spark.yarn.submit.file.replication", 3).toShort

Instead of a hardcoded "3" (which is the default value for HDFS), it should be using the default value from the HDFS conf ("dfs.replication").

Issue Links

• links to: [Github] Pull Request #2831 (vanzin)