[SPARK-1808] bin/pyspark does not load default configuration properties

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.0
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 12/May/14 19:14
• Updated: 05/Nov/14 10:45
• Resolved: 17/May/14 05:35

Description

... because it doesn't go through spark-submit. Either we make it go through spark-submit (hard), or we extract the load default configurations logic and set them for the JVM that launches the py4j GatewayServer (easier).

Right now, the only way to set config values for bin/pyspark is to do it through SPARK_JAVA_OPTS in spark-env.sh, which is supposedly deprecated.