[SPARK-19307] SPARK-17387 caused ignorance of conf object passed to SparkContext:

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: PySpark
• Target Version/s: 2.1.1, 2.2.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: yuriy_hupalo
• Created: 20/Jan/17 07:57
• Updated: 29/Aug/17 22:48
• Resolved: 25/Jan/17 20:08

Description

after patch SPARK-17387 was applied – Sparkconf object is ignored when launching SparkContext programmatically via python from spark-submit:

https://github.com/apache/spark/blob/master/python/pyspark/context.py#L128:

in case when we are running python SparkContext(conf=xxx) from spark-submit:
conf is set, conf._jconf is None ()

passed as arg conf object is ignored (and used only when we are launching java_gateway).

how to fix:

python/pyspark/context.py:132

python/pyspark/context.py:132

        if conf is not None and conf._jconf is not None:
            # conf has been initialized in JVM properly, so use conf directly. This represent the
            # scenario that JVM has been launched before SparkConf is created (e.g. SparkContext is
            # created and then stopped, and we create a new SparkConf and new SparkContext again)
            self._conf = conf
        else:
            self._conf = SparkConf(_jvm=SparkContext._jvm)
+             if conf:
+                 for key, value in conf.getAll():
+                     self._conf.set(key,value)
+                     print(key,value)

Issue Links

• is duplicated by: SPARK-19369 SparkConf not getting properly initialized in PySpark 2.1.0
• is duplicated by: SPARK-20362 spark submit not considering user defined Configs (Pyspark)
• is related to: SPARK-19301 SparkContext is ignoring SparkConf when _jvm is not initialized on spark-submit
• relates to: SPARK-17387 Creating SparkContext() from python without spark-submit ignores user conf
• links to: [Github] Pull Request #16682 (vanzin)