Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19307

SPARK-17387 caused ignorance of conf object passed to SparkContext:

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.1, 2.2.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      after patch SPARK-17387 was applied – Sparkconf object is ignored when launching SparkContext programmatically via python from spark-submit:

      https://github.com/apache/spark/blob/master/python/pyspark/context.py#L128:

      in case when we are running python SparkContext(conf=xxx) from spark-submit:
      conf is set, conf._jconf is None ()

      passed as arg conf object is ignored (and used only when we are launching java_gateway).

      how to fix:

      python/pyspark/context.py:132

      python/pyspark/context.py:132
              if conf is not None and conf._jconf is not None:
                  # conf has been initialized in JVM properly, so use conf directly. This represent the
                  # scenario that JVM has been launched before SparkConf is created (e.g. SparkContext is
                  # created and then stopped, and we create a new SparkConf and new SparkContext again)
                  self._conf = conf
              else:
                  self._conf = SparkConf(_jvm=SparkContext._jvm)
      +             if conf:
      +                 for key, value in conf.getAll():
      +                     self._conf.set(key,value)
      +                     print(key,value)
      

        Attachments

        1. SPARK-19307.patch
          0.6 kB
          yuriy_hupalo

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                yuriy_hupalo yuriy_hupalo
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: