Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19307

SPARK-17387 caused ignorance of conf object passed to SparkContext:

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 2.1.1, 2.2.0
    • PySpark
    • None

    Description

      after patch SPARK-17387 was applied – Sparkconf object is ignored when launching SparkContext programmatically via python from spark-submit:

      https://github.com/apache/spark/blob/master/python/pyspark/context.py#L128:

      in case when we are running python SparkContext(conf=xxx) from spark-submit:
      conf is set, conf._jconf is None ()

      passed as arg conf object is ignored (and used only when we are launching java_gateway).

      how to fix:

      python/pyspark/context.py:132

      python/pyspark/context.py:132
              if conf is not None and conf._jconf is not None:
                  # conf has been initialized in JVM properly, so use conf directly. This represent the
                  # scenario that JVM has been launched before SparkConf is created (e.g. SparkContext is
                  # created and then stopped, and we create a new SparkConf and new SparkContext again)
                  self._conf = conf
              else:
                  self._conf = SparkConf(_jvm=SparkContext._jvm)
      +             if conf:
      +                 for key, value in conf.getAll():
      +                     self._conf.set(key,value)
      +                     print(key,value)
      

      Attachments

        1. SPARK-19307.patch
          0.6 kB
          yuriy_hupalo

        Issue Links

          Activity

            People

              vanzin Marcelo Masiero Vanzin
              yuriy_hupalo yuriy_hupalo
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: