Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-5517

SparkSubmitOperator: spark-binary parameter no longer taken from connection extra

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.4, 1.10.5
    • Fix Version/s: 1.10.7
    • Component/s: contrib
    • Labels:
      None

      Description

      We have an extra parameters in spark connection:

      {"deploy-mode": "cluster", "spark-binary": "spark2-submit"}
      

      After upgrade to 1.10.5 from 1.10.3 parameter 'spark-binary' in extra is no longer take effect.
      Broken after this commit, I think

      Workaround: call SparkSubmitOperator with spark_binary=None argument

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Boiler Alexander Kazarin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: