Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-4494

Unwanted kwarg shell=True in DaskExecutor

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersLinkUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.10.3
    • Fix Version/s: 1.10.4
    • Component/s: executors
    • Labels:
      None

      Description

      This line:

      https://github.com/apache/airflow/blob/master/airflow/executors/dask_executor.py#L68

      If Popen is with shell=True and args is a list, the first item of list specifies the command string, the other items are the arguments of /bin/sh on POSIX.

      So the command airflow run ... will raise an error from dask-worker:

      usage: airflow [-h]
      {backfill,list_dag_runs,list_tasks,clear,pause,unpause,trigger_dag,delete_dag,pool,variables,kerberos,render,run,initdb,list_dags,dag_state,task_failed_deps,task_state,serve_logs,test,webserver,resetdb,upgradedb,scheduler,worker,flower,version,connections,create_user,delete_user,list_users,sync_perm,next_execution,rotate_fernet_key}
      ...
      airflow: error: the following arguments are required: subcommand
      distributed.worker - WARNING - Compute Failed
      Function: airflow_run
      args: ()
      kwargs: {}
      Exception: CalledProcessError(2, ['airflow', 'run', 'tutorial', 'print_date', '2019-05-08T00:00:00+00:00', '--local', '-sd', '/Users/<me>/projects/dask_airflow/.venv/lib/python3.6/site-packages/airflow/example_dags/tutorial.py'])

       

        Attachments

          Activity

            People

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment