Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-2738

XCOM's don't work with PythonVirtualenvOperator

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.8.0, 1.9.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When you set provide_context to True in PythonVirtualenvOperator, and try to use xcom_push, Airflow is trying to pickle context - and throws error. __ 

      Traceback (most recent call last):
        File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
          result = task_copy.execute(context=context)
        File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 93, in execute
          return_value = self.execute_callable()
        File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 249, in execute_callable
          self._write_args(input_filename)
        File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 305, in _write_args
          pickle.dump(arg_dict, f)
      TypeError: can't pickle module objects
      

      I tried to debug - and it seems that macrostask_instanceti and conf  in context are guilty in this case.
      Using dill doesn't fix this problem.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Depta Cezary
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: