Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6791

AWSBatchOperator crash when "parameter" arguments not passed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: aws, operators
    • Labels:
      None
    • Environment:
      Python 3.7 on Linux

      Description

      Hey folks,

      After upgrading to 1.10.9 we've discovered that using the AWSBatchOperator without passing the parameter argument (e.g., defaults to None) that it causes boto to explode in a gnarly error.

      task2 = AWSBatchOperator(
          job_name="dc-test-alchemy-batch",
          job_definition="slingshot-dc-test-alchemy-batch-jobdef-1",
          job_queue="AwsBatch-batch-queue",
          aws_conn_id="aws_skyscanner",
          dag=(dag),
          overrides={},
          task_id="dc-test-alchemy-batch"
      )

       

      [2020-02-13 10:48:20.298] ERROR:airflow.task.handle_failure:1128 Parameter validation failed:
      Invalid type for parameter parameters, value: None, type: <class 'NoneType'>, valid types: <class 'dict'>
      Traceback (most recent call last):
        File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/awsbatch_operator.py", line 137, in execute
          containerOverrides=self.overrides,
        File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 276, in _api_call
          return self._make_api_call(operation_name, kwargs)
        File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 559, in _make_api_call
          api_params, operation_model, context=request_context)
        File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 607, in _convert_to_request_dict
          api_params, operation_model)
        File "/usr/local/lib/python3.7/site-packages/botocore/validate.py", line 297, in serialize_to_request
          raise ParamValidationError(report=report.generate_report())
      botocore.exceptions.ParamValidationError: Parameter validation failed:
      Invalid type for parameter parameters, value: None, type: <class 'NoneType'>, valid types: <class 'dict'>

      It seems boto is upset anything was passed at all rather than an empty dict. Specifically passing parameters={} solves the issue.

       

      It seems there's precedent for this as array_properties already gets changed from None to {} if not passed. https://github.com/apache/airflow/blob/1.10.9/airflow/contrib/operators/awsbatch_operator.py#L109

      This is with boto3==1.11.16 and botocore==1.14.16 on airflow==1.10.9. I've attached our full frozen requirements. The Airflow version we use is forked but we've only added some extra metric collection — I absolutely promise the changes are tiny and unrelated to this.

      Mostly it seems like this is a bug in boto3 itself but I figured it was worth reporting incase anyone else runs into it (haven't been able to find any other issues).

        Attachments

        1. requirements.txt
          8 kB
          Daniel Callander

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              danielcallander_skyscanner Daniel Callander
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: