Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-3069

Decode output of S3 file transform operator

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.10.0
    • Fix Version/s: 1.10.1
    • Component/s: aws
    • Labels:
      None

      Description

      Current behaviour

      S3FileTransformOperator logs stdout of the underlying process as such:

      [2018-09-15 23:17:13,850] {{s3_file_transform_operator.py:122}} INFO - Transform script stdout b'Copying /tmp/tmpd5rjo8g0 to /tmp/tmpd3vkhzte\nDone\n'
      

      While stderr is omitted entirely, unless exit code is not 0 (in this case it's included in the exception message only).

      Proposed behaviour

      1. Both streams are logged, regardless of the underlying process outcome (i.e. success or failure).
      2. Stream output is decoded before logging (e.g. \n is replaced with an actual new line).
      3. If transform_script fails, the exception message contains return code of the process.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sbilinski Szymon Bilinski
                Reporter:
                sbilinski Szymon Bilinski
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: