Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6783

s3_to_redshift operator's usage is too limited

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: aws
    • Labels:
      None

      Description

      The s3_to_redshift operator has too limited usage, because the table argument is also used in the FROM part.
       
      Given:
      schema='public',
      table='table_name_in_redshift',
      s3_bucket="s3_bucket",
      s3_key="path_to_files",
      redshift_conn_id='redshift_connection',
      aws_conn_id='s3_default'
       
      This generates the following sql:
      COPY public.table_name_in_redshift
      FROM 's3://s3_bucket/path_to_files/table_name_in_redshift'
      with credentials
      'aws_access_key_id='XXXXXXX';aws_secret_access_key='YYYYYYYYYYYY''
      ;
       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ringpeter Peter Ring
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: