Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-7117

Honor self.schema in sql_to_gcs as schema to upload

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.10.5, 1.10.6, 1.10.7, 1.10.8, 1.10.9
    • Fix Version/s: None
    • Component/s: operators
    • Labels:
      None
    • Flags:
      Patch

      Description

      In sql_to_gcs, there is a self.schema, which was introduced in pr-3091 to allow user to provide their own schema while loading from sql to Google BigQuery.

      This is very useful when user want custom type mapping. For example, mysql.date is default to map bigquery.TIMESTAMP, but sometimes, people want it to be bigquery.DATE.

      However, during mysql_to_gcs re-organization, we lost that ability. (Only use self.schema for pre-processing, while data type is no longer configurable). 

      As a conclusion, we need to honor self.schema (if provided) to override default schema, like we did before. 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                whynick1 Hongyi Wang
                Reporter:
                whynick1 Hongyi Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: