Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-3503

GoogleCloudStorageHook delete return success when nothing was done

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Cannot Reproduce
    • 1.10.1
    • None
    • gcp
    • Patch, Important

    Description

      I'm loading files to BigQuery from Storage using:

       

      gcs_export_uri = BQ_TABLE_NAME + '/' + EXEC_TIMESTAMP_PATH + '/*' gcs_to_bigquery_op = GoogleCloudStorageToBigQueryOperator( dag=dag, task_id='load_products_to_BigQuery', bucket=GCS_BUCKET_ID, destination_project_dataset_table=table_name_template, source_format='NEWLINE_DELIMITED_JSON', source_objects=[gcs_export_uri], src_fmt_configs={'ignoreUnknownValues': True}, create_disposition='CREATE_IF_NEEDED', write_disposition='WRITE_TRUNCATE', skip_leading_rows = 1, google_cloud_storage_conn_id=CONNECTION_ID, bigquery_conn_id=CONNECTION_ID)

       

      After that I want to delete the files so I do:

      def delete_folder():
          """
          Delete files Google cloud storage
          """
          hook = GoogleCloudStorageHook(
                  google_cloud_storage_conn_id=CONNECTION_ID)

          hook.delete(
              bucket=GCS_BUCKET_ID,
              object=gcs_export_uri)

       

       

      This runs with PythonOperator.

      The task marked as Success even though nothing was deleted.

      Log:

      [2018-12-12 11:31:29,247] {base_task_runner.py:98} INFO - Subtask: [2018-12-12 11:31:29,247] {transport.py:151} INFO - Attempting refresh to obtain initial access_token [2018-12-12 11:31:29,249] {base_task_runner.py:98} INFO - Subtask: [2018-12-12 11:31:29,249] {client.py:795} INFO - Refreshing access_token [2018-12-12 11:31:29,584] {base_task_runner.py:98} INFO - Subtask: [2018-12-12 11:31:29,583] {python_operator.py:90} INFO - Done. Returned value was: None

       

       

      I expect the function to fail and return something like "file was not found" if there is nothing to delete Or let the user decide with specific flag if he wants the function to fail or success if files were not found.

       

      Attachments

        Activity

          People

            yohei Yohei Shimomae
            lot lot
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: