Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-3128

SparkSqlOperator prints blank info endlessly after SparkContext stopped

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.10.0
    • Component/s: contrib
    • Labels:
      None

      Description

      environment: hdp 2.6.2 with spark 2 

      spark_sql_test = SparkSqlOperator(task_id='spark_sql_test',dag=dag,sql='select * from default.test limit 2;',name='airflow_spark_sql_test',executor_cores=2,num_executors=2)

      Log:

      [2018-09-28 17:14:08,754] {spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!\n'
      [2018-09-28 17:14:08,778] {spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO SparkContext: Successfully stopped SparkContext\n'
      [2018-09-28 17:14:08,780] {spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO ShutdownHookManager: Shutdown hook called\n'
      [2018-09-28 17:14:08,780] {spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-ba3b2fef-ae75-4e65-afb2-1b029b2c5c23\n'
      [2018-09-28 17:14:08,781] {spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-ba397c83-4003-4006-8ce9-179cc2df2dd7\n'
      [2018-09-28 17:14:09,336] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,336] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,336] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,336] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,337] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,337] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,337] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,337] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,337] {spark_sql_hook.py:148} INFO - b''
      [2018-09-28 17:14:09,337] {spark_sql_hook.py:148} INFO - b''

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Forest Lin Forestlzj
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: