Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-7025

Fix SparkSqlHook.run_query to handle its parameter properly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.10.9
    • Fix Version/s: 1.10.10
    • Component/s: hooks
    • Labels:
      None

      Description

      SparkSqlHook.run_query() has a parameter called cmd, which type is documented as string, but it doesn't work as expected.

      In [1]: from airflow.providers.apache.spark.hooks.spark_sql import SparkSqlHook                                                                                              
      
      In [2]: SparkSqlHook(sql="SELECT 1", master="local[*]", conn_id="spark_default").run_query(cmd="--help")                                                                     
      
      (snip)
      
      [2020-03-09 23:37:35,088] {spark_sql.py:149} INFO - b'  childArgs               [-e SELECT 1 - - h e l p]\n'
      

      The passed argument "--help" is splitted into single characters. This is because the cmd parameter is concatenated to connection_cmd as a list of characters, as follows:

      airflow/providers/apache/spark/hooks/spark_sql.py
       90     def _prepare_command(self, cmd):
       91         """
       92         Construct the spark-sql command to execute. Verbose output is enabled
       93         as default.
       94 
       95         :param cmd: command to append to the spark-sql command
       96         :type cmd: str
       97         :return: full command to be executed
       98         """
       99         connection_cmd = ["spark-sql"]
      
      (snip)
      
      130         connection_cmd += cmd
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                sekikn Kengo Seki
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: