Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-7024

Add the verbose parameter support to SparkSqlOperator

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.10.9
    • Fix Version/s: 1.10.10
    • Component/s: operators
    • Labels:
      None

      Description

      The docstring for SparkSqlOperator says that it supports a verbose parameter, but actually it doesn't.

      airflow/providers/apache/spark/operators/spark_sql.py
      class SparkSqlOperator(BaseOperator):
          """
      
          (snip)
      
          :param verbose: Whether to pass the verbose flag to spark-sql
          :type verbose: bool
          :param yarn_queue: The YARN queue to submit to (Default: "default")
          :type yarn_queue: str
          """
      
          (snip)
      
          @apply_defaults
          def __init__(self,
                       sql,
                       conf=None,
                       conn_id='spark_sql_default',
                       total_executor_cores=None,
                       executor_cores=None,
                       executor_memory=None,
                       keytab=None,
                       principal=None,
                       master='yarn',
                       name='default-name',
                       num_executors=None,
                       yarn_queue='default',
                       *args,
                       **kwargs):
              super().__init__(*args, **kwargs)
              self._sql = sql
              self._conf = conf
              self._conn_id = conn_id
              self._total_executor_cores = total_executor_cores
              self._executor_cores = executor_cores
              self._executor_memory = executor_memory
              self._keytab = keytab
              self._principal = principal
              self._master = master
              self._name = name
              self._num_executors = num_executors
              self._yarn_queue = yarn_queue
              self._hook = None
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                sekikn Kengo Seki
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: