Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6994

SparkSubmitOperator re launches spark driver even when original driver still running

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.8, 1.10.9
    • Fix Version/s: 2.0.0
    • Component/s: scheduler
    • Labels:
      None

      Description

      https://issues.apache.org/jira/browse/AIRFLOW-6229 introduced a bug

      Due to temporary network blip in connection to spark the state goes to unknown (as no tags found in curl response) and forces retry

      fix in spark_submit_hook.py:

        def _process_spark_status_log(self, itr):
              """
              parses the logs of the spark driver status query process
      
              :param itr: An iterator which iterates over the input of the subprocess
              """
              response_found = False
              driver_found = False
              # Consume the iterator
              for line in itr:
                  line = line.strip()
      
                  if "submissionId" in line:
                      response_found = True
                  
                  # Check if the log line is about the driver status and extract the status.
                  if "driverState" in line:
                      self._driver_status = line.split(' : ')[1] \
                          .replace(',', '').replace('\"', '').strip()
                      driver_found = True
      
                  self.log.debug("spark driver status log: {}".format(line))
      
              if response_found and not driver_found:
                  self._driver_status = "UNKNOWN"
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                toopt4 t oo
                Reporter:
                toopt4 t oo
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: