Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12447

Only update AM's internal state when executor is successfully launched by NM

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 2.0.0
    • Component/s: YARN
    • Labels:
      None

      Description

      Currently YarnAllocator will update its managed states like numExecutorsRunning after container is allocated but before executor are successfully launched.

      This happened when Spark configuration is wrong (like spark_shuffle aux-service is not configured in NM occasionally), which makes executor fail to launch, or NM lost when NMClient is communicated.

      In the current implementation, state will also be updated even executor is failed to launch, this will lead to incorrect state of AM. Also lingering container will only be release after timeout, this will introduce resource waste.

      So here we should update the states only after executor is correctly launched, otherwise we should release container ASAP to make it fail fast and retry.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jerryshao Saisai Shao
                Reporter:
                jerryshao Saisai Shao
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: