Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2604

Spark Application hangs on yarn in edge case scenario of executor memory requirement

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0
    • None
    • Spark Core
    • None

    Description

      In yarn environment, let's say :
      MaxAM = Maximum allocatable memory
      ExecMem - Executor's memory

      if (MaxAM > ExecMem && ( MaxAM - ExecMem) > 384m ))
      then Maximum resource validation fails w.r.t executor memory , and application master gets launched, but when resource is allocated and again validated, they are returned and application appears to be hanged.

      Typical use case is to ask for executor memory = maximum allowed memory as per yarn config

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              twinkle Twinkle Sachdeva
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: