Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2604

Spark Application hangs on yarn in edge case scenario of executor memory requirement

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      In yarn environment, let's say :
      MaxAM = Maximum allocatable memory
      ExecMem - Executor's memory

      if (MaxAM > ExecMem && ( MaxAM - ExecMem) > 384m ))
      then Maximum resource validation fails w.r.t executor memory , and application master gets launched, but when resource is allocated and again validated, they are returned and application appears to be hanged.

      Typical use case is to ask for executor memory = maximum allowed memory as per yarn config

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                twinkle Twinkle Sachdeva
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: