Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28577

Ensure executorMemoryHead requested value not less than MEMORY_OFFHEAP_SIZE when MEMORY_OFFHEAP_ENABLED is true

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core, YARN
    • Labels:
    • Docs Text:
      On YARN, The off heap memory size is now separately included for the user in the container size it requests from YARN. Previously you had to add that into the overhead memory you requested, that is no longer needed.

      Description

      If MEMORY_OFFHEAP_ENABLED is true, we should ensure executorOverheadMemory not less than MEMORY_OFFHEAP_SIZE, otherwise the memory resource requested for executor may be not enough.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                LuciferYang Yang Jie
                Reporter:
                LuciferYang Yang Jie
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: