Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26539

Remove spark.memory.useLegacyMode memory settings + StaticMemoryManager

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
    • Target Version/s:
    • Docs Text:
      Hide
      The legacy memory manager, enabled by spark.memory.useLegacyMode, has been removed in Spark 3. It is effectively always false, its default since Spark 1.6. Associated settings spark.shuffle.memoryFraction, spark.storage.memoryFraction,and spark.storage.unrollFraction have been removed as well.
      Show
      The legacy memory manager, enabled by spark.memory.useLegacyMode, has been removed in Spark 3. It is effectively always false, its default since Spark 1.6. Associated settings spark.shuffle.memoryFraction, spark.storage.memoryFraction,and spark.storage.unrollFraction have been removed as well.

      Description

      The old memory manager was superseded by UnifiedMemoryManager in Spark 1.6, and has been the default since. I think we could remove it to simplify the code a little, but more importantly to reduce the variety of memory settings users are confronted with when using Spark.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                srowen Sean R. Owen
                Reporter:
                srowen Sean R. Owen
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: