Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7271 Redesign shuffle interface for binary processing
  3. SPARK-10710

Remove ability to set spark.shuffle.spill=false and spark.sql.planner.externalSort=false

    XMLWordPrintableJSON

    Details

      Description

      The spark.shuffle.spill=false configuration doesn't make much sense nowadays: I think that this configuration was only added as an escape-hatch to guard against bugs when spilling was first added. Similarly, setting spark.sql.planner.externalSort=false doesn't make sense in newer releases: many new implementations, such as Tungsten, completely ignore this flag, so it's not applied in a consistent way.

      In order to reduce complexity, I think that we should remove the ability to disable spilling. Note that the tungsten-shuffle manager already does not respect this setting, so removing this configuration is a blocker to being able to unify the two sort-shuffle implementations.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joshrosen Josh Rosen
                Reporter:
                joshrosen Josh Rosen
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: