[SPARK-26143] Shuffle shuffle default storage level

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Shuffle, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Avi minsky
• Created: 21/Nov/18 19:32
• Updated: 12/Dec/22 18:10
• Resolved: 25/May/21 01:39

Description

Currently developer can set storage level explicitly only on persist command but shuffling can occur in many cases (group by, join, etc..) why can we set a default persist type, this can be helpful in many cases (also will automatically allow replication of shuffle blocks if set to memory_only_2, for example)