[SPARK-3655] Support sorting of values in addition to keys (i.e. secondary sort)

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 1.1.0, 1.2.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Koert Kuipers
• Reporter: koert kuipers
• Created: 23/Sep/14 06:50
• Updated: 06/Jun/19 16:57
• Resolved: 06/Jun/19 16:57

Description

Now that spark has a sort based shuffle, can we expect a secondary sort soon? There are some use cases where getting a sorted iterator of values per key is helpful.

Issue Links

• is related to: SPARK-2045 Sort-based shuffle implementation
• is related to: SPARK-10405 Support takeOrdered and topK values per key
• relates to: PIG-4504 Enable Secondary key sort feature in spark mode
• relates to: SPARK-15798 Secondary sort in Dataset/DataFrame
• links to: [Github] Pull Request #3632 (koertkuipers)