[SPARK-14042] Add support for custom coalescers

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Assignee: Nezih Yigitbasi
• Reporter: Nezih Yigitbasi
• Created: 21/Mar/16 20:02
• Updated: 19/Apr/16 21:35
• Resolved: 19/Apr/16 21:35

Description

Per our discussion on the mailing list (please see here) it would be nice to specify a custom coalescing policy as the current coalesce() method only allows the user to specify the number of partitions and we cannot really control much. The need for this feature popped up when I wanted to merge small files by coalescing them by size.

Issue Links

• links to: [Github] Pull Request #11865 (nezihyigitbasi)