[SPARK-12598] Bug in setMinPartitions function of StreamFileInputFormat

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.2
• Fix Version/s: 1.6.1, 2.0.0
• Component/s: Spark Core
• Flags: Patch
• Assignee: Darek Blasiak
• Reporter: Darek Blasiak
• Created: 01/Jan/16 19:37
• Updated: 07/Jan/16 21:19
• Resolved: 07/Jan/16 21:18

Description

The maxSplitSize should be computed as:

val maxSplitSize = Math.ceil(totalLen * 1.0 / minPartitions).toLong

Issue Links

• links to: [Github] Pull Request #10546 (datafarmer)