[SPARK-16961] Utils.randomizeInPlace does not shuffle arrays uniformly

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: Spark Core
• Assignee: Nicholas
• Reporter: Nicholas
• Created: 09/Aug/16 00:10
• Updated: 20/Aug/16 08:05
• Resolved: 19/Aug/16 09:13

Description

The Utils.randomizeInPlace method, which is meant to uniformly shuffle the elements on an input array, never shuffles elements to their starting position. That is, every permutation of the input array is equally likely to be returned, except for any permutation in which any element is in the same position where it began. These permutations are never output.
This is because line 827 of Utils.scala should be
val j = rand.nextInt(i + 1)
instead of
val j = rand.nextInt( i )

Issue Links

• links to: [Github] Pull Request #14551 (nicklavers)
• links to: [Github] Pull Request #14730 (yanboliang)