[SPARK-3356] Document when RDD elements' ordering within partitions is nondeterministic

• Type: Documentation
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Documentation
• Assignee: Sean R. Owen
• Reporter: Matei Alexandru Zaharia
• Created: 02/Sep/14 23:01
• Updated: 31/Mar/15 00:56
• Resolved: 30/Sep/14 18:16

Description

As reported in SPARK-3098 for example, for users using zipWithIndex, zipWithUniqueId, etc, (and maybe even things like mapPartitions) it's confusing that the order of elements in each partition after a shuffle operation is nondeterministic (unless the operation was sortByKey). We should explain this in the docs for the zip and partition-wise operations.

Another subtle issue is that the order of values for each key in groupBy / join / etc can be nondeterministic – we need to explain that too.

Issue Links

• is related to: SPARK-6617 Word2Vec is nondeterministic
• links to: [Github] Pull Request #2508 (srowen)