[SPARK-15382] monotonicallyIncreasingId doesn't work when data is upsampled

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.1
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: SQL
• Assignee: Takeshi Yamamuro
• Reporter: Mateusz Buśkiewicz
• Created: 18/May/16 12:23
• Updated: 12/Dec/22 18:10
• Resolved: 19/Aug/16 17:12

Description

Assigned ids are not unique

from pyspark.sql import Row
from pyspark.sql.functions import monotonicallyIncreasingId

hiveContext.createDataFrame([Row(a=1), Row(a=2)]).sample(True, 10.0).withColumn('id', monotonicallyIncreasingId()).collect()

Output:

[Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792)]

Issue Links

• duplicates: SPARK-16686 Dataset.sample with seed: result seems to depend on downstream usage
• links to: [Github] Pull Request #14181 (maropu)
• links to: [Github] Pull Request #14800 (maropu)