[SPARK-32051] Dataset.foreachPartition returns object

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Frank Oosterhuis
• Created: 22/Jun/20 11:53
• Updated: 27/Jun/20 15:40

Description

I'm trying to map values from the Dataset[Row], but since 3.0.0 this fails.

In 3.0.0 I'm dealing with an error: "Error:(28, 38) value map is not a member of Object"

 

This is the simplest code that works in 2.4.x, but fails in 3.0.0:

spark.range(100)
  .repartition(10)
  .foreachPartition(part => println(part.toList))