[SPARK-16694] Use for/foreach rather than map for Unit expressions whose side effects are required

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: DStreams, Examples, MLlib, Spark Core, SQL
• Target Version/s: 2.1.0
• Assignee: Sean R. Owen
• Reporter: Sean R. Owen
• Created: 24/Jul/16 07:27
• Updated: 10/Jan/21 05:21
• Resolved: 30/Jul/16 11:42

Description

map is misused in many places where foreach is intended. This caused a bug in https://issues.apache.org/jira/browse/SPARK-16664 and might be a latent bug elsewhere; it's also easy to find with IJ inspections. Worth patching up.

To illustrate the general problem, map happens to work in Scala where the collection isn't lazy, but will fail to execute the code when it is. map also causes a collection of Unit to be created pointlessly.

scala> val foo = Seq(1,2,3)
foo: Seq[Int] = List(1, 2, 3)

scala> foo.map(println)
1
2
3
res0: Seq[Unit] = List((), (), ())

scala> foo.view.map(println)
res1: scala.collection.SeqView[Unit,Seq[_]] = SeqViewM(...)

scala> foo.view.foreach(println)
1
2
3

Issue Links

• is cloned by: SPARK-34059 Use for/foreach rather than map to make sure execute it eagerly
• links to: [Github] Pull Request #14332 (srowen)