[SPARK-4824] Join should use `Iterator` rather than `Iterable`

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Shixiong Zhu
• Created: 11/Dec/14 02:56
• Updated: 11/Dec/14 03:17
• Resolved: 11/Dec/14 03:17

Description

In Scala, `map` and `flatMap` of `Iterable` will copy the contents of `Iterable` to a new `Seq`. Such as,

  val iterable = Seq(1, 2, 3).map(v => {
    println(v)
    v
  })
  println("Iterable map done")

  val iterator = Seq(1, 2, 3).iterator.map(v => {
    println(v)
    v
  })
  println("Iterator map done")

outputed

1
2
3
Iterable map done
Iterator map done

So we should use 'iterator' to reduce memory consumed by join.

Found by johannes.simon

Issue Links

• duplicates: SPARK-4818 Join operation should use iterator/lazy evaluation
• links to: [Github] Pull Request #3671 (zsxwing)