[SPARK-12061] Persist for Map/filter with Lambda Functions don't always read from Cache

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.0, 2.1.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Xiao Li
• Created: 01/Dec/15 00:16
• Updated: 21/May/19 04:15
• Resolved: 21/May/19 04:15

Description

So far, the existing caching mechanisms do not work on dataset operations when using map/filter with lambda functions. For example,

  test("persist and then map/filter with lambda functions") {
    val f = (i: Int) => i + 1

    val ds = Seq(1, 2, 3).toDS()
    val mapped = ds.map(f)
    mapped.cache()

    val mapped2 = ds.map(f)
    assertCached(mapped2)
  }