[SPARK-35246] Streaming-batch intersects are incorrectly allowed through UnsupportedOperationsChecker

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.0
• Fix Version/s: 3.2.0
• Component/s: Structured Streaming
• Assignee: Jose Torres
• Reporter: Jose Torres
• Created: 27/Apr/21 17:25
• Updated: 12/Dec/22 18:10
• Resolved: 28/Apr/21 01:47

Description

The UnsupportedOperationChecker currently rejects streaming intersects only if both sides are streaming, but they don't work if even one side is streaming. The following simple test, for example, fails with a cryptic None.get error because the state store can't plan itself properly.

  test("intersect") {
    val input = MemoryStream[Long]
    val df = input.toDS().intersect(spark.range(10).as[Long])
    testStream(df) (
      AddData(input, 1L),
      CheckAnswer(1)
    )
  }

Issue Links

• links to: [Github] Pull Request #32371 (jose-torres)