Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35246

Streaming-batch intersects are incorrectly allowed through UnsupportedOperationsChecker

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 3.1.0
    • Fix Version/s: 3.2.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      The UnsupportedOperationChecker currently rejects streaming intersects only if both sides are streaming, but they don't work if even one side is streaming. The following simple test, for example, fails with a cryptic None.get error because the state store can't plan itself properly.

        test("intersect") {
          val input = MemoryStream[Long]
          val df = input.toDS().intersect(spark.range(10).as[Long])
          testStream(df) (
            AddData(input, 1L),
            CheckAnswer(1)
          )
        }
      

        Attachments

          Activity

            People

            • Assignee:
              joseph.torres Jose Torres
              Reporter:
              joseph.torres Jose Torres
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: