Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27624

Fix CalenderInterval to show an empty interval correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.3, 2.4.2, 3.0.0
    • Fix Version/s: 2.3.4, 2.4.4, 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      BEFORE

      scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "1 microsecond").explain
      == Physical Plan ==
      EventTimeWatermark ts#0: timestamp, interval 1 microseconds
      +- StreamingRelation FileSource[/tmp/t], [ts#0]
      
      scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "0 microsecond").explain
      == Physical Plan ==
      EventTimeWatermark ts#3: timestamp, interval
      +- StreamingRelation FileSource[/tmp/t], [ts#3]
      

      AFTER

      scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "1 microsecond").explain
      == Physical Plan ==
      EventTimeWatermark ts#0: timestamp, interval 1 microseconds
      +- StreamingRelation FileSource[/tmp/t], [ts#0]
      
      scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "0 microsecond").explain
      == Physical Plan ==
      EventTimeWatermark ts#3: timestamp, interval 0 microseconds
      +- StreamingRelation FileSource[/tmp/t], [ts#3]
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dongjoon Dongjoon Hyun
                Reporter:
                dongjoon Dongjoon Hyun
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: