Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38214

No need to filter windows when windowDuration is multiple of slideDuration

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.2.1
    • 3.3.0
    • Structured Streaming
    • None

    Description

      At present, the sliding window adopts the form of expand + filter, but in some cases, filter is not necessary.

      Filter is required if the sliding window is irregular. When the window length is divided by the slide length the result is an integer (I believe this is also the case for most work scenarios in practice for sliding window), there is no need to filter, which can save calculation resources and improve performance.

      Attachments

        Activity

          People

            nyingping nyingping
            nyingping nyingping
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: