Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-3271

Table-valued function windowing and EMIT syntax support in Calcite

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.20.0, 1.21.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      Copied from the mailing list:
      Calcite has not implemented the syntax in that paper. I would support an effort to add it (unsurprising, since I am a co-author of that paper).

      EMIT STREAM is equivalent to the current SELECT STREAM syntax.

      There is no equivalent in current Calcite of the EMIT AFTER WATERMARK, or EMIT STREAM AFTER DELAY.

      HOP, TUMBLE and SESSION are supported in Calciteā€™s SQL parser, but following the paper would be replaced with a table function call. We could need to add HOP, TUMBLE and SESSION table functions. We would also need to make the system aware of how watermarks flow through these table functions (an area that the paper does not go into).

      Julian

      From Rui Wang

      Table-value function windowing means implementing TUMBLE/TOP/SESSION by table-value function. As table value functions, TUMBLE/HOP/SESSION assigns each row of input table to one or more intervals containing the specified watermarked event timestamp column. Those intervals are determined by the same way of current windowing support in Calcite: TUMBLE generates fixed length, non-overlapping intervals; HOP generates fixed length, overlapping intervals; SESSION generates sessionized intervals. The output table of this proposed implementation has all columns of input table plus two additional columns wstart and wend, which represent the start and the end of the interval respectively.

      possible gaps in current Calcite:
      1. TABLE parameter support in table-value function.
      2. DESCRIPTOR parameter support in table-value function.
      3. TUMBLE/TOP/SESSION implemented as built-in table-value function.
      4. EMIT AFTER WATERMARK and EMIT STREAM AFTER DELAY.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                amaliujia Rui Wang
                Reporter:
                danny0405 Danny Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 54h 40m
                  54h 40m