Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-552

Implement window operator in Samza

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0
    • Fix Version/s: None
    • Component/s: sql
    • Labels:

      Description

      The discussion is based on how to support tuple and/or time based window operators in Samza physical operator layer.

      Here are the few observations:

      1. Tuple represents the “physical ordering” of events while time-based window has semantic meanings to users
      2. Total ordering between tuples are possible within Samza/Kafka given a deterministic MessageSelector on all input streams and offsets within each stream
      3. No matter whether tuple or time is used to measure the window size, the window termination condition is needed to close a window to avoid the job to be wedged forever

      The following questions have to be answered to fully implement a window operator:

      1. how to determine that a window is closed and no new tuples will be added?
        1. For tuple based, how do we close the window if messages do not come or get delayed?
        2. For time based, how do we close the window if
          1. the messages are not strictly in order w/ the time?
          2. the message w/ timestamp greater than the window boundary does not come or gets delayed?

        Attachments

        1. DESIGN-SAMZA-552-3.md
          40 kB
          Yi Pan
        2. DESIGN-SAMZA-552-3.pdf
          93 kB
          Yi Pan
        3. DESIGN-SAMZA-552-6.md
          52 kB
          Yi Pan
        4. DESIGN-SAMZA-552-6.pdf
          101 kB
          Yi Pan
        5. DESIGN-SAMZA-552-7.md
          53 kB
          Yi Pan
        6. DESIGN-SAMZA-552-7.pdf
          101 kB
          Yi Pan
        7. SAMZA-552-0.patch
          143 kB
          Yi Pan

          Activity

            People

            • Assignee:
              nickpan47 Yi Pan
              Reporter:
              nickpan47 Yi Pan
            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated: