Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4499

Add "getAllKeys" API for querying windowed KTable stores

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: streams
    • Labels:

      Description

      Currently, both KTable and windowed-KTable stores can be queried via IQ feature. While ReadOnlyKeyValueStore (for KTable stores) provide method all() to scan the whole store (ie, returns an iterator over all stored key-value pairs), there is no similar API for ReadOnlyWindowStore (for windowed-KTable stores).

      This limits the usage of a windowed store, because the user needs to know what keys are stored in order the query it. It would be useful to provide possible APIs like this (only a rough sketch):

      • keys() returns all keys available in the store (maybe together with available time ranges)
      • all(long timeFrom, long timeTo) that returns all window for a specific time range
      • allLatest() that returns the latest window for each key

      Because this feature would require to scan multiple segments (ie, RockDB instances) it would be quite inefficient with current store design. Thus, this feature also required to redesign the underlying window store itself.

      Because this is a major change, a KIP (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) is required. The KIP should cover the actual API design as well as the store refactoring.

        Attachments

        1. 4499-CachingWindowStore-v1.patch
          5 kB
          Richard Yu
        2. 4499-All-test-v1.patch
          23 kB
          Richard Yu

          Activity

            People

            • Assignee:
              Yohan123 Richard Yu
              Reporter:
              mjsax Matthias J. Sax
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: