Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9929

Support reverse iterator on WindowStore

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: streams
    • Labels:

      Description

      KIP-617: https://cwiki.apache.org/confluence/display/KAFKA/KIP-617%3A+Allow+Kafka+Streams+State+Stores+to+be+iterated+backwards 

      Currently, WindowStore fetch operations return an iterator sorted from earliest to latest result:

      ```

      • For each key, the iterator guarantees ordering of windows, starting from the oldest/earliest
      • available window to the newest/latest window.

      ```

       

      We have a use-case where traces are stored in a WindowStore<String, Trace> and use Kafka Streams to create a materialized view of traces. A query request comes with a time range (e.g. now-1h, now) and want to return the most recent results, i.e. fetch from this period of time, iterate and pattern match latest/most recent traces, and if enough results, then reply without moving further on the iterator.

      Same store is used to search for previous traces. In this case, it search a key for the last day, if found traces, we would also like to iterate from the most recent.

      RocksDb seems to support iterating backward and forward: https://github.com/facebook/rocksdb/wiki/Iterator#iterating-upper-bound-and-lower-bound

       

      For reference: This in some way extracts some bits from this previous issue: https://issues.apache.org/jira/browse/KAFKA-4212:

       

      > The RocksDBWindowsStore, a WindowsStore, can expire items via segment dropping, but it stores multiple items per key, based on their timestamp. But this store can be repurposed as a cache by fetching the items in reverse chronological order and returning the first item found.

       

      Would like to know if there is any impediment on RocksDb or  WindowStore to support this.

      Adding an argument to reverse in current fetch methods would be great:

      ```

      WindowStore.fetch(from,to,Direction.BACKWARD|FORWARD)

      ```

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jeqo Jorge Esteban Quilcate Otoya
                Reporter:
                jeqo Jorge Esteban Quilcate Otoya
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: