[KAFKA-13479] Interactive Query v2

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Component/s: streams
• Assignee: John Roesler
• Reporter: John Roesler
• Created: 25/Nov/21 00:22
• Updated: 07/Apr/22 03:30
• Resolved: 07/Apr/22 03:30

Description

Kafka Streams supports an interesting and innovative API for "peeking" into the internal state of running stateful stream processors from outside of the application, called Interactive Query (IQ). This functionality has proven invaluable to users over the years for everything from debugging running applications to serving low latency queries straight from the Streams runtime.

However, the actual interfaces for IQ were designed in the very early days of Kafka Streams, before the project had gained significant adoption, and in the absence of much precedent for this kind of API in peer projects. With the benefit of hindsight, we can observe several problems with the original design that we hope to address in a revised framework that will serve Streams users well for many years to come.

 

This ticket tracks the implementation of KIP-796: https://cwiki.apache.org/confluence/x/34xnCw

Issue Links

• relates to: KAFKA-13523 Implement IQv2 support in global stores
• relates to: KAFKA-13526 IQv2: Consider more generic logic for mapping between binary and typed queries
• relates to: KAFKA-13541 Make IQv2 query/store interface type safe
• relates to: KAFKA-13548 IQv2: revisit WindowKeyQuery and WindowRangeQuery
• relates to: KAFKA-13622 Revisit the complexity of position tracking in state stores
• links to: GitHub Pull Request #11598

Sub-Tasks

• IQv2: Track Position in KeyValue stores: Closed
• Implement IQv2 Framework: Closed
• IQ Parity: queries for key/value store range and scan: Resolved
• Session and Window Queries for IQv2: Resolved
• IQv2: Track Position in remaining stores: Closed
• IQv2: Transmit position to standbys: Closed
• IQv2: Implement position tracking and bounding in API: Resolved
• IQv2: Implement KeyQuery from the RecordCache: Resolved
• IQv2: Implement KeyQuery from the KIP: Resolved
• Add PAPI stores to IQv2StoreIntegrationTest: Resolved
• Rename RangeQuery to KeyRangeQuery: Closed
• IQv2: Remove swapResult from the public API: Resolved
• Implement Position restoration for all in-memory state stores: Resolved
• Checkpoint position in state stores: Resolved