Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8410

Strengthen the types of Processors, at least in the DSL, maybe in the PAPI as well

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      Presently, it's very difficult to have confidence when adding to or modifying processors in the DSL. There's a lot of raw types, duck-typing, and casting that contribute to this problem.

      The root, though, is that the generic types on `Processor<K,V>` refer only to the input key and value types. No information is captured or verified about what the output types of a processor are. For example, this leads to widespread confusion in the code base about whether a processor produces `V`s or `Change<V>`s. The type system actually makes matters worse, since we use casts to make the processors conform to declared types that are in fact wrong, but are never checked due to erasure.

      We can start to make some headway on this tech debt by adding some types to the ProcessorContext that bound the `<K,V>` that may be passed to `context.forward`. Then, we can build on this by fully specifying the input and output types of the Processors, which in turn would let us eliminate the majority of unchecked casts in the DSL operators.

      I'm not sure whether adding these generic types to the existing ProcessorContext and Processor interfaces, which would also affect the PAPI has any utility, or whether we should make this purely an internal change by introducing GenericProcessorContext and GenericProcessor peer interfaces for the DSL to use.

        Attachments

          Issue Links

          1.
          Introduce the KIP-478 processors with shims Sub-task Resolved John Roesler
          2.
          Implement the KIP-478 StreamBuilder#addGlobalStore() Sub-task Resolved John Roesler
          3.
          Implement KIP-478 Topology changes Sub-task Resolved John Roesler
          4.
          KIP-478: Implement test-utils changes Sub-task Resolved John Roesler
          5.
          KIP-478: Implement StateStoreContext and Record Sub-task Resolved John Roesler
          6.
          KIP-478: Implement KStream changes Sub-task Resolved John Roesler
          7.
          Convert KStreamImpl filters to new PAPI Sub-task Resolved Jorge Esteban Quilcate Otoya
          8.
          Convert KStreamImpl maps to new PAPI Sub-task Resolved Jorge Esteban Quilcate Otoya
          9.
          Convert KStreamImpl joins to new PAPI Sub-task Open Jorge Esteban Quilcate Otoya
          10.
          Convert KStream aggregations to new PAPI Sub-task In Progress Jorge Esteban Quilcate Otoya
          11.
          Convert KTable filters to new PAPI Sub-task Resolved John Roesler
          12.
          Convert KTable suppress to new PAPI Sub-task Resolved Jorge Esteban Quilcate Otoya
          13.
          Convert KTable maps to new PAPI Sub-task Resolved Jorge Esteban Quilcate Otoya
          14.
          Convert KTable joins to new PAPI Sub-task Open Jorge Esteban Quilcate Otoya
          15.
          Convert KTable aggregations to new PAPI Sub-task In Progress Jorge Esteban Quilcate Otoya
          16.
          KIP-478: Deprecate the old PAPI interfaces Sub-task Resolved John Roesler
          17.
          KIP-478: Delegate the store wrappers to the new init method Sub-task Resolved John Roesler
          18.
          KIP-478: deprecate the replaced Processor API members Sub-task Resolved John Roesler
          19.
          After migrating processors, search the codebase for missed migrations Sub-task Open Unassigned
          20.
          After processors, migrate TupleForwarder and CacheFlushListener Sub-task Open Unassigned

            Activity

              People

              • Assignee:
                vvcephei John Roesler
                Reporter:
                vvcephei John Roesler
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: