Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-18026

Allow custom processor wrapping

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0
    • kip, streams

    Description

      See KIP-1112: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1112%3A+allow+custom+processor+wrapping

      Implementation plan/wrapping procedure for DSL operators

      • make sure processor is added via ProcessorParameters#addProcessorTo instead of calling the  InternalTopologyBuilder's #addProcessor and #addStateStore methods directly 
      • convert stateful operators to implement the ProcessorSupplier#stores method, rather than directly calling #addStateStore and/or #connectProcessorAndStateStore
      • update and/or add ProcessorWrapper tests to StreamsBuilderTest (some existing tests may use processors that haven't been converted yet and are expected to break/need fixing)  

      TODO list:

      1. Non-source/sink PAPI processors: https://github.com/apache/kafka/pull/17892 --  ableegoldman
      2. ProcessorGraphNode (stateless DSL KStream operators): https://github.com/apache/kafka/pull/17892   --  ableegoldman
      3. KTableSource: https://github.com/apache/kafka/pull/17903 -- agavra 
      4. KStream/TableAggregate (ie count, reduce, aggregate): https://github.com/apache/kafka/pull/17929 -- ableegoldman 
      5. TableProcessorNode (stateless table operators eg KTable#filter):
        1. filter: https://github.com/apache/kafka/pull/18205 – rohan
        2. mapValues: – rohan
        3. transformValues: won't do, will be replaced with processValues anyway
      6. StreamToTableNode:  https://github.com/apache/kafka/pull/18149 – agavra 
      7. Stream-Table join: https://github.com/apache/kafka/pull/18047 – ableegoldman 
      8. StreamStreamJoinNode: https://github.com/apache/kafka/pull/18111 – ableegoldman 
      9. KTableKTableJoinNode: https://github.com/apache/kafka/pull/18048 -– agavra 
      10. StatefulProcessorNode:
        1. TableSuppressNode: https://github.com/apache/kafka/pull/18150  – agavra 
        2. ForeignTableJoinNode: https://github.com/apache/kafka/pull/18194 – ableegoldman 
        3. clean up StatefulProcessorNode  and migrate all converted operators to use ProcessorGraphNode instead: https://github.com/apache/kafka/pull/18195 sophie

      Follow up:

      • convert source & sink nodes to using ProcessorSupplier and wrap them too
      • clean up StoreFactory<->StoreBuilder wrapping and configuration
      • future-proof the wrapping mechanism:
        • ensure new processor implementations get wrapped, eg by protecting the InternalTopologyBuilder#addProcessor (also #addSource/Sink?)
        • protect #addStateStore from being called out-of-band to prevent new state stores (whether from new DSL operators or modifications to existing ones) from being added to processors without being returned by the ProcessorSupplier#stores method 
      • consider deprecating older alternative to ProcessorSupplier#stores
        • cons: using lambdas for processor suppliers is very nice

      docs: https://github.com/apache/kafka/pull/17906

      Attachments

        Activity

          People

            ableegoldman A. Sophie Blee-Goldman
            ableegoldman A. Sophie Blee-Goldman
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: