Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7663

Custom Processor supplied on addGlobalStore is not used when restoring state from topic

    XMLWordPrintableJSON

Details

    Description

      I have implemented a StreamBuilder#addGlobalStore supplying a custom processor responsible to transform a K,V record from the input stream into a V,K records. It works fine and my store.all() does print the correct persisted V,K records. However, if I clean the local store and restart the stream app, the global table is reloaded but without going through the processor supplied; instead, it calls GlobalStateManagerImp#restoreState which simply stores the input topic K,V records into rocksDB (hence bypassing the mapping function of my custom processor). I believe this must not be the expected result?

       This is a follow up on stackoverflow discussion around storing a K,V topic as a global table with some stateless transformations based on a "custom" processor added on the global store:

      https://stackoverflow.com/questions/50993292/kafka-streams-shared-changelog-topic#comment93591818_50993729

      If we address this issue, we should also apply `default.deserialization.exception.handler` during restore (cf. KAFKA-8037)

       

      Attachments

        1. image-2018-11-20-11-42-14-697.png
          157 kB
          Frederic Tardif

        Issue Links

          Activity

            People

              Unassigned Unassigned
              frederic.tardif Frederic Tardif
              Votes:
              9 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated: