Major Description
Here's integration test: https://github.com/apache/kafka/pull/9237
From the first glance, issue is that when one joins stream to table, and table source topic doesn't have same number of partitions as stream topic, `StateChangelogReader` tries to recover state from changelog (which in this case is the same as source topic) for table from partitions that don't exist. Logs are spammed with:
[2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,517] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,524] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,524] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,525] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,525] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,526] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
And Kafka Streams never moves to RUNNING state.