Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10454

Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0, 2.6.1, 2.8.0
    • Component/s: streams
    • Labels:
      None

      Description

      Here's integration test: https://github.com/apache/kafka/pull/9237

       

      From the first glance, issue is that when one joins stream to table, and table source topic doesn't have same number of partitions as stream topic, `StateChangelogReader` tries to recover state from changelog (which in this case is the same as source topic) for table from partitions that don't exist. Logs are spammed with: 

       

      [2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,517] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,524] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,524] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,525] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,525] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,526] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
      

      And Kafka Streams never moves to RUNNING state.

        Attachments

          Activity

            People

            • Assignee:
              lkokhreidze Levani Kokhreidze
              Reporter:
              lkokhreidze Levani Kokhreidze
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: