Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6373

Log end offset of input table changing during restore

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: streams
    • Labels:
      None
    • Environment:
      Client 1.0.0, Brokers 1.0.0 with 1.0.0 message format and inter-broker protocol

      Description

      I am receiving a confusing error from a Kafka Streams application. Most of the time when I try to bring up just a single replica of the task for the first time, I get this:

      Detected a task that got migrated to another thread. This implies that this thread missed a rebalance and dropped out of the consumer group. Trying to rejoin the consumer group now.
      org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of [Name of Topic]-36 should not change while restoring: old end offset 37559, current offset 37561
      

      The confusing thing is that [Name of Topic] is not a change log topic created by the stream app. Rather it is a topic published from a completely different service. And since that other service is publishing to that topic actively, of course the end offset is constantly changing.

      Here is a rough view of my stream topology. I'll call the topic that's showing up in the above error "ExternalTableTopic".

      externalTable = table(ExternalTableTopic)
      
      stream(ExternalStreamTopic)
        .leftJoin(externalTable, joiner)
        .aggregate(aggregator, SomeQueryableStoreName)
      

      ...and that's it. If I take out the left join this appears not to happen. Is it illegal to join a table to a stream if that table is being published from somewhere else?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ccrain_kuka Charles Crain
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: