[KAFKA-6373] Log end offset of input table changing during restore

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.0.0
• Component/s: streams
• Environment: Client 1.0.0, Brokers 1.0.0 with 1.0.0 message format and inter-broker protocol
• Assignee: Unassigned
• Reporter: Charles Crain
• Created: 15/Dec/17 19:52
• Updated: 18/Dec/17 18:58
• Resolved: 18/Dec/17 18:58

Description

I am receiving a confusing error from a Kafka Streams application. Most of the time when I try to bring up just a single replica of the task for the first time, I get this:

Detected a task that got migrated to another thread. This implies that this thread missed a rebalance and dropped out of the consumer group. Trying to rejoin the consumer group now.
org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of [Name of Topic]-36 should not change while restoring: old end offset 37559, current offset 37561

The confusing thing is that [Name of Topic] is not a change log topic created by the stream app. Rather it is a topic published from a completely different service. And since that other service is publishing to that topic actively, of course the end offset is constantly changing.

Here is a rough view of my stream topology. I'll call the topic that's showing up in the above error "ExternalTableTopic".

externalTable = table(ExternalTableTopic)

stream(ExternalStreamTopic)
  .leftJoin(externalTable, joiner)
  .aggregate(aggregator, SomeQueryableStoreName)

...and that's it. If I take out the left join this appears not to happen. Is it illegal to join a table to a stream if that table is being published from somewhere else?

Issue Links

• is related to: KAFKA-6269 KTable state restore fails after rebalance