[KAFKA-9390] Non-key joining of KTable not compatible with confluent avro serdes

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.5.0, 2.4.1
• Component/s: streams
• Assignee: John Roesler
• Reporter: Andy Bryant
• Created: 09/Jan/20 05:20
• Updated: 12/Feb/20 00:30
• Resolved: 12/Feb/20 00:30

Description

I was trying out the new one-to-many KTable joins against some CDC data in Avro format and kept getting serialisation errors.

 

org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"Key","namespace":"dbserver1.inventory.orders","fields":[

{"name":"order_number","type":"int"}

],"connect.name":"dbserver1.inventory.orders.Key"}
 Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema; error code: 409
  

Both tables have avro keys of different types (one is an order key, the other a customer key).

This looks like it will cause issues.

https://github.com/apache/kafka/blob/2.4/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/CombinedKeySchema.java#L57-L60

 They will both attempt to register schemas with the same subject to the schema registry which will fail a backward compatibility check.

I also noticed in the schema registry there were some subjects that didn't have the application id prefix. This is probably caused by this...

 https://github.com/apache/kafka/blob/2.4/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/ForeignJoinSubscriptionSendProcessorSupplier.java#L88

Where here repartitionTopicName doesn't have the application prefix.

 

 

 

 

 

Issue Links

• relates to: KAFKA-3705 Support non-key joining in KTable
• links to: GitHub Pull Request #8054