Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9390

Non-key joining of KTable not compatible with confluent avro serdes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0, 2.4.1
    • Component/s: streams
    • Labels:
      None

      Description

      I was trying out the new one-to-many KTable joins against some CDC data in Avro format and kept getting serialisation errors.

       

      org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"Key","namespace":"dbserver1.inventory.orders","fields":[
      
      {"name":"order_number","type":"int"}
      
      ],"connect.name":"dbserver1.inventory.orders.Key"}
       Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema; error code: 409
        
      

      Both tables have avro keys of different types (one is an order key, the other a customer key).

      This looks like it will cause issues.

      https://github.com/apache/kafka/blob/2.4/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/CombinedKeySchema.java#L57-L60

       They will both attempt to register schemas with the same subject to the schema registry which will fail a backward compatibility check.

      I also noticed in the schema registry there were some subjects that didn't have the application id prefix. This is probably caused by this...

       https://github.com/apache/kafka/blob/2.4/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/ForeignJoinSubscriptionSendProcessorSupplier.java#L88

      Where here repartitionTopicName doesn't have the application prefix.

       

       

       

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vvcephei John Roesler
                Reporter:
                kiwiandy Andy Bryant
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: