Details
Description
I'm installing an Apache Kafka MirrorMaker instance to replicate one cluster data to one another cluster. Both on the source and on the target clusters I'm using the Confluent Avro schema registry and the data is binarized with Avro.
I'm using the latest released version of Confluent 3.3.0 (kafka 0.11). Moreover, the source broker is on a Windows machine while the target broker is on a Linux machine.
The two Kafka clusters are independent, thus they have different schema registries.
This are my configuration files for the MirrroMaker
group.id=test-mirrormaker-group bootstrap.servers=host01:9092 exclude.internal.topics=true client.id=mirror_maker_consumer0 auto.commit.enabled=false # Avro schema registry properties key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://host01:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://host01:8081 internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter.schemas.enable=false internal.value.converter.schemas.enable=false
bootstrap.servers=host02:9093 compression.type=none acks=1 client.id=mirror_maker_producer0 # Avro schema registry properties key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://host02:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://host02:8081 internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter.schemas.enable=false internal.value.converter.schemas.enable=false
I run the MirrorMaker on the host01 Windows machine with this command
C:\kafka>.\bin\windows\kafka-mirror-maker.bat --consumer.config .\etc\kafka\consumer.properties --producer.config .\etc\kafka\producer.properties --whitelist=MY_TOPIC [2017-09-26 10:09:58,555] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-09-26 10:09:58,555] WARN The configuration 'value.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-09-26 10:09:58,571] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-09-26 10:09:58,586] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-09-26 10:09:58,602] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-09-26 10:09:58,633] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-09-26 10:09:58,649] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-09-26 10:09:58,649] WARN The configuration 'key.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-09-26 10:09:58,727] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-09-26 10:09:58,727] WARN The configuration 'value.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-09-26 10:09:58,727] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-09-26 10:09:58,742] WARN The configuration 'auto.commit.enabled' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-09-26 10:09:58,774] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-09-26 10:09:58,789] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-09-26 10:09:58,805] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-09-26 10:09:58,805] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-09-26 10:09:58,821] WARN The configuration 'key.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
Using the topic UI utility (https://github.com/Landoop/kafka-topics-ui) I can see that on the target broker the data is sent, but it is shown binarized and I think this is caused by the misconfiguration of the schema registry.
It seems that the MirrorMaker serializes both key and value data with the ByteArraySerializer, so it ignores the Avro Schema registry case
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L237
It would be very useful if the Kafka MirrorMaker would read the key/value serialization class parameters for producer and consumer, allowing to configue the Avro schema serde.