[BEAM-11851] ConfluentSchemaRegistryProvider fails when authentication is required

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.27.0
• Fix Version/s: 2.31.0
• Component/s: io-java-kafka
• Labels: stale-P2
• Assignee: Tom Underhill
• Reporter: Maxim Ivanov
• Created: 23/Feb/21 06:10
• Updated: 21/May/21 17:33
• Resolved: 21/May/21 17:33

Description

When configuring KafkaIO.reader with ConfluentSchemaRegistryProvider as value deserializer with `basic.auth.credentials.source=AUTH_INO` and `basic.auth.user.info=user:password` properties, it fails to start the pipepline

[error] java.lang.RuntimeException: Unable to get latest schema metadata for subject: identity_users_v2-value
[error]   at org.apache.beam.sdk.io.kafka.ConfluentSchemaRegistryDeserializerProvider.getSchemaMetadata(ConfluentSchemaRegistryDeserializerProvider.java:119)
[error]   at org.apache.beam.sdk.io.kafka.ConfluentSchemaRegistryDeserializerProvider.getAvroSchema(ConfluentSchemaRegistryDeserializerProvider.java:110)
[error]   at org.apache.beam.sdk.io.kafka.ConfluentSchemaRegistryDeserializerProvider.getCoder(ConfluentSchemaRegistryDeserializerProvider.java:106)
[error]   at org.apache.beam.sdk.io.kafka.KafkaIO$Read.getValueCoder(KafkaIO.java:1147)
[error]   at org.apache.beam.sdk.io.kafka.KafkaIO$Read.expand(KafkaIO.java:1052)
[error]   at org.apache.beam.sdk.io.kafka.KafkaIO$Read.expand(KafkaIO.java:481)
[error]   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:547)
[error]   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:498)
[error]   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
[error]   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:191) 

 

I suspect that it because it reconfigured KafkaAvroDeserializer only in getDeserializer, but doesn't do so in getCoder.

Issue Links

• links to: GitHub Pull Request #14840