[BEAM-12076] Update Python cross-language Kafka source to read metadata

• Type: New Feature
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: 2.32.0
• Component/s: cross-language, io-py-kafka
• Assignee: Chamikara Madhusanka Jayalath
• Reporter: Chamikara Madhusanka Jayalath
• Created: 31/Mar/21 02:22
• Updated: 01/Jul/21 21:31
• Resolved: 01/Jul/21 21:31

Description

Currently Python cross-language Kafka source [1] uses Java Kafka TypedWithoutMetadata transform [2]. We should update this so that users who need to use Kafka metadata in Python side have access to it.

[1] https://github.com/apache/beam/blob/fd3075fa52303a0d684c2d7a91814eea5305476e/sdks/python/apache_beam/io/kafka.py#L107

[2] https://github.com/apache/beam/blob/fd3075fa52303a0d684c2d7a91814eea5305476e/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L717

Issue Links

• links to: GitHub Pull Request #15028