[BEAM-9525] KafkaIO attempting to make connection when uploading via Dataflow Runner

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: io-java-kafka
• Assignee: Unassigned
• Reporter: Ross Rothenstine
• Created: 16/Mar/20 22:51
• Updated: 04/Jun/22 15:29

Description

Given our Kafka is running in GCP, within a VPC that is not accessible outside of the VPC.

When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD worker, it attempts to make a connection to the broker. Our broker isn't available outside the VPC, so this process fails. We can remedy this by running a pod from within the VPC to upload, but it causes an awkward step within our CI/CD process.

If there is a more elegant solution to this, such as KafkaIO taking in a ValueProvider for bootstrap servers.