[BEAM-10936] PubsubIO can't provide PubsubClientFactory

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.19.0, 2.20.0, 2.21.0, 2.22.0, 2.23.0
• Component/s: io-java-gcp
• Environment: Dataflow
• Assignee: Unassigned
• Reporter: Jian Zheng
• Created: 21/Sep/20 15:00
• Updated: 04/Jun/22 18:57

Description

PubsubIO provides a method to pass the PubsubClientFactory.

/**
 * The default client to write to Pub/Sub is the {@link PubsubJsonClient}, created by the {@link
 * PubsubJsonClient.PubsubJsonClientFactory}. This function allows to change the Pub/Sub client
 * by providing another {@link PubsubClient.PubsubClientFactory} like the {@link
 * PubsubGrpcClientFactory}.
 */
public Read<T> withClientFactory(PubsubClient.PubsubClientFactory factory) {
  return toBuilder().setPubsubClientFactory(factory).build();
}

The comment here explains that can pass a PubsubClient.

However, the IncomingMessage used in the pull() method of PubsubClient's abstract method is actually an abstract internal class, in other words, the abstract class PubsubClient can't be extended in external of the org.apache.beam.sdk.io.gcp.pubsub package.

To overcome the above problem, I created the org.apache.beam.sdk.io.gcp.pubsub directory to extend PubsubClient and passes it to the PubsubIO.withClientFactory method.

This seems to run locally without any problems, but when I submit my job to Dataflow, the withClientFactory method is useless and be replaced with the default PubsubJsonClientFactory.