[BEAM-10926] Specify the event time when consuming pubsub data.

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: io-java-gcp
• Assignee: Unassigned
• Reporter: Jian Zheng
• Created: 18/Sep/20 02:13
• Updated: 04/Jun/22 19:00

Description

I need to specify the event time when consuming pubsub data.

PCollection<PubsubMessage> pubsubMessages = pipeline.apply("Read Pub/Sub Data",
       PubsubIO.readMessagesWithAttributes()
                .withTimestampAttribute(options.getTimeAttribute())
                .fromSubscription(options.getInputSubscription()));

The only way to do this is to use the withTimestampAttribute() method.

But if  I use a timestamp in some other format, such as a 19-bit nanosecond, or if save the event time in the payload. The method won't work !

So I had to extend the PubsubClient class and override the PubsubClient. extractTimestamp() method.

I'am hoping to provide a way to pass in some implementation class that would allow to parse out timestamp from the current pubsub message.

 

My beam version is 2.19.0.