[BAHIR-300] Source topic info in received message attribute for multi topic stream

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Spark Streaming Connectors
• Labels: GCP pubsub spark
• Assignee: Unassigned
• Reporter: Nitin Chauhan
• Created: 18/Feb/22 09:59
• Updated: 18/Feb/22 10:06

Description

In spark streaming pubsub connector, currently there is no way of getting source topic from the received messages.
We have a use case where in we are creating multiple streams and taking union of them in order to process stream from several pubsub topics. 

Here we were getting issue to determine which message is coming from which topic which is required to solve our use case.

 

Here we have two case

1. Topic info was provided during stream creation:
   1. We can append source topic info in pubsub message attributes before store it to store. 
2. Topic info was not provided during stream creation:
   1. We don't have any source topic info so nothing will be appended.

 

This Jira is for adding support for adding source as part of PubSubMessage attributes which is available to be consume during processing RDDs.

Issue Links

• is a clone of: BAHIR-295 Backpressure & Rate limiting support in streaming pubsub connector