[SPARK-26213] Custom Receiver for Structured streaming

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 2.4.0
• Component/s: Structured Streaming
• Assignee: Unassigned Assign to me
• Reporter: Aarthi
• Created: 29/Nov/18 09:47
• Updated: 06/Dec/18 18:30
• Resolved: 06/Dec/18 18:30

Description

Hi,

I have implemented a Custom Receiver for a https/json data source by implementing the Receievr abstract class as provided in the documentation here https://spark.apache.org/docs/latest//streaming-custom-receivers.html

This approach works on Spark streaming context  where the custom receiver class is passed it receiverStream. However I would like the implement the same for Structured streaming as each of the DStreams have a complex structure and need to be joined with each other based on complex rules. (https://stackoverflow.com/questions/53449599/join-two-spark-dstreams-with-complex-nested-structure)

Structured streaming uses the Spark Session object that takes in DataStreamReader which is a final class. Please advice on how to implement the custom receiver for Strucutred Streaming. 

Thanks,