[BAHIR-104] MQTT Dstream returned by the new multi topic support API is not a pairRDD

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Spark-2.1.0
• Fix Version/s: Spark-2.2.1, Spark-2.3.0
• Component/s: Spark Streaming Connectors
• Labels: MQTT SPARK
• Flags: Important
• Assignee: Francesco Beneventi
• Reporter: Francesco Beneventi
• Created: 05/Apr/17 23:16
• Updated: 21/Dec/17 00:45
• Resolved: 21/Dec/17 00:45

Description

The new multi topic support API added with BAHIR-89, when used in pyspark, does not return a Dstream of <topic,message> tuples.
Example:
In pyspark, when creating a Dstream using the new API ( mqttstream = MQTTUtils.createPairedStream(ssc, brokerUrl, topics) ) the expected contents of mqttstream should be a collections of tuples:

(topic,message) , (topic,message) , (topic,message) , ...

Instead, the current content is a flattened list:

topic, message, topic, message, topic, message, ...

that is hard to use.

Issue Links

• links to: GitHub Pull Request #55