Uploaded image for project: 'Bahir'
  1. Bahir
  2. BAHIR-104

MQTT Dstream returned by the new multi topic support API is not a pairRDD

    Details

    • Flags:
      Important

      Description

      The new multi topic support API added with BAHIR-89, when used in pyspark, does not return a Dstream of <topic,message> tuples.
      Example:
      In pyspark, when creating a Dstream using the new API ( mqttstream = MQTTUtils.createPairedStream(ssc, brokerUrl, topics) ) the expected contents of mqttstream should be a collections of tuples:

      (topic,message) , (topic,message) , (topic,message) , ...

      Instead, the current content is a flattened list:

      topic, message, topic, message, topic, message, ...

      that is hard to use.

        Activity

        Hide
        luciano resende Luciano Resende added a comment -

        Anntinu Josy would you be able to take a quick look at this ?
        Francesco Beneventi you are also welcomed to work on this...

        Show
        luciano resende Luciano Resende added a comment - Anntinu Josy would you be able to take a quick look at this ? Francesco Beneventi you are also welcomed to work on this...
        Hide
        anntinutj Anntinu Josy added a comment -

        Francesco Beneventi Thanks for reporting this issue, In mqtt.py , helper.createStream is called instead of helper.createPairedStream . Can you please fix this and add python test case and one example.

        Show
        anntinutj Anntinu Josy added a comment - Francesco Beneventi Thanks for reporting this issue, In mqtt.py , helper.createStream is called instead of helper.createPairedStream . Can you please fix this and add python test case and one example.
        Hide
        fbeneventi Francesco Beneventi added a comment - - edited

        Anntinu Josy, I fixed the wrong method call before running my tests. You can check it in my test repo Github

        Even using the correct API (helper.createPairedStream) I have the same issue.

        I'm not really a java/scala programmer but can you help me figure out how to address this problem?
        After solving this problem I can provide a simple test case to use this API in python.

        Show
        fbeneventi Francesco Beneventi added a comment - - edited Anntinu Josy , I fixed the wrong method call before running my tests. You can check it in my test repo Github Even using the correct API (helper.createPairedStream) I have the same issue. I'm not really a java/scala programmer but can you help me figure out how to address this problem? After solving this problem I can provide a simple test case to use this API in python.
        Hide
        anntinutj Anntinu Josy added a comment -

        Francesco Beneventi I had tested the Java and scala API and Its working as expected. I am having less experience in python. I will test and fix it in the weekend.

        Show
        anntinutj Anntinu Josy added a comment - Francesco Beneventi I had tested the Java and scala API and Its working as expected. I am having less experience in python. I will test and fix it in the weekend.
        Hide
        fbeneventi Francesco Beneventi added a comment -

        Anntinu Josy do you have any news about this issue?

        Show
        fbeneventi Francesco Beneventi added a comment - Anntinu Josy do you have any news about this issue?
        Hide
        anntinutj Anntinu Josy added a comment -

        Sorry for the delay, I was away due to personal emergency. I will start working on it

        Show
        anntinutj Anntinu Josy added a comment - Sorry for the delay, I was away due to personal emergency. I will start working on it

          People

          • Assignee:
            Unassigned
            Reporter:
            fbeneventi Francesco Beneventi
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development