Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7851

PubsubIO.readStrings() and DataflowRunner

    XMLWordPrintableJSON

    Details

      Description

      Hi all,

      I have a potential issue I am debugging with `org.apache.beam.sdk.io.gcp.pubsub`. When using the GCP PubSub service no messages are received on the DirectRunner or the DataflowRunner when using PubsubIO.readStrings(). When using the DirectRunner and the Pubsub Emulator it works correctly locally.

      I have been working with this issue for the past few days and believe it may be related to the PubsubIO library.

       

      I am listening to a topic (also tried a subscription) using:

      PubsubIO.Read<String> read = PubsubIO
          .readStrings()
          .withTimestampAttribute("timestamp")
          .fromTopic(options.getTopic());

       

      Alternatively this method works:

      PubsubIO.Read<PubsubMessage> read = PubsubIO
          .readMessagesWithAttributes()
          .withTimestampAttribute("timestamp")
          .fromTopic(options.getTopic()); 

      The code works perfectly with the Pubsub Emulator using readStrings() and after switching to readMessagesWithAttributes() everything works correctly on both the emulator as well as GCP.

       

      I am using Beam "2.13.0" with "beam-runners-google-cloud-dataflow-java" with Java 8. 

      My "gcloud" versions:

      Google Cloud SDK 256.0.0
      beta 2019.05.17
      bigtable
      bq 2.0.46
      core 2019.07.26
      gsutil 4.41
      pubsub-emulator 2019.04.26 

        Attachments

          Activity

            People

            • Assignee:
              chamikara Chamikara Madhusanka Jayalath
              Reporter:
              gregorskii Gregory Parsons
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: