Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8088

PCollection boundedness should be tracked and propagated in python sdk

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.16.0
    • Component/s: sdk-py-core
    • Labels:
      None

      Description

      As far as I can tell Python does not care about boundedness of PCollections even in streaming mode, but external transforms do.  In my ongoing effort to get PubsubIO external transforms working I discovered that I could not generate an unbounded write. 

      My pipeline looks like this:

          (
              pipe
              | 'PubSubInflow' >> external.pubsub.ReadFromPubSub(subscription=subscription, with_attributes=True)
              | 'PubSubOutflow' >> external.pubsub.WriteToPubSub(topic=OUTPUT_TOPIC, with_attributes=True)
          )
      

      The PCollections returned from the external Read are Unbounded, as expected, but python is responsible for creating the intermediate PCollection, which is always Bounded, and thus external Write is always Bounded.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                chadrik Chad Dombrova
                Reporter:
                chadrik Chad Dombrova
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m