Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11998

Portable runners should be able to issue checkpoints to Splittable DoFn

    XMLWordPrintableJSON

Details

    Description

      To execute unbounded Splittable DoFn over fnapi in streaming mode properly, portable runners should issue split(ProcessBundleSplitRequest with fraction_of_remainder > 0) or simply checkpoint(ProcessBundleSplitRequest with fraction_of_remainder == 0) to SDK regularly to make current bundle finished processing instead of running forever.

      Attachments

        1. read.png
          35 kB
          zhao

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. BEAM-11991 Python Kafka source not emitting messages for streaming pipelines with Flink Runner

          • P1 - Critical bug: data loss, total loss of function, or loss of testing signal due to test failures or flakiness
          • Resolved

          Activity

            People

              Unassigned Unassigned
              boyuanz Boyuan Zhang
              Votes:
              11 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated: