Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2939

Fn API SDF support

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • Missing
    • beam-model

    Description

      The Fn API should support streaming SDF. Detailed design TBD.

      Once design is ready, expand subtasks similarly to BEAM-2822.

      Attachments

        Issue Links

          is a parent of

          New Feature - A new feature of the product, which has yet to be developed. BEAM-5605 Support Portable SplittableDoFn for batch

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2940 Portability milestone: mobile gaming runs everywhere

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-3301 Go SplittableDoFn support

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3698 Support SDF over Fn API

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-6142 Improve ByteBuddy DoFnInvoker implementation wrt SplittableDoFns / BundleFinalization

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          supercedes

          New Feature - A new feature of the product, which has yet to be developed. BEAM-71 Watermark library

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          links to

          Web Link GitHub Pull Request #5566

          Web Link GitHub Pull Request #6969

          Web Link GitHub Pull Request #7094

          Web Link GitHub Pull Request #7177

          Web Link GitHub Pull Request #7800

          Web Link GitHub Pull Request #8088

          Web Link GitHub Pull Request #8235

          Web Link GitHub Pull Request #8317

          Web Link GitHub Pull Request #8338

          Web Link GitHub Pull Request #8412

          Web Link GitHub Pull Request #8517

          Web Link GitHub Pull Request #10065

          Web Link GitHub Pull Request #10897

          Web Link GitHub Pull Request #10992

          Web Link GitHub Pull Request #11065

          Web Link GitHub Pull Request #11068

          Web Link GitHub Pull Request #11080

          Web Link GitHub Pull Request #11409

          Web Link GitHub Pull Request #11716

          Web Link GitHub Pull Request #11735

          Web Link GitHub Pull Request #11781

          Web Link GitHub Pull Request #11818

          Web Link GitHub Pull Request #11922

          Web Link GitHub Pull Request #11941

          1.
          		 Proto changes for splitting over Fn API Sub-task Resolved Eugene Kirpichov

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 8h 50m
          2.
          		 Support for running a streaming SDF in Python SDK Sub-task Resolved Boyuan Zhang

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 20m
          3.
          		 Support for SDF splitting protocol in ULR Sub-task Resolved Robert Bradshaw  
          4.
          		 Streaming Dataflow runner harness should understand BundleSplit returned from ProcessBundle Sub-task Resolved Luke Cwik  
          5.
          		 Java SDK harness should detect SDF ProcessFn and proactively checkpoint it Sub-task Resolved Eugene Kirpichov  
          6.
          		 Python SDK harness should detect SDF ProcessFn and proactively checkpoint it Sub-task Resolved Luke Cwik  
          7.
          		 Streaming Dataflow runner harness should understand a BundleSplit returned during execution of a bundle Sub-task Resolved Luke Cwik  
          8.
          		 Java SDK harness should understand a BundleSplitRequest and respond with a BundleSplit before bundle finishes Sub-task Resolved Luke Cwik

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          9.
          		 Python SDK harness should understand a BundleSplitRequest and respond with a BundleSplit before bundle finishes Sub-task Resolved Boyuan Zhang

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 50m
          10.
          		 Make Dataflow service receive required restriction encoding parameter on SplittableParDo Sub-task Resolved Luke Cwik

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h
          11.
          		 Make the standard double coder well known within the Java SDK Sub-task Resolved Luke Cwik

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          12.
          		 Migrate from ProcessContext#updateWatermark to WatermarkEstimators Sub-task Triage Needed Luke Cwik

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 8h 20m
          13.
          		 Make Dataflow executed UnboundedSources using SDF as the default Sub-task Resolved Luke Cwik  
          14.
          		 Add support for splitting at fractions > 0 to org.apache.beam.sdk.transforms.splittabledofn.ByteKeyRangeTracker Sub-task Resolved Boyuan Zhang

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 5h

          Activity

            People

              lcwik Luke Cwik
              herohde Henning Rohde
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 71h 50m
                  71h 50m