Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-4076 Schema followups
  3. BEAM-6756

Support lazy iterables in schemas

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Triage Needed
    • P2
    • Resolution: Fixed
    • None
    • 2.18.0
    • sdk-java-core

    Description

      The iterables returned by GroupByKey and CoGroupByKey are lazy; this allows a runner to page data into memory if the full iterable is too large. We currently don't support this in Schemas, so the Schema Group and CoGroup transforms materialize all data into memory. We should add support for this.

      Attachments

        Issue Links

          is depended upon by

          Sub-task - The sub-task of the issue BEAM-7116 Remove KV from Schema transforms

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          links to

          Web Link GitHub Pull Request #10003

          Activity

            People

              reuvenlax Reuven Lax
              reuvenlax Reuven Lax
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2.5h
                  2.5h