Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-4076 Schema followups
  3. BEAM-6756

Support lazy iterables in schemas

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.18.0
    • Component/s: sdk-java-core
    • Labels:
      None

      Description

      The iterables returned by GroupByKey and CoGroupByKey are lazy; this allows a runner to page data into memory if the full iterable is too large. We currently don't support this in Schemas, so the Schema Group and CoGroup transforms materialize all data into memory. We should add support for this.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                reuvenlax Reuven Lax
                Reporter:
                reuvenlax Reuven Lax
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2.5h
                  2.5h