Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-13130

[Go SDK] Side Input Memory Leak

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.35.0
    • sdk-go
    • None

    Description

      Keeping a list of the open stateKeyReader structs in the ScopedStateReader is holding on to memory for the life of bundles, leading to significant memory usage in large batch jobs. This list needs to be removed to the garbage collector can clean up stateKeyReader structs sooner.

       

      stateKeyReader: https://github.com/apache/beam/blob/d916c1f55e57a61b54135d0922ad8660735bd287/sdks/go/pkg/beam/core/runtime/harness/statemgr.go#L105

      List kept by ScopedStateReader: https://github.com/apache/beam/blob/d916c1f55e57a61b54135d0922ad8660735bd287/sdks/go/pkg/beam/core/runtime/harness/statemgr.go#L39

      Attachments

        Issue Links

          Activity

            People

              jrmccluskey Jack McCluskey
              jrmccluskey Jack McCluskey
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m