[BEAM-13130] [Go SDK] Side Input Memory Leak

• Type: Bug
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.35.0
• Component/s: sdk-go
• Assignee: Jack McCluskey
• Reporter: Jack McCluskey
• Created: 27/Oct/21 15:22
• Updated: 13/Apr/23 11:13
• Resolved: 28/Oct/21 17:26

Description

Keeping a list of the open stateKeyReader structs in the ScopedStateReader is holding on to memory for the life of bundles, leading to significant memory usage in large batch jobs. This list needs to be removed to the garbage collector can clean up stateKeyReader structs sooner.

 

stateKeyReader: https://github.com/apache/beam/blob/d916c1f55e57a61b54135d0922ad8660735bd287/sdks/go/pkg/beam/core/runtime/harness/statemgr.go#L105

List kept by ScopedStateReader: https://github.com/apache/beam/blob/d916c1f55e57a61b54135d0922ad8660735bd287/sdks/go/pkg/beam/core/runtime/harness/statemgr.go#L39

Issue Links

• links to: GitHub Pull Request #15815