[BEAM-13212] [Go SDK] Add support for setting upper memory limit for side input cache

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: sdk-go
• Assignee: Unassigned
• Reporter: Jack McCluskey
• Created: 09/Nov/21 19:20
• Updated: 04/Jun/22 21:40

Description

This is implementing https://issues.apache.org/jira/browse/BEAM-8297 for the Go SDK's side input cache. The current implementation reads the entire side input into memory and keeps track of how many distinct inputs to hold; however, each side input is not guaranteed to occupy the same amount of memory. An extra parameter should be created to be passed in to the SideInputCache (https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go) to set a maximum memory limit in place of a maximum element capacity, then add a hook (see https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/cache_hooks.go) and corresponding function to the harnessopts package (https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/harnessopts/cache.go) so users can set this value themselves. 

Issue Links

• is a child of: BEAM-11097 [Go SDK] Windowed Side Input Caching (Cross Bundle)