Details
-
Improvement
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
-
None
Description
This is implementing https://issues.apache.org/jira/browse/BEAM-8297 for the Go SDK's side input cache. The current implementation reads the entire side input into memory and keeps track of how many distinct inputs to hold; however, each side input is not guaranteed to occupy the same amount of memory. An extra parameter should be created to be passed in to the SideInputCache (https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go) to set a maximum memory limit in place of a maximum element capacity, then add a hook (see https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/cache_hooks.go) and corresponding function to the harnessopts package (https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/harnessopts/cache.go) so users can set this value themselves.
Attachments
Issue Links
- is a child of
-
BEAM-11097 [Go SDK] Windowed Side Input Caching (Cross Bundle)
-
- Triage Needed
-