[BEAM-9964] Setting workerCacheMb to make its way to the WindmillStateCache Constructor

• Type: Improvement
• Status: Triage Needed
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.22.0
• Component/s: runner-dataflow
• Assignee: Omar Ismail
• Reporter: Omar Ismail
• Created: 12/May/20 15:27
• Updated: 13/Apr/23 11:01
• Resolved: 18/May/20 18:06

Description

Setting --workerCacheMB seems to affect batch pipelines only. For Streaming, the cache seems to be hardcoded to 100Mb [1]. If possible, I would like to make it allowable to change the cache value in Streaming when setting -workerCacheMB.

I've never made changes to the Beam SDK, so I am super excited to work on this! 

 

[1] https://github.com/apache/beam/blob/5e659bb80bcbf70795f6806e05a255ee72706d9f/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateCache.java#L73

Issue Links

• links to: GitHub Pull Request #11710
• links to: GitHub Pull Request #11741
• links to: GitHub Pull Request #11743
• links to: GitHub Pull Request #11849