[BEAM-12204] Portable Java caches entirety of iterable side inputs.

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.30.0
• Component/s: java-fn-execution
• Assignee: Robert Bradshaw
• Reporter: Robert Bradshaw
• Created: 21/Apr/21 16:37
• Updated: 29/Apr/21 15:53
• Resolved: 29/Apr/21 15:53

Description

Though the data is read over the state channel in a paginated way, it is wrapped in LazyCachingIteratorToIterable which forces the entire iterable to be cached. On the other hand, not caching at all for re-iteration could be expensive. We should do what Python does and cache a bit (e.g. the first page) and still support re-iteration by re-requesting the data over the state channel.

Issue Links

• links to: GitHub Pull Request #14604