Details
-
Improvement
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
-
None
Description
We can add it as an optional field to beam.SideInput (see: pkg/beam/option.go). The execution side needs to support it as well. However, with the various side input forms, it's not clear how valuable such a feature would be.
It is strongly recommended to add Map Side Inputs https://issues.apache.org/jira/browse/BEAM-3293 before implementing this suggestion, and required to have caching implemented https://issues.apache.org/jira/browse/BEAM-11097. Otherwise very little benefit is acheived.
See https://issues.apache.org/jira/browse/BEAM-3293 for where code might need to be changed.
Make providing a ViewFn part of the beam.SideInput struct, either as a method to validate & update unexported fields, or adding an optional field.
The ViewFn would be a function that takes in the iterator/multimap version of the Side Input and returns whatever the pre-processed type is. That value would be what is cached to reduce work per common window.
The DoFn should then be able to use that type in the position of the ProcessElement method for that side input. This avoids issues that only work in of the Global Window like trying to process the side input in StartBundle. Something would need to be changed about handling the DoFn method signature analysis to support additional alternative side input types.