[BEAM-8804] PCollectionList support in cross-language transforms

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: beam-model
• Assignee: Unassigned
• Reporter: Heejong Lee
• Created: 21/Nov/19 22:01
• Updated: 04/Jun/22 13:50

Description

Currently, Beam model doesn't have any information on the order of input/output PCollections from PTransforms. Therefore, PCollectionList needs to be converted to PCollectionTuple when it goes across the cross-language boundaries (or even in the same language, whenever it is converted between in-memory object and proto) and it's impossible to recreate PCollectionList from proto with the original order. The possible workaround is just to use PCollectionTuple with integer id (starting from 0 like indexes) instead of PCollectionList. In that case, we should first well-define how we generate proto from PCollectionList since each SDK uses a different convention.