[BEAM-4591] beam.Create should be splittable

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-java-core, sdk-py-core
• Labels: Done
• Assignee: Unassigned
• Reporter: Stephan Hoyer
• Created: 19/Jun/18 22:15
• Updated: 06/Oct/20 21:56
• Resolved: 19/Jun/18 23:23

Description

beam.Create() should be splittable. This would allow the unintuitive "Reshuffle" step below to be safely omitted:
 
pipeline = (
    beam.Create(range(large_number))
    | beam.Reshuffle()  # prevent task fusion
    | beam.Map(very_expensive_function)
    ...
)
 
These sort of pipelines with small inputs to expensive CPU bound tasks arise frequently in scientific computing use-cases.