[BEAM-3537] Remove DirectRunner-specific internal PValue cache, allow more general eager in-process pipeline execution

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: sdk-py-core
• Assignee: Charles Chen
• Reporter: Charles Chen
• Created: 25/Jan/18 23:21
• Updated: 16/May/20 13:54
• Resolved: 27/Jun/18 00:09

Description

Currently, the Python SDK supports an eager execution mode.  For example, a list can be directly passed into a PTransform to obtain its result:

result = [1, 2, 3] | MyPTransform()

To support this use, the Python DirectRunner has an option to cache its intermediate results into a PValueCache.  The above line, when run, implicitly creates an ephemeral pipeline and runs it with the DirectRunner.  This, however, adds a lot of complexity to the DirectRunner, and is not generalizable to other in-process Python runners (like the in-process Python FnApiRunner).

To improve this, we should remove this DirectRunner-specific implementation and add functionality that allows all in-process Python runners to be run in eager mode.

Issue Links

• links to: GitHub Pull Request #4492