Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.10.1
-
None
-
None
Description
Currently, operators are chained more aggressively when the parallelism is one. That makes debugging tougher as it changes threading behavior.
The benefits are also limited: Real installations where that type of efficiency would be needed would not run in parallelism 1, or would not use a partitioning/broadcast step there (if explicitly required to run parallelism 1).
In the future, when we want to allow parallelism to be adjusted dynamically, this will be even more tricky.