Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Implemented
-
3.1.1-incubating
-
None
Description
Steps like NoOpCollectingBarrier and AggregateStep are Barrier-steps, but they do not need to reduce their results down to a single machine in OLAP. In fact, it is wildly expensive to do so. We should have a LocalBarrier interface which allows such steps to "barrier" but not reduce. What happens is the Steps will hold the data on the workers, merge "nothing" to master. Then master will respawn "nothing" back to the workers and the steps continue their execution. In this way, we know the barriers maintain their synchronization, but there is not unnecessary data shuffling around the cluster.
-
-
- Note that SupplyBarrierStep is analogous to this where its "nothing" reduction is simply a single boolean that gets and'd but means nothing.
-