[REEF-942] Local aggregation

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 0.14
• Component/s: Vortex
• Assignee: Andrew Chung
• Reporter: Zhengping Qian
• Created: 13/Nov/15 08:26
• Updated: 25/Feb/16 06:30
• Resolved: 03/Feb/16 18:34

Description

In many applications (e.g., Logistical Regression), the amount of data sent over network between tasklets and Vortex master is proportional to the number of tasklets. Such an I/O cost could potentially become a performance and/or scalability bottleneck for distributing a computation to run across many machines.

Similar to the cases in MapReduce, results from tasklets can often be combined or aggregated locally before sending back to reduce the total amount of I/O. In this (sub-)task, we explore the API and runtime enhancement for enabling such local aggregation whenever possible.

Issue Links

• incorporates: REEF-1075 Create Tasklet abstraction to enable one-to-many mapping of "Future-like" object to TaskletIds
• incorporates: REEF-1077 Allow VortexWorker reports to report about more than a single Tasklet
• incorporates: REEF-1084 Modify TaskletReport to allow aggregation of Tasklets for a single result/error
• incorporates: REEF-1129 Driver side VortexAggregate submission
• incorporates: REEF-1130 Worker side VortexAggregateFunction reception
• incorporates: REEF-1131 Create, define, and apply VortexAggregationPolicy
• incorporates: REEF-1132 Define VortexAggregateFuture and VortexAggregateFunction