[REEF-154] .NET Group Communication improvements

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: REEF.NET
• Assignee: Julia Wang
• Reporter: Anthony Majestro
• Created: 10/Feb/15 21:47
• Updated: 07/Aug/18 20:12
• Resolved: 07/Aug/18 20:12

Description

Improvements include

• Reduce number of serialization layers .NET Wake
• Support binary tree topology
• Don't pull all of the values to be reduced into memory at once for flat topology; stagger them to not run out of memory for large number of nodes
• Pipelining for aggregation trees
• Elasticity and fault tolerance
• Simplify API
• Use Tang Names instead of strings for configuration of names (Group name, Operator Name, Task ID)
• Use types to be injected instead of instantiated objects in Operator Specifications
• Use .NET async features when applicable for network

Issue Links

• incorporates: REEF-166 Support tree topology in Group Communication at .Net
• incorporates: REEF-175 Using injection in tests for MpiDriver
• incorporates: REEF-162 Adding MPI tests
• incorporates: REEF-174 Use type instead of instance for Codec in MpiDriver
• incorporates: REEF-329 Properly use Tang Injection in Group Communication
• incorporates: REEF-173 Simplyfy Group Communication Driver API

Sub-Tasks

• We want to introduce pipelining in our REEF Group Communication operators: Resolved
• Refactor Org.Apache.REEF.Network.Group.Driver.Impl.TaskStarter: Closed
• Rename Mpi in Group Communication: Resolved