[REEF-1157] Improve memory usage and performance of the Java group communications code

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: GroupCommunications
• Labels: GSOC gsoc2016 java
• Assignee: Unassigned
• Reporter: Markus Weimer
• Created: 26/Jan/16 20:37
• Updated: 05/Mar/16 04:25

Description

The Group Communications library in .NET has seen two areas of improvement that we should follow up on in the Java code:

• Zero copies of the data being sent: the current code takes the user data, serializes it and then wraps it with the operator's envelope. That package is serialized and wrapped in the group communications envelope. Again, this is serialized and wrapped in the network service's envelope. That is what is eventually serialized and sent. All of these wrappings and serializations create copies of the (potentially large) user data. We should move to a zero-copy regime where the data is only serialized once.
• Pipelining: Many times, group communications is applied with vector types, e.g. when summing up double[]. Those operations can be done on a per-dimension basis, which reduces the memory load and improves throughput.

This JIRA shall serve as an anchor for work done in this area.