Uploaded image for project: 'REEF'
  1. REEF
  2. REEF-1157

Improve memory usage and performance of the Java group communications code

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: GroupCommunications
    • Labels:

      Description

      The Group Communications library in .NET has seen two areas of improvement that we should follow up on in the Java code:

      • Zero copies of the data being sent: the current code takes the user data, serializes it and then wraps it with the operator's envelope. That package is serialized and wrapped in the group communications envelope. Again, this is serialized and wrapped in the network service's envelope. That is what is eventually serialized and sent. All of these wrappings and serializations create copies of the (potentially large) user data. We should move to a zero-copy regime where the data is only serialized once.
      • Pipelining: Many times, group communications is applied with vector types, e.g. when summing up double[]. Those operations can be done on a per-dimension basis, which reduces the memory load and improves throughput.

      This JIRA shall serve as an anchor for work done in this area.

        Activity

        Hide
        amil101 amil shanaka added a comment - - edited

        Hi,
        I'm a 3rd year under graduate in the department of Computer Science and Engineering at University of Moratuwa,Sri Lanka. In my experience, I've been worked with Java nio.ByteBuffer and also off-heap memory usage like java Unsafe, nio.directByteBuffer and netty Direct Byte Buf for enhancing the performance with communicating data. So i would like to know more details regrading this project in detail that I could proceed .
        Thank you,
        amil

        Show
        amil101 amil shanaka added a comment - - edited Hi, I'm a 3rd year under graduate in the department of Computer Science and Engineering at University of Moratuwa,Sri Lanka. In my experience, I've been worked with Java nio.ByteBuffer and also off-heap memory usage like java Unsafe, nio.directByteBuffer and netty Direct Byte Buf for enhancing the performance with communicating data. So i would like to know more details regrading this project in detail that I could proceed . Thank you, amil

          People

          • Assignee:
            Unassigned
            Reporter:
            markus.weimer Markus Weimer
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development