Description
After HAMA-521 is done, we can add a spilling queue which just holds the messages in RAM that fit into the heap space. The rest can be flushed to disk.
We may call this a HybridQueue or something like that.
The benefits should be that we don't have to flush to disk so often and get faster. However we may have more GC so it is always overall faster.
The requirements for this queue also include:
- The message object once written to the queue (after returning from the write call) could be modified, but the changes should not be reflected in the messages stored in the queue.
- For now let's implement a queue that does not support concurrent reading and writing. This feature is needed when we implement asynchronous communication.
Attachments
Attachments
Issue Links
- is a clone of
-
HAMA-552 Add a sorted message queue
- Closed
- is blocked by
-
HAMA-521 Improve message buffering to save memory
- Resolved
- is depended upon by
-
HAMA-704 Optimization of memory usage during message processing
- Resolved
-
HAMA-706 Keep new object creation to the minimum by reusing the Writable object
- Resolved
- is related to
-
MAPREDUCE-4755 Rewrite MapOutputBuffer to use direct buffers & allow parallel sort+collect
- Resolved
- relates to
-
HAMA-644 Messaging Scalability Improvement
- Resolved