Uploaded image for project: 'Hama'
  1. Hama
  2. HAMA-644 Messaging Scalability Improvement
  3. HAMA-559

Add a spilling message queue

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.5.0
    • 0.6.1
    • bsp core
    • None

    Description

      After HAMA-521 is done, we can add a spilling queue which just holds the messages in RAM that fit into the heap space. The rest can be flushed to disk.
      We may call this a HybridQueue or something like that.

      The benefits should be that we don't have to flush to disk so often and get faster. However we may have more GC so it is always overall faster.

      The requirements for this queue also include:

      • The message object once written to the queue (after returning from the write call) could be modified, but the changes should not be reflected in the messages stored in the queue.
      • For now let's implement a queue that does not support concurrent reading and writing. This feature is needed when we implement asynchronous communication.

      Attachments

        1. HAMA-559.patch-v4
          69 kB
          Suraj Menon
        2. HAMA-559.patch-v2
          52 kB
          Suraj Menon
        3. spillbench_code.tar.gz
          17 kB
          Suraj Menon
        4. spilling_buffer_profile_LOCK.JPG
          20 kB
          Thomas Jungblut
        5. SpillingBufferProfile-2012-10-27.snapshot
          620 kB
          Thomas Jungblut
        6. spilling_buffer_profile_writeUTF.png
          142 kB
          Suraj Menon
        7. spilling_buffer_profile_cpugraph_writeUTF.png
          131 kB
          Suraj Menon
        8. spillingbuffer_profile_cpu_writeUTF.png
          128 kB
          Suraj Menon
        9. spilling_buffer_profile_timesplit_text_write.png
          192 kB
          Suraj Menon
        10. spilling_buffer_profile_cpu_graph_test_write.png
          131 kB
          Suraj Menon
        11. spilling_buffer_cpu_usage_text_write.png
          136 kB
          Suraj Menon
        12. HAMA-559.patch-v1
          34 kB
          Suraj Menon

        Issue Links

          is a clone of

          New Feature - A new feature of the product, which has yet to be developed. HAMA-552 Add a sorted message queue

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is blocked by

          Sub-task - The sub-task of the issue HAMA-521 Improve message buffering to save memory

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HAMA-704 Optimization of memory usage during message processing

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HAMA-706 Keep new object creation to the minimum by reusing the Writable object

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4755 Rewrite MapOutputBuffer to use direct buffers & allow parallel sort+collect

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HAMA-644 Messaging Scalability Improvement

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              surajsmenon Suraj Menon
              thomas.jungblut Thomas Jungblut
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: