Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1965

Handle map output buffers better

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.16.0
    • 0.16.0
    • None
    • None

    Description

      Today, the map task stops calling the map method while sort/spill is using the (single instance of) map output buffer. One improvement that can be done to improve performance of the map task is to have another buffer for writing the map outputs to, while sort/spill is using the first buffer.

      Attachments

        1. 1965_single_proc_150mb_gziped.pdf
          2 kB
          Amar Kamat
        2. 1965_single_proc_150mb_gziped_breakup.png
          5 kB
          Amar Kamat
        3. 1965_single_proc_150mb_gziped.jpeg
          81 kB
          Amar Kamat
        4. HADOOP-1965-1.patch
          8 kB
          Amar Kamat
        5. HADOOP-1965-Benchmark.patch
          24 kB
          Amar Kamat
        6. HADOOP-1965-Benchmark.patch
          27 kB
          Amar Kamat
        7. HADOOP-1965-Benchmark.patch
          28 kB
          Amar Kamat
        8. HADOOP-1965-Benchmark.patch
          28 kB
          Amar Kamat
        9. HADOOP-1965-Benchmark.patch
          28 kB
          Amar Kamat
        10. HADOOP-2419.patch
          29 kB
          Amar Kamat
        11. HADOOP-2419.patch
          29 kB
          Amar Kamat
        12. HADOOP-2419.patch
          34 kB
          Amar Kamat
        13. HADOOP-2419.patch
          30 kB
          Amar Kamat

        Activity

          People

            amar_kamat Amar Kamat
            ddas Devaraj Das
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: