Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4502

Node-level aggregation with combining the result of maps

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: None
    • Component/s: applicationmaster, mrv2
    • Labels:

      Description

      The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner.

      This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs.

        Attachments

        1. design_v2.pdf
          347 kB
          Tsuyoshi Ozawa
        2. design_v3.pdf
          440 kB
          Tsuyoshi Ozawa
        3. MAPREDUCE-4502.1.patch
          113 kB
          Tsuyoshi Ozawa
        4. MAPREDUCE-4502.10.patch
          109 kB
          Tsuyoshi Ozawa
        5. MAPREDUCE-4502.2.patch
          115 kB
          Tsuyoshi Ozawa
        6. MAPREDUCE-4502.3.patch
          115 kB
          Tsuyoshi Ozawa
        7. MAPREDUCE-4502.4.patch
          115 kB
          Tsuyoshi Ozawa
        8. MAPREDUCE-4502.5.patch
          115 kB
          Tsuyoshi Ozawa
        9. MAPREDUCE-4502.6.patch
          116 kB
          Tsuyoshi Ozawa
        10. MAPREDUCE-4502.7.patch
          117 kB
          Tsuyoshi Ozawa
        11. MAPREDUCE-4502.8.patch
          117 kB
          Tsuyoshi Ozawa
        12. MAPREDUCE-4502.8.patch
          117 kB
          Tsuyoshi Ozawa
        13. MAPREDUCE-4502.9.patch
          109 kB
          Tsuyoshi Ozawa
        14. MAPREDUCE-4502.9.patch
          109 kB
          Tsuyoshi Ozawa
        15. MAPREDUCE-4525-pof.diff
          71 kB
          Tsuyoshi Ozawa
        16. speculative_draft.pdf
          142 kB
          Tsuyoshi Ozawa

          Issue Links

            Activity

              People

              • Assignee:
                ozawa Tsuyoshi Ozawa
                Reporter:
                ozawa Tsuyoshi Ozawa
              • Votes:
                0 Vote for this issue
                Watchers:
                35 Start watching this issue

                Dates

                • Created:
                  Updated: