Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3289

Make use of fadvise in the NM's shuffle handler

    Details

      Description

      Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.

      1. 3289-1.txt
        3 kB
        Todd Lipcon
      2. 3289-2.txt
        4 kB
        Todd Lipcon
      3. MAPREDUCE-3289.branch-1.patch
        4 kB
        Brandon Li
      4. MAPREDUCE-3289.branch-1.patch
        4 kB
        Brandon Li
      5. MR3289_trunk_2.txt
        13 kB
        Siddharth Seth
      6. MR3289_trunk_3.txt
        13 kB
        Siddharth Seth
      7. MR3289_trunk.txt
        9 kB
        Siddharth Seth
      8. mr-3289.txt
        5 kB
        Todd Lipcon

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development