Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3289

Make use of fadvise in the NM's shuffle handler

    Details

      Description

      Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.

      1. MAPREDUCE-3289.branch-1.patch
        4 kB
        Brandon Li
      2. MR3289_trunk_3.txt
        13 kB
        Siddharth Seth
      3. MR3289_trunk_2.txt
        13 kB
        Siddharth Seth
      4. MR3289_trunk.txt
        9 kB
        Siddharth Seth
      5. MAPREDUCE-3289.branch-1.patch
        4 kB
        Brandon Li
      6. 3289-2.txt
        4 kB
        Todd Lipcon
      7. 3289-1.txt
        3 kB
        Todd Lipcon
      8. mr-3289.txt
        5 kB
        Todd Lipcon

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Fix Version/s 1.1.0 [ 12317960 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Fix Version/s 2.2.0-alpha [ 12322471 ]
          Resolution Fixed [ 1 ]
          Brandon Li made changes -
          Attachment MAPREDUCE-3289.branch-1.patch [ 12538856 ]
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Siddharth Seth made changes -
          Attachment MR3289_trunk_3.txt [ 12538853 ]
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Siddharth Seth made changes -
          Attachment MR3289_trunk_2.txt [ 12538835 ]
          Siddharth Seth made changes -
          Attachment MR3289_trunk.txt [ 12538129 ]
          Brandon Li made changes -
          Attachment MAPREDUCE-3289.branch-1.patch [ 12537761 ]
          Todd Lipcon made changes -
          Attachment 3289-1.txt [ 12536728 ]
          Attachment 3289-2.txt [ 12536729 ]
          Robert Joseph Evans made changes -
          Target Version/s 0.23.1 [ 12318883 ] 2.0.0, 3.0.0 [ 12320354, 12320355 ]
          Arun C Murthy made changes -
          Priority Blocker [ 1 ] Major [ 3 ]
          Arun C Murthy made changes -
          Target Version/s 0.23.0 [ 12315570 ] 0.23.1 [ 12318883 ]
          Priority Major [ 3 ] Blocker [ 1 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Todd Lipcon made changes -
          Link This issue is part of HADOOP-7714 [ HADOOP-7714 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment mr-3289.txt [ 12501200 ]
          Todd Lipcon created issue -

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development