Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3289

Make use of fadvise in the NM's shuffle handler

    Details

      Description

      Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.

      1. mr-3289.txt
        5 kB
        Todd Lipcon
      2. 3289-1.txt
        3 kB
        Todd Lipcon
      3. 3289-2.txt
        4 kB
        Todd Lipcon
      4. MAPREDUCE-3289.branch-1.patch
        4 kB
        Brandon Li
      5. MR3289_trunk.txt
        9 kB
        Siddharth Seth
      6. MR3289_trunk_2.txt
        13 kB
        Siddharth Seth
      7. MR3289_trunk_3.txt
        13 kB
        Siddharth Seth
      8. MAPREDUCE-3289.branch-1.patch
        4 kB
        Brandon Li

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment mr-3289.txt [ 12501200 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Link This issue is part of HADOOP-7714 [ HADOOP-7714 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Target Version/s 0.23.0 [ 12315570 ] 0.23.1 [ 12318883 ]
          Priority Major [ 3 ] Blocker [ 1 ]
          Arun C Murthy made changes -
          Priority Blocker [ 1 ] Major [ 3 ]
          Robert Joseph Evans made changes -
          Target Version/s 0.23.1 [ 12318883 ] 2.0.0, 3.0.0 [ 12320354, 12320355 ]
          Todd Lipcon made changes -
          Attachment 3289-1.txt [ 12536728 ]
          Attachment 3289-2.txt [ 12536729 ]
          Brandon Li made changes -
          Attachment MAPREDUCE-3289.branch-1.patch [ 12537761 ]
          Siddharth Seth made changes -
          Attachment MR3289_trunk.txt [ 12538129 ]
          Siddharth Seth made changes -
          Attachment MR3289_trunk_2.txt [ 12538835 ]
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Siddharth Seth made changes -
          Attachment MR3289_trunk_3.txt [ 12538853 ]
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Brandon Li made changes -
          Attachment MAPREDUCE-3289.branch-1.patch [ 12538856 ]
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Fix Version/s 2.2.0-alpha [ 12322471 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 1.1.0 [ 12317960 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development