Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3278

0.20: avoid a busy-loop in ReduceTask scheduling

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 1.1.0
    • Component/s: mrv1, performance, task
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Looking at profiling results, it became clear that the ReduceTask has the following busy-loop which was causing it to suck up 100% of CPU in the fetch phase in some configurations:

      • the number of reduce fetcher threads is configured to more than the number of hosts
      • therefore "busyEnough()" never returns true
      • the "scheduling" portion of the code can't schedule any new fetches, since all of the pending fetches in the mapLocations buffer correspond to hosts that are already being fetched (the hosts are in the uniqueHosts map)
      • getCopyResult() immediately returns null, since there are no completed maps.
        Hence ReduceTask spins back and forth between trying to schedule things (and failing), and trying to grab completed results (of which there are none), with no waits.
      1. reducer-cpu-usage.png
        99 kB
        Todd Lipcon
      2. mr-3278.txt
        4 kB
        Todd Lipcon

        Activity

        Todd Lipcon created issue -
        Todd Lipcon made changes -
        Field Original Value New Value
        Attachment reducer-cpu-usage.png [ 12501031 ]
        Todd Lipcon made changes -
        Attachment mr-3278.txt [ 12501034 ]
        Eli Collins made changes -
        Target Version/s 0.20.206.0 [ 12317960 ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.20.206.0 [ 12317960 ]
        Resolution Fixed [ 1 ]
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development