Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4586

Reduce large output segments directly from remote host

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None
    • Target Version/s:

      Description

      For some jobs, copying large output segments to the local host is inefficient. The reduce can construct iterators on remote hosts, provided the stream is restartable. This should reduce task latency by amortizing the cost of the data transfer over the entire reduce, rather than paying it upfront.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cdouglas Christopher Douglas
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: