Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2131

Speculative execution should be allowed for reducers only

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Hadoop job, map fetches data from external systems

      Description

      Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.

      Speculative execution only on reducers would be a great way to solve this problem.

        Attachments

        1. patch-2131.txt
          8 kB
          Arun Murthy
        2. patch-2131.txt
          8 kB
          Amareshwari Sriramadasu
        3. patch-2131.txt
          7 kB
          Amareshwari Sriramadasu

          Activity

            People

            • Assignee:
              amareshwari Amareshwari Sriramadasu
              Reporter:
              srikantk Srikanth Kakani
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: