Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2131

Speculative execution should be allowed for reducers only

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 0.16.0
    • None
    • None
    • Hadoop job, map fetches data from external systems

    Description

      Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.

      Speculative execution only on reducers would be a great way to solve this problem.

      Attachments

        1. patch-2131.txt
          8 kB
          Arun Murthy
        2. patch-2131.txt
          8 kB
          Amareshwari Sriramadasu
        3. patch-2131.txt
          7 kB
          Amareshwari Sriramadasu

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            amareshwari Amareshwari Sriramadasu
            srikantk Srikanth Kakani
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment