Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4381

Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task, tasktracker
    • Labels:
      None

      Description

      Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We tried making it a tunable and experimented with different values. In some cases setting it to a smaller value like 1000 msec helps significantly improve performance of short running jobs such as piEstimator. This is because the task threads do not end up blocking for as many as 3 seconds for their last progress update event. We also noticed close to 14% improvement on Mahout KMeans iteration jobs which take more than 5 minutes on the test cluster that we are using. Please let me know if this seems to be a good idea. I have an initial patch that I have attached here. This is based on branch-1 tree. It may need some rework on MRv2 based branches I think. Also note that I have not changed the variable naming style for PROGRESS_INTERVAL even though it is not a public static final anymore. I can revise the patch if there are no objections to this idea.
      Thanks.

        Attachments

        1. progress_interval.patch
          2 kB
          Shrinivas Joshi
        2. MAPREDUCE-4381-branch-1.patch
          3 kB
          Shrinivas Joshi

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jshrinivas Shrinivas Joshi

              Dates

              • Created:
                Updated:

                Issue deployment