Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1906

Lower default minimum heartbeat interval for tasktracker > Jobtracker

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed
    • The default minimum heartbeat interval has been dropped from 3 seconds to 300ms to increase scheduling throughput on small clusters. Users may tune mapreduce.jobtracker.heartbeats.in.second to adjust this value.

    Description

      I get a 0% to 15% performance increase for smaller clusters by making the heartbeat throttle stop penalizing clusters with less than 300 nodes.

      Between 0.19 and 0.20, the default minimum heartbeat interval increased from 2s to 3s. If a JobTracker is throttled at 100 heartbeats / sec for large clusters, why should a cluster with 10 nodes be throttled to 3.3 heartbeats per second?

      Attachments

        1. MAPREDUCE-1906-0.21.patch
          1 kB
          Scott Carey
        2. mapreduce-1906.txt
          11 kB
          Todd Lipcon
        3. mapreduce-1906.txt
          11 kB
          Todd Lipcon
        4. MAPREDUCE-1906.branch-1.patch
          1 kB
          Brandon Li

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tlipcon Todd Lipcon Assign to me
            scott_carey Scott Carey
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment