Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1906

Lower default minimum heartbeat interval for tasktracker > Jobtracker

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      The default minimum heartbeat interval has been dropped from 3 seconds to 300ms to increase scheduling throughput on small clusters. Users may tune mapreduce.jobtracker.heartbeats.in.second to adjust this value.

      Description

      I get a 0% to 15% performance increase for smaller clusters by making the heartbeat throttle stop penalizing clusters with less than 300 nodes.

      Between 0.19 and 0.20, the default minimum heartbeat interval increased from 2s to 3s. If a JobTracker is throttled at 100 heartbeats / sec for large clusters, why should a cluster with 10 nodes be throttled to 3.3 heartbeats per second?

      1. MAPREDUCE-1906.branch-1.patch
        1 kB
        Brandon Li
      2. mapreduce-1906.txt
        11 kB
        Todd Lipcon
      3. mapreduce-1906.txt
        11 kB
        Todd Lipcon
      4. MAPREDUCE-1906-0.21.patch
        1 kB
        Scott Carey

        Activity

        Arun C Murthy made changes -
        Fix Version/s 1.1.0 [ 12317960 ]
        Fix Version/s 1.2.0 [ 12321661 ]
        Matt Foley made changes -
        Summary Lower minimum heartbeat interval for tasktracker > Jobtracker Lower default minimum heartbeat interval for tasktracker > Jobtracker
        Release Note The minimum heartbeat interval has been dropped from 3 seconds to 300ms to increase scheduling throughput on small clusters. Users may tune mapreduce.jobtracker.heartbeats.in.second to adjust this value. The default minimum heartbeat interval has been dropped from 3 seconds to 300ms to increase scheduling throughput on small clusters. Users may tune mapreduce.jobtracker.heartbeats.in.second to adjust this value.
        Siddharth Seth made changes -
        Fix Version/s 1.2.0 [ 12321661 ]
        Brandon Li made changes -
        Affects Version/s 1.2.0 [ 12321661 ]
        Brandon Li made changes -
        Attachment MAPREDUCE-1906.branch-1.patch [ 12538084 ]
        Suresh Srinivas made changes -
        Component/s performance [ 12316500 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Todd Lipcon made changes -
        Release Note The minimum heartbeat interval has been dropped from 3 seconds to 300ms to increase scheduling throughput on large clusters. Users may tune mapreduce.jobtracker.heartbeats.in.second to adjust this value. The minimum heartbeat interval has been dropped from 3 seconds to 300ms to increase scheduling throughput on small clusters. Users may tune mapreduce.jobtracker.heartbeats.in.second to adjust this value.
        Component/s tasktracker [ 12312906 ]
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Release Note The minimum heartbeat interval has been dropped from 3 seconds to 300ms to increase scheduling throughput on large clusters. Users may tune mapreduce.jobtracker.heartbeats.in.second to adjust this value.
        Fix Version/s 0.22.0 [ 12314184 ]
        Fix Version/s 0.20.3 [ 12314813 ]
        Resolution Fixed [ 1 ]
        Eli Collins made changes -
        Component/s jobtracker [ 12312907 ]
        Eli Collins made changes -
        Fix Version/s 0.20.3 [ 12314813 ]
        Fix Version/s 0.22.0 [ 12314184 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Todd Lipcon made changes -
        Assignee Todd Lipcon [ tlipcon ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Todd Lipcon made changes -
        Attachment mapreduce-1906.txt [ 12466983 ]
        Todd Lipcon made changes -
        Attachment mapreduce-1906.txt [ 12466982 ]
        Scott Carey made changes -
        Attachment MAPREDUCE-1906-0.21-v2.patch [ 12449382 ]
        Scott Carey made changes -
        Attachment MAPREDUCE-1906-0.21.patch [ 12449679 ]
        Scott Carey made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Carey made changes -
        Attachment MAPREDUCE-1906-0.21.patch [ 12448507 ]
        Scott Carey made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Carey made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Carey made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Carey made changes -
        Attachment MAPREDUCE-1906-0.21-v2.patch [ 12449382 ]
        Scott Carey made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Carey made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Carey made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-1906-0.21.patch [ 12448507 ]
        Scott Carey created issue -

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Scott Carey
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development