With the patch attached, I ran sort benchmarks on 390 node cluster and 120 node cluster. The performance is almost the same as with the trunk.
To simulate busyness at the job tracker, I ran the sort benchmarks on 120 node cluster with number of handlers=4 and with max queue size per handler =10, but there are drops and lost task trackers with the patch and without.
Thus Cluster size factor as (clusterSize/50+1) should be fine. But the busyFactor has to be better tuned.
I propose the following to tune busy factor:
We have threshouldDropCount = clusterSize/10;
We increment busyFactor by HEARTBEAT_BUSY_FACTOR (say 2secs) for every 10% cluster size drops.
if(dropCount > threshouldDropCount)
busyFactor += (dropCount/threshouldDropCount)*HEARTBEAT_BUSY_FACTOR;
For example, on a 100 node cluster, if we see 40 drops, busyFactor is incremented by 8 seconds (40/10*2).
If job tracker is not busy for 'observationInterval' , then we will decrement busyFactor by HEARTBEAT_BUSY_FACTOR;
To calculate observationInterval,
We have, 2 rpcs to be processed as at the jobtracker i.e. heartbeat and task completion events. let processing time per rpc be 2 seconds.
Here, observationInterval is calculated as:
observationInterval = (clusterSize/#handlers)*processingTime*2;
Assuming that we don't see drops at a certain observationInterval (and the corresponding busyFactor), we decrement the busyFactor by HEARTBEAT_BUSY_FACTOR. This can be done in a loop, until we see drops. When we see drops, we increment it by the constant HEARTBEAT_BUSY_FACTOR, and stabilize there .. until we see drops.
For example, On a 100 on cluster, We start with 2 seconds heartbeat interval.
We see 40 drops, then busyFactor = 8; then, new interval = (2+8) =10;
We dont see drops for 40 seconds; new interval = 10-2 =8;
We dont see drops for 40 seconds; new interval = 8-2 =6;
We dont see drops for 40 seconds; new interval = 6-2 =4;
We see drops; then new interval = 6;
We dont see drops for lone time, say. we stabilize here.
Say we see 30 drops after some time, busyFactor =6; new interval = 6+6 =12;
And the loop repeats.