Hey Luke, Thanks for the comments
# Have you tried to benchmark the patch at scale? Calling job.getCounters in completeJob would bring down a busy JT on a large cluster to its knee. Think about calling getCounters (which is essentially a O( n ) operation) a few hundred times per second!
You are right about that getCounters(). The method is really expansive.
But here we do this in JobTrackerMetricsInst.doUpdates() which is called only every 5 seconds. So it has very minor impact on JT performance.
We have put this on our 3000 nodes cluster that has many big jobs for months and it has been running fine.
The necessity of having these total aggregate counts in real time. Rumen or other MR log processing tools can get these aggregates for performance analysis without impacting JT performance.
We have an internal tool that graphs these metrics on a dashboard. It is really useful in real-time debugging for the cluster issues. I believe Y! and other people also have similar use case.
If you really want these counters in real time, you should implement it in TT where it can send the metrics to distributed metrics aggregators with UDP etc. and can be easily disabled/enabled via the metrics system.
That sounds like a good solution too. But I like the current way better because it is very simple.
Anything we add in JobCounter and TaskCounter will automatically go to the metrics. We don't need to add more codes to make that happen.