[YARN-1297] FairScheduler: Move some logs to debug and check if debug logging is enabled

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: fairscheduler
• Hadoop Flags: Reviewed
• Assignee: Yufei Gu
• Reporter: Sandy Ryza
• Created: 11/Oct/13 07:38
• Updated: 30/Aug/16 01:32
• Resolved: 26/Apr/16 13:19

Description

I ran the Fair Scheduler's core scheduling loop through a profiler tool and identified a bunch of minimally invasive changes that can shave off a few milliseconds.

The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark down from 16000 ms to 6000.

A few others (which had way less of an impact) were

• Most of the time in comparisons was being spent in Math.signum. I switched this to direct ifs and elses and it halved the percent of time spent in comparisons.
• I removed some unnecessary instantiations of Resource objects
• I made it so that queues' usage wasn't calculated from the applications up each time getResourceUsage was called.

Issue Links

• is duplicated by: YARN-4090 Make Collections.sort() more efficient by caching resource usage
• relates to: YARN-305 Fair scheduler logs too many "Node offered to app:..." messages