[YARN-5078] [Umbrella] NodeManager health checker improvements

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: nodemanager
• Labels: supportability
• Assignee: Ray Chiang
• Reporter: Ray Chiang
• Created: 12/May/16 21:00
• Updated: 12/May/16 21:00

Description

There have been a bunch of NodeManager health checker improvement requests in the past.

Right now, I expect that initially there just need to be a bunch of base functionality added. The most obvious parts are:

• Finding appropriate measurements of health
• Storing measurements as metrics. This should allow easy comparison of good nodes and bad nodes. This should eventually lead to threshold blacklisting/whitelisting.
• Adding metrics to the NodeManager UI

After this basic functionality is added, we can start consider some enhanced form of NodeManager health status conditions.

Sub-Tasks

• Make the NodeManager's health checker service pluggable: Open
• NodeManager not blacklisting the disk (shuffle) with errors: Open
• Make "good" local directories available to ContainerExecutors at initialization time: Open
• Need a default NodeManager health check script: Resolved
• "Health-Report" column of NodePage should display more information.: Open
• NM disk health checker should have a timeout: Patch Available
• Make DiskChecker pluggable in NodeManager: Resolved
• Use smartctl to determine health of disks: Open
• Create new DiskValidator class with metrics: Resolved
• Define exit code for allowing NodeManager health script to mar: Resolved
• Better handling when bad script is configured as Node's HealthScript: Resolved
• Add default value for NM disk validator: Resolved