[YARN-4301] NM disk health checker should have a timeout

• Type: Sub-task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: nodemanager
• Labels: oct16-medium
• Assignee: Akihiro Suda
• Reporter: Akihiro Suda
• Created: 27/Oct/15 05:20
• Updated: 27/Oct/16 19:04

Description

The disk health checker verifies a disk by executing mkdir and rmdir periodically.
If these operations does not return in a moderate timeout, the disk should be marked bad, and thus nodeInfo.nodeHealthy should flip to false.

I confirmed that current YARN does not have an implicit timeout (on JDK7, Linux 4.2, ext4) using Earthquake, our fault injector for distributed systems.
(I'll introduce the reproduction script in a while)

I consider we can fix this issue by making NodeHealthCheckerServer.isHealthy() return false if the value of this.getLastHealthReportTime() is too old.

Issue Links

• is related to: YARN-4426 unhealthy disk makes NM LOST