[HADOOP-3585] Hardware Failure Monitoring in large clusters running Hadoop/HDFS

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.19.0
• Component/s: metrics

Environment

Linux

• Hadoop Flags: Reviewed
• Release Note: Added FailMon as a contrib project for hardware failure monitoring and analysis, under /src/contrib/failmon. Created User Manual and Quick Start Guide.
• Assignee: Ioannis Koltsidas
• Reporter: Ioannis Koltsidas
• Created: 17/Jun/08 21:00
• Updated: 20/Nov/08 23:38
• Resolved: 15/Aug/08 09:04

Description

At IBM we're interested in identifying hardware failures on large clusters running Hadoop/HDFS. We are working on a framework that will enable nodes to identify failures on their hardware using the Hadoop log, the system log and various OS hardware diagnosing utilities. The implementation details are not very clear, but you can see a draft of our design in the attached document. We are pretty interested in Hadoop and system logs from failed machines, so if you are in possession of such, you are very welcome to contribute them; they would be of great value for hardware failure diagnosing.

Some details about our design can be found in the attached document failmon.doc. More details will follow in a later post.

Issue Links

• is related to: HADOOP-3964 javadoc warnings by failmon