[SLIDER-1246] Application health should not be affected by faulty nodes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Slider 0.92
• Fix Version/s: Slider 1.0.0
• Component/s: appmaster, core
• Assignee: Gour Saha
• Reporter: Prasanth Jayachandran
• Created: 05/Sep/17 22:20
• Updated: 04/Oct/17 07:54
• Resolved: 02/Oct/17 05:27

Description

In case of a faulty node, multiple container failures will be deemed as an application failure.
Observed this in HIVE-16927, where container failures in certain nodes brings down entire application. Slider has to provide a way to not mark application as unhealthy if certain threshold of containers are running. Tuning failure threshold is not optimal as setting the correct default on large cluster is not trivial. Beyond certain failures, slider should mark the node as unhealthy and report that back to client/AM. Application could continue to run as long as container request is satisfied partially (example: 80% containers are running).

Issue Links

• blocks: HIVE-16927 LLAP: Slider takes down all daemons when some daemons fail repeatedly

Sub-Tasks

• Tests for Health Threshold Monitoring Feature: Resolved