[KUDU-3025] Add metric for the open file descriptors usage vs the limit

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: master, metrics, tserver
• Labels: Availability observability scalability
• Assignee: Unassigned
• Reporter: Alexey Serbin
• Created: 17/Dec/19 07:20
• Updated: 03/Jun/20 15:59

Description

In the case of even replica distribution across all available nodes, once one tablet server hits the maximum number of open file descriptors and go down (e.g., upon hosting another tablet replica), the system will automatically re-replicate tablet replicas from the tablet server, most likely bringing other tablet servers down as well. That's a cascading failure scenario that nobody wants to experience.

Monitoring the number of open file descriptors vs the limit can help to prevent full Kudu cluster outage in such case, if operators are given a chance to handle those situations proactively. Once some threshold is reached (e.g., 90%), an operator could update the limit via corresponding ulimit setting, preventing an outage.

Issue Links

• is related to: KUDU-3026 tserver: refuse to host another tablet replica if the number of open file descriptors is close to the limit