[KUDU-3531] Limit the amount of resources used by tombstoned tablet replicas

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: scalability tablet
• Assignee: Unassigned
• Reporter: Alexey Serbin
• Created: 06/Dec/23 22:41
• Updated: 11/Dec/23 20:27

Description

I came across a case where a tablet server had just about 2K live tablet replicas, but it opened about 24K files in its WAL and data directories. The issue stems from the fact that tombstoned tablet replica's files are opened by the FS manager the same as for a live replica, and those are kept open even if they are never about to change. It would be prudent to avoid keeping tombstoned tablet replicas' files open, if possible: maybe, just read the required information (last voted term and opId index?) and keep it in runtime structures, but close corresponding files right after bootstrapping?

Otherwise, this doesn't seem to scale well.

Issue Links

• relates to: KUDU-3486 Tserver: Too many tombstone tablet may lead to high memory usage.