[HDFS-14575] LeaseRenewer#daemon threads leak in DFSClient

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.4.0, 3.2.3, 3.3.2, 3.2.4
• Component/s: dfsclient
• Target Version/s: 3.4.0, 3.2.3, 3.3.2, 3.2.4
• Hadoop Flags: Reviewed
• Assignee: Renukaprasad C
• Reporter: Tao Yang
• Created: 17/Jun/19 11:37
• Updated: 11/Feb/24 14:19
• Resolved: 21/Jun/21 16:38

Description

Currently LeaseRenewer (and its daemon thread) without clients should be terminated after a grace period which defaults to 60 seconds. A race condition may happen when a new request is coming just after LeaseRenewer expired.
Reproduce this race condition:

1. Client#1 creates File#1: creates LeaseRenewer#1 and starts Daemon#1 thread, after a few seconds, File#1 is closed , there is no clients in LeaseRenewer#1 now.
2. 60 seconds (grace period) later, LeaseRenewer#1 just expires but daemon#1 thread is still in sleep, Client#1 creates File#2, lead to the creation of Daemon#2.
3. Daemon#1 is awake then exit, after that, LeaseRenewer#1 is removed from factory.
4. File#2 is closed after a few seconds, LeaseRenewer#2 is created since it can’t get renewer from factory.

Daemon#2 thread leaks from now on, since Client#1 in it can never be removed and it won't have a chance to stop.

To solve this problem, IIUIC, a simple way I think is to make sure that all clients are cleared when LeaseRenewer is removed from factory. Please feel free to give your suggestions. Thanks!

Issue Links

• is related to: HDFS-16235 Deadlock in LeaseRenewer for static remove method