[HDFS-4504] DFSOutputStream#close doesn't always release resources (such as leases)

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Labels: BB2015-05-TBR
• Target Version/s: 2.1.0-beta
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 15/Feb/13 06:40
• Updated: 06/May/15 03:33

Description

DFSOutputStream#close can throw an IOException in some cases. One example is if there is a pipeline error and then pipeline recovery fails. Unfortunately, in this case, some of the resources used by the DFSOutputStream are leaked. One particularly important resource is file leases.

So it's possible for a long-lived HDFS client, such as Flume, to write many blocks to a file, but then fail to close it. Unfortunately, the LeaseRenewerThread inside the client will continue to renew the lease for the "undead" file. Future attempts to close the file will just rethrow the previous exception, and no progress can be made by the client.

Issue Links

• blocks: HDFS-4688 DFSClient should not allow multiple concurrent creates for the same file
• is duplicated by: HDFS-4855 DFSOutputStream reference should be cleared from DFSClient#filesBeingWritten if the file closure fails.
• is related to: HDFS-6652 RecoverLease cannot succeed and file cannot be closed under high load