[FLUME-2973] Deadlock in hdfs sink

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.9.0
• Component/s: Sinks+Sources
• Labels: hdfssink
• Assignee: Denes Arvay
• Reporter: Denes Arvay
• Created: 17/Aug/16 13:37
• Updated: 12/Oct/18 12:13
• Resolved: 12/Oct/18 12:13

Description

Automatic close of BucketWriters (when open file count reached hdfs.maxOpenFiles) and the file rolling thread can end up in deadlock.

When creating a new BucketWriter in HDFSEventSink it locks HDFSEventSink.sfWritersLock and the close() called in HDFSEventSink.sfWritersLock.removeEldestEntry tries to lock the BucketWriter instance.
On the other hand if the file is being rolled in BucketWriter.close(boolean) it locks the BucketWriter instance first and in the close callback it tries to lock the sfWritersLock.

The chances for this deadlock is higher when the hdfs.maxOpenFiles's value is low (1).

Script to reproduce: https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff
(put to flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs)
Deadlock usually occurs before ~30 iterations.

Issue Links

• links to: GitHub Pull Request #226
• links to: Reviewboard