Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2973

Deadlock in hdfs sink

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.7.0
    • 1.9.0
    • Sinks+Sources

    Description

      Automatic close of BucketWriters (when open file count reached hdfs.maxOpenFiles) and the file rolling thread can end up in deadlock.

      When creating a new BucketWriter in HDFSEventSink it locks HDFSEventSink.sfWritersLock and the close() called in HDFSEventSink.sfWritersLock.removeEldestEntry tries to lock the BucketWriter instance.
      On the other hand if the file is being rolled in BucketWriter.close(boolean) it locks the BucketWriter instance first and in the close callback it tries to lock the sfWritersLock.

      The chances for this deadlock is higher when the hdfs.maxOpenFiles's value is low (1).

      Script to reproduce: https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff
      (put to flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs)
      Deadlock usually occurs before ~30 iterations.

      Attachments

        1. FLUME-2973-min2.patch
          2 kB
          Endre Major
        2. FLUME-2973-1.patch
          22 kB
          Yan Jian
        3. FLUME-2973.patch
          17 kB
          Denes Arvay

        Activity

          People

            denes Denes Arvay
            denes Denes Arvay
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: