Flume
  1. Flume
  2. FLUME-1610

HDFSEventSink and bucket writer have a race condition

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.3.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      I have seen a scenerio where an exception was thrown during HDFSEventSink.process when the flush on the bucket writer was called but the BucketWriter was already closed.

      Assumptions:
      1) In HDFSEventSink.process when done, we flush all buckets written to once channel returns null or batch size is exceeded
      2) The BucketWriter.flush method does not check the isOpen flag.
      3) Our time roll interval code assumes the next call to the bucket writer will be append as such the isOpen flag will be checked and the underlying writer re-opened.

      As such, I think what is happening is this:
      1) In HDFSEventSink.process the bucket writer is written to
      2) In BucketWriter the time based roll trips
      3) In HDFSEventSink.process the channel returns null or batch size is exceeded
      4) In HDFSEventSink.process bucket writer flush is called throwing the exception logged above.

      1. FLUME-1610-3.patch
        5 kB
        Mike Percy
      2. FLUME-1610-0.patch
        3 kB
        Brock Noland

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in flume-trunk #309 (See https://builds.apache.org/job/flume-trunk/309/)
          FLUME-1610: HDDSEventSink and bucket writer have a race condition (Revision d1137017866f9c666f47b55452531e01a27fbdd8)

          Result = SUCCESS
          brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=d1137017866f9c666f47b55452531e01a27fbdd8
          Files :

          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
          Show
          Hudson added a comment - Integrated in flume-trunk #309 (See https://builds.apache.org/job/flume-trunk/309/ ) FLUME-1610 : HDDSEventSink and bucket writer have a race condition (Revision d1137017866f9c666f47b55452531e01a27fbdd8) Result = SUCCESS brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=d1137017866f9c666f47b55452531e01a27fbdd8 Files : flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
          Hide
          Brock Noland added a comment -
          Show
          Brock Noland added a comment - Committed here https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=d1137017866f9c666f47b55452531e01a27fbdd8 Thank you for your contribution Mike!
          Hide
          Mike Percy added a comment -

          Patch to fix race condition in HDFS sink

          Show
          Mike Percy added a comment - Patch to fix race condition in HDFS sink
          Hide
          Brock Noland added a comment -

          Attached is a patch which shows the scenario I believe is occurring.

          Show
          Brock Noland added a comment - Attached is a patch which shows the scenario I believe is occurring.

            People

            • Assignee:
              Mike Percy
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development