Flume
  1. Flume
  2. FLUME-2014

Race condition when using local timestamp with BucketPath

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.3.1
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      There is a race condition in the BucketPath code logic which was exposed by adding local (non-event) timestamp capability to BucketPath and HDFS sink.

      Patch forthcoming

      1. FLUME-2014-2.patch
        8 kB
        Mike Percy
      2. FLUME-2014-1.patch
        8 kB
        Mike Percy

        Issue Links

          Activity

          Hide
          Hari Shreedharan added a comment -

          Excellent catch, Mike! Yes, looks like escapeString ends up calling currentTimeMillis() multiple times - so across hour boundaries, it will cause issues. I will review the patch and commit tmrw.

          Show
          Hari Shreedharan added a comment - Excellent catch, Mike! Yes, looks like escapeString ends up calling currentTimeMillis() multiple times - so across hour boundaries, it will cause issues. I will review the patch and commit tmrw.
          Hide
          Mike Percy added a comment -

          Updated patch

          Show
          Mike Percy added a comment - Updated patch
          Hide
          Hari Shreedharan added a comment -

          +1. Will run tests and commit.

          Show
          Hari Shreedharan added a comment - +1. Will run tests and commit.
          Hide
          Hari Shreedharan added a comment -

          Patch committed, rev: 0a85548. Thanks Mike!

          Show
          Hari Shreedharan added a comment - Patch committed, rev: 0a85548. Thanks Mike!
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #397 (See https://builds.apache.org/job/flume-trunk/397/)
          FLUME-2014. Race condition when using local timestamp with BucketPath (Revision 0a855488a152cf695e2e1e2f89b67ed2e3095422)

          Result = FAILURE
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0a855488a152cf695e2e1e2f89b67ed2e3095422
          Files :

          • flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java
          Show
          Hudson added a comment - Integrated in flume-trunk #397 (See https://builds.apache.org/job/flume-trunk/397/ ) FLUME-2014 . Race condition when using local timestamp with BucketPath (Revision 0a855488a152cf695e2e1e2f89b67ed2e3095422) Result = FAILURE hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0a855488a152cf695e2e1e2f89b67ed2e3095422 Files : flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java

            People

            • Assignee:
              Mike Percy
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development