Flume
  1. Flume
  2. FLUME-1922

HDFS Sink should optionally insert the timestamp at the sink

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      The HDFS Sink currently looks for the timestamp in the event headers. If some events are delayed then the events will end up in the old directory itself. We should allow the HDFS Sink to use the local timestamp to decide where the event goes.

      1. FLUME-1922-1.patch
        13 kB
        Hari Shreedharan
      2. FLUME-1922.patch
        13 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in flume-trunk #362 (See https://builds.apache.org/job/flume-trunk/362/)
          FLUME-1922. HDFS Sink should optionally insert the timestamp at the sink. (Revision 6293146224a261d340b39824ce7dadcda9b1efd7)

          Result = SUCCESS
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6293146224a261d340b39824ce7dadcda9b1efd7
          Files :

          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
          • flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          Hudson added a comment - Integrated in flume-trunk #362 (See https://builds.apache.org/job/flume-trunk/362/ ) FLUME-1922 . HDFS Sink should optionally insert the timestamp at the sink. (Revision 6293146224a261d340b39824ce7dadcda9b1efd7) Result = SUCCESS mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6293146224a261d340b39824ce7dadcda9b1efd7 Files : flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java flume-ng-doc/sphinx/FlumeUserGuide.rst
          Hide
          Mike Percy added a comment -

          Pushed to trunk & flume-1.4 branches. Thanks for the patch, Hari!

          Show
          Mike Percy added a comment - Pushed to trunk & flume-1.4 branches. Thanks for the patch, Hari!
          Hide
          Mike Percy added a comment -

          +1

          Show
          Mike Percy added a comment - +1
          Hide
          Hari Shreedharan added a comment -

          Thanks for the review, Mike.

          The patch already has user doc update. I will add @VisibleForTesting to the setBucketClock method.

          Show
          Hari Shreedharan added a comment - Thanks for the review, Mike. The patch already has user doc update. I will add @VisibleForTesting to the setBucketClock method.
          Hide
          Mike Percy added a comment -

          Patch looks good to me. I agree we need this features. Couple of comments:
          1. Needs user doc
          2. Can you please add a comment to HDFSEventSink.setBucketClock() indicating that it calls a static method and should only be used by tests? Or just add a @VisibleForTesting annotation to that method too.

          Thanks!

          Show
          Mike Percy added a comment - Patch looks good to me. I agree we need this features. Couple of comments: 1. Needs user doc 2. Can you please add a comment to HDFSEventSink.setBucketClock() indicating that it calls a static method and should only be used by tests? Or just add a @VisibleForTesting annotation to that method too. Thanks!

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development