Flume
  1. Flume
  2. FLUME-1509

Flume HDFS sink should allow for the use of different timezones when resolving sink paths

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.3.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, if you have an HDFS sink configured with a custom path, which specifies the date, the %Y-%m-%d-%H variables will always be resolved to the agent's default timezone. It would be nice to be able to specify the timezone that should be used for this path resolution.

      1. FLUME-1509.patch
        9 kB
        Jonathan Natkins
      2. FLUME-1509.patch
        8 kB
        Jonathan Natkins

        Activity

        Hide
        Jonathan Natkins added a comment -

        Submitting a patch which adds a timeZone parameter to the HDFS sink. It uses standard timezone strings to specify what timezone the timestamp header should be resolved in.

        Show
        Jonathan Natkins added a comment - Submitting a patch which adds a timeZone parameter to the HDFS sink. It uses standard timezone strings to specify what timezone the timestamp header should be resolved in.
        Hide
        Hari Shreedharan added a comment -

        Thanks Jonathan. Could you please submit the patch to review board : https://reviews.apache.org/

        Show
        Hari Shreedharan added a comment - Thanks Jonathan. Could you please submit the patch to review board : https://reviews.apache.org/
        Hide
        Jonathan Natkins added a comment -
        Show
        Jonathan Natkins added a comment - Sorry about that! https://reviews.apache.org/r/6886/
        Hide
        Jonathan Natkins added a comment -

        Attaching a final version of the patch, post-review

        Show
        Jonathan Natkins added a comment - Attaching a final version of the patch, post-review
        Hide
        Hari Shreedharan added a comment -

        Patch committed, rev: 80699c46275fd8ad4be30de3cd1e3cd746b3ac26! Thanks Jonathan!

        Show
        Hari Shreedharan added a comment - Patch committed, rev: 80699c46275fd8ad4be30de3cd1e3cd746b3ac26! Thanks Jonathan!
        Hide
        Hudson added a comment -

        Integrated in flume-1.3.0 #7 (See https://builds.apache.org/job/flume-1.3.0/7/)
        FLUME-1509. HDFS sink should allow for the use of different timezones when resolving sink paths. (Revision 939bf1074ebcae345608288306d274687c97fc8d)

        Result = FAILURE
        hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=939bf1074ebcae345608288306d274687c97fc8d
        Files :

        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java
        • flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java
        • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
        Show
        Hudson added a comment - Integrated in flume-1.3.0 #7 (See https://builds.apache.org/job/flume-1.3.0/7/ ) FLUME-1509 . HDFS sink should allow for the use of different timezones when resolving sink paths. (Revision 939bf1074ebcae345608288306d274687c97fc8d) Result = FAILURE hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=939bf1074ebcae345608288306d274687c97fc8d Files : flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #293 (See https://builds.apache.org/job/flume-trunk/293/)
        FLUME-1509. HDFS sink should allow for the use of different timezones when resolving sink paths. (Revision 80699c46275fd8ad4be30de3cd1e3cd746b3ac26)

        Result = UNSTABLE
        hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=80699c46275fd8ad4be30de3cd1e3cd746b3ac26
        Files :

        • flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java
        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
        • flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java
        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        Show
        Hudson added a comment - Integrated in flume-trunk #293 (See https://builds.apache.org/job/flume-trunk/293/ ) FLUME-1509 . HDFS sink should allow for the use of different timezones when resolving sink paths. (Revision 80699c46275fd8ad4be30de3cd1e3cd746b3ac26) Result = UNSTABLE hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=80699c46275fd8ad4be30de3cd1e3cd746b3ac26 Files : flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java

          People

          • Assignee:
            Jonathan Natkins
            Reporter:
            Jonathan Natkins
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development