Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2725

HDFS Sink does not use configured timezone for rounding

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      When a BucketPath used by an HDFS sink is configured to run with some roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by BucketPath does not actually round the date correctly.

      That function calls TimestampRoundDownUtil which creates a Calendar instance using the local timezone to truncate a unix timestamp rather than the TimeZone that the sink was configured to convert dates to paths with (and that timezone is already available in the BucketPath class but it just isn't passed to TimestampRoundDownUtil).

      The net effect of this is that if a flume jvm is running on a system with an EST clock while trying to write, say, 6 hour directories in UTC time, the directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, 18 like you would expect.

      I found a workaround for this by passing "-Duser.timezone=<hdfs_sink_timezone>" as a system property, but I wanted to create a ticket for this since it seems like it would be very minimal effort to carry that configured timezone down into the rounding utility as well.

        Attachments

        1. FLUME-2725-3.patch
          22 kB
          Denes Arvay
        2. FLUME-2725-2.patch
          16 kB
          Denes Arvay
        3. FLUME-2725.patch
          16 kB
          Denes Arvay

          Issue Links

            Activity

              People

              • Assignee:
                denes Denes Arvay
                Reporter:
                eczech Eric Czech
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: