Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1096

Add support to pass headers through AvroCLIClient

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:

      Description

      Configured HDFSEvent Sink to generate file path's using regex based date format.

      Config for hdfs-sink -> collector1.sinks.hdfs-sink1.hdfs.filePrefix = flume/%D
      Hit NumberFormatException in BucketPath.replaceShorthand() as it reads timestamp from headers

      Date date = new Date(Long.valueOf(headers.get("timestamp")));

      java.lang.NumberFormatException: null
      at java.lang.Long.parseLong(Long.java:375)
      at java.lang.Long.valueOf(Long.java:525)
      at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185)
      at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219)
      at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:341)
      at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
      at java.lang.Thread.run(Thread.java:680)

      Sending the events using - ./flume-ng avro-client --conf ../conf/ -H localhost -p 41415 -F ~/.bash_profile

      Issue - AvroCLiClient isn't adding timestamp in headers

        Attachments

        1. FLUME-1096-7.patch
          4 kB
          Inder SIngh
        2. FLUME-1096-4.patch
          4 kB
          Inder SIngh
        3. FLUME-1096-3.patch
          4 kB
          Inder SIngh
        4. FLUME-1096-2.patch
          3 kB
          Inder SIngh

          Activity

            People

            • Assignee:
              inder Inder SIngh
              Reporter:
              inder Inder SIngh
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: