Flume
  1. Flume
  2. FLUME-1096

Add support to pass headers through AvroCLIClient

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: None
    • Labels:

      Description

      Configured HDFSEvent Sink to generate file path's using regex based date format.

      Config for hdfs-sink -> collector1.sinks.hdfs-sink1.hdfs.filePrefix = flume/%D
      Hit NumberFormatException in BucketPath.replaceShorthand() as it reads timestamp from headers

      Date date = new Date(Long.valueOf(headers.get("timestamp")));

      java.lang.NumberFormatException: null
      at java.lang.Long.parseLong(Long.java:375)
      at java.lang.Long.valueOf(Long.java:525)
      at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185)
      at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219)
      at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:341)
      at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
      at java.lang.Thread.run(Thread.java:680)

      Sending the events using - ./flume-ng avro-client --conf ../conf/ -H localhost -p 41415 -F ~/.bash_profile

      Issue - AvroCLiClient isn't adding timestamp in headers

      1. FLUME-1096-7.patch
        4 kB
        Inder SIngh
      2. FLUME-1096-4.patch
        4 kB
        Inder SIngh
      3. FLUME-1096-3.patch
        4 kB
        Inder SIngh
      4. FLUME-1096-2.patch
        3 kB
        Inder SIngh

        Activity

        Inder SIngh created issue -
        Inder SIngh made changes -
        Field Original Value New Value
        Assignee Inder SIngh [ inder ]
        Description Configured HDFSEvent Sink to generate file path's using regex based date format.

        Config for hdfs-sink -> collector1.sinks.hdfs-sink1.hdfs.filePrefix = flume/%D
        Hit NumberFormatException in BucketPath.replaceShorthand() as it reads timestamp from headers
            
        Date date = new Date(Long.valueOf(headers.get("timestamp")));

        java.lang.NumberFormatException: null
                at java.lang.Long.parseLong(Long.java:375)
                at java.lang.Long.valueOf(Long.java:525)
                at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185)
                at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219)
                at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:341)
                at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
                at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
                at java.lang.Thread.run(Thread.java:680)

        Sending the events using - ./flume-ng avro-client --conf ../conf/ -H localhost -p 41415 -F ~/.bash_profile

        Issue - AvroCLiClient isn't adding timestamp in headers

        Inder SIngh made changes -
        Summary Issues in using AvroClient with HDFSEvent for regex based destination paths Issues in using AvroClient with HDFSEventSink for regex based destination paths
        Inder SIngh made changes -
        Attachment FLUME-1096-2.patch [ 12521934 ]
        Inder SIngh made changes -
        Attachment FLUME-1096-3.patch [ 12521935 ]
        Inder SIngh made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note Adding support to pass headers in AvroCLIClient
        Affects Version/s v1.2.0 [ 12320243 ]
        Labels avroclient
        Fix Version/s v1.2.0 [ 12320243 ]
        Inder SIngh made changes -
        Summary Issues in using AvroClient with HDFSEventSink for regex based destination paths Add support to pass headers through AvroCLIClient
        Inder SIngh made changes -
        Attachment FLUME-1096-4.patch [ 12523468 ]
        Inder SIngh made changes -
        Attachment FLUME-1096-7.patch [ 12523923 ]
        Arvind Prabhakar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Release Note Adding support to pass headers in AvroCLIClient
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Inder SIngh
            Reporter:
            Inder SIngh
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development