Flume
  1. Flume
  2. FLUME-1096

Add support to pass headers through AvroCLIClient

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: None
    • Labels:

      Description

      Configured HDFSEvent Sink to generate file path's using regex based date format.

      Config for hdfs-sink -> collector1.sinks.hdfs-sink1.hdfs.filePrefix = flume/%D
      Hit NumberFormatException in BucketPath.replaceShorthand() as it reads timestamp from headers

      Date date = new Date(Long.valueOf(headers.get("timestamp")));

      java.lang.NumberFormatException: null
      at java.lang.Long.parseLong(Long.java:375)
      at java.lang.Long.valueOf(Long.java:525)
      at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185)
      at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219)
      at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:341)
      at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
      at java.lang.Thread.run(Thread.java:680)

      Sending the events using - ./flume-ng avro-client --conf ../conf/ -H localhost -p 41415 -F ~/.bash_profile

      Issue - AvroCLiClient isn't adding timestamp in headers

      1. FLUME-1096-2.patch
        3 kB
        Inder SIngh
      2. FLUME-1096-3.patch
        4 kB
        Inder SIngh
      3. FLUME-1096-4.patch
        4 kB
        Inder SIngh
      4. FLUME-1096-7.patch
        4 kB
        Inder SIngh

        Activity

        Hide
        Hudson added a comment -

        Integrated in flume-trunk #181 (See https://builds.apache.org/job/flume-trunk/181/)
        FLUME-1096. Support for headers through AvroCLIClient

        (Inder Singh via Arvind Prabhakar) (Revision 1329673)

        Result = SUCCESS
        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1329673
        Files :

        • /incubator/flume/trunk/bin/flume-ng
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
        Show
        Hudson added a comment - Integrated in flume-trunk #181 (See https://builds.apache.org/job/flume-trunk/181/ ) FLUME-1096 . Support for headers through AvroCLIClient (Inder Singh via Arvind Prabhakar) (Revision 1329673) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1329673 Files : /incubator/flume/trunk/bin/flume-ng /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Inder!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Inder!
        Hide
        Inder SIngh added a comment -

        Patch incorporating all review comments from - https://reviews.apache.org/r/4622/

        Show
        Inder SIngh added a comment - Patch incorporating all review comments from - https://reviews.apache.org/r/4622/
        Hide
        Inder SIngh added a comment -

        Reading headers to be sent through avroclient through a property file

        Show
        Inder SIngh added a comment - Reading headers to be sent through avroclient through a property file
        Hide
        Inder SIngh added a comment -

        Updated the client to read headers from a property file.
        Sample header.properties
        ------------------------------------
        streamName = sample-stream
        timestamp=2189128912

        Client command line looks like ->

        ./flume-ng avro-client --conf ../conf/ -H localhost -p 41414 -F ~/.bash_profile -R headers.properties

        Show
        Inder SIngh added a comment - Updated the client to read headers from a property file. Sample header.properties ------------------------------------ streamName = sample-stream timestamp=2189128912 Client command line looks like -> ./flume-ng avro-client --conf ../conf/ -H localhost -p 41414 -F ~/.bash_profile -R headers.properties
        Hide
        Inder SIngh added a comment -

        Latest Patch available for review - https://issues.apache.org/jira/secure/attachment/12521935/FLUME-1096-3.patch.
        Review Board Link - https://reviews.apache.org/r/4622/
        Testing Done - Tested AvroCLIClient by passing timestamp header with the following command.

        "flume-ng avro-client --conf ../conf/ -H 0.0.0.0 -p 41415 -F ~/.bash_profile -R timestamp=389908323"

        Show
        Inder SIngh added a comment - Latest Patch available for review - https://issues.apache.org/jira/secure/attachment/12521935/FLUME-1096-3.patch . Review Board Link - https://reviews.apache.org/r/4622/ Testing Done - Tested AvroCLIClient by passing timestamp header with the following command. "flume-ng avro-client --conf ../conf/ -H 0.0.0.0 -p 41415 -F ~/.bash_profile -R timestamp=389908323"
        Hide
        Inder SIngh added a comment -

        Patch to support passing headers in AvroCLIClient with changes in bin/flume-ng

        Show
        Inder SIngh added a comment - Patch to support passing headers in AvroCLIClient with changes in bin/flume-ng
        Hide
        Inder SIngh added a comment -

        Patch to support passing headers in AvroCLIClient

        Show
        Inder SIngh added a comment - Patch to support passing headers in AvroCLIClient
        Hide
        Inder SIngh added a comment -

        Please review the patch at - https://reviews.apache.org/r/4622/

        Show
        Inder SIngh added a comment - Please review the patch at - https://reviews.apache.org/r/4622/

          People

          • Assignee:
            Inder SIngh
            Reporter:
            Inder SIngh
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development