Flume
  1. Flume
  2. FLUME-1096

Add support to pass headers through AvroCLIClient

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: None
    • Labels:

      Description

      Configured HDFSEvent Sink to generate file path's using regex based date format.

      Config for hdfs-sink -> collector1.sinks.hdfs-sink1.hdfs.filePrefix = flume/%D
      Hit NumberFormatException in BucketPath.replaceShorthand() as it reads timestamp from headers

      Date date = new Date(Long.valueOf(headers.get("timestamp")));

      java.lang.NumberFormatException: null
      at java.lang.Long.parseLong(Long.java:375)
      at java.lang.Long.valueOf(Long.java:525)
      at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185)
      at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219)
      at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:341)
      at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
      at java.lang.Thread.run(Thread.java:680)

      Sending the events using - ./flume-ng avro-client --conf ../conf/ -H localhost -p 41415 -F ~/.bash_profile

      Issue - AvroCLiClient isn't adding timestamp in headers

      1. FLUME-1096-7.patch
        4 kB
        Inder SIngh
      2. FLUME-1096-4.patch
        4 kB
        Inder SIngh
      3. FLUME-1096-3.patch
        4 kB
        Inder SIngh
      4. FLUME-1096-2.patch
        3 kB
        Inder SIngh

        Activity

        Inder SIngh created issue -
        Inder SIngh made changes -
        Field Original Value New Value
        Assignee Inder SIngh [ inder ]
        Description Configured HDFSEvent Sink to generate file path's using regex based date format.

        Config for hdfs-sink -> collector1.sinks.hdfs-sink1.hdfs.filePrefix = flume/%D
        Hit NumberFormatException in BucketPath.replaceShorthand() as it reads timestamp from headers
            
        Date date = new Date(Long.valueOf(headers.get("timestamp")));

        java.lang.NumberFormatException: null
                at java.lang.Long.parseLong(Long.java:375)
                at java.lang.Long.valueOf(Long.java:525)
                at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185)
                at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219)
                at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:341)
                at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
                at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
                at java.lang.Thread.run(Thread.java:680)

        Sending the events using - ./flume-ng avro-client --conf ../conf/ -H localhost -p 41415 -F ~/.bash_profile

        Issue - AvroCLiClient isn't adding timestamp in headers

        Inder SIngh made changes -
        Summary Issues in using AvroClient with HDFSEvent for regex based destination paths Issues in using AvroClient with HDFSEventSink for regex based destination paths
        Hide
        Inder SIngh added a comment -

        Please review the patch at - https://reviews.apache.org/r/4622/

        Show
        Inder SIngh added a comment - Please review the patch at - https://reviews.apache.org/r/4622/
        Hide
        Inder SIngh added a comment -

        Patch to support passing headers in AvroCLIClient

        Show
        Inder SIngh added a comment - Patch to support passing headers in AvroCLIClient
        Inder SIngh made changes -
        Attachment FLUME-1096-2.patch [ 12521934 ]
        Hide
        Inder SIngh added a comment -

        Patch to support passing headers in AvroCLIClient with changes in bin/flume-ng

        Show
        Inder SIngh added a comment - Patch to support passing headers in AvroCLIClient with changes in bin/flume-ng
        Inder SIngh made changes -
        Attachment FLUME-1096-3.patch [ 12521935 ]
        Hide
        Inder SIngh added a comment -

        Latest Patch available for review - https://issues.apache.org/jira/secure/attachment/12521935/FLUME-1096-3.patch.
        Review Board Link - https://reviews.apache.org/r/4622/
        Testing Done - Tested AvroCLIClient by passing timestamp header with the following command.

        "flume-ng avro-client --conf ../conf/ -H 0.0.0.0 -p 41415 -F ~/.bash_profile -R timestamp=389908323"

        Show
        Inder SIngh added a comment - Latest Patch available for review - https://issues.apache.org/jira/secure/attachment/12521935/FLUME-1096-3.patch . Review Board Link - https://reviews.apache.org/r/4622/ Testing Done - Tested AvroCLIClient by passing timestamp header with the following command. "flume-ng avro-client --conf ../conf/ -H 0.0.0.0 -p 41415 -F ~/.bash_profile -R timestamp=389908323"
        Inder SIngh made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note Adding support to pass headers in AvroCLIClient
        Affects Version/s v1.2.0 [ 12320243 ]
        Labels avroclient
        Fix Version/s v1.2.0 [ 12320243 ]
        Inder SIngh made changes -
        Summary Issues in using AvroClient with HDFSEventSink for regex based destination paths Add support to pass headers through AvroCLIClient
        Hide
        Inder SIngh added a comment -

        Updated the client to read headers from a property file.
        Sample header.properties
        ------------------------------------
        streamName = sample-stream
        timestamp=2189128912

        Client command line looks like ->

        ./flume-ng avro-client --conf ../conf/ -H localhost -p 41414 -F ~/.bash_profile -R headers.properties

        Show
        Inder SIngh added a comment - Updated the client to read headers from a property file. Sample header.properties ------------------------------------ streamName = sample-stream timestamp=2189128912 Client command line looks like -> ./flume-ng avro-client --conf ../conf/ -H localhost -p 41414 -F ~/.bash_profile -R headers.properties
        Hide
        Inder SIngh added a comment -

        Reading headers to be sent through avroclient through a property file

        Show
        Inder SIngh added a comment - Reading headers to be sent through avroclient through a property file
        Inder SIngh made changes -
        Attachment FLUME-1096-4.patch [ 12523468 ]
        Hide
        Inder SIngh added a comment -

        Patch incorporating all review comments from - https://reviews.apache.org/r/4622/

        Show
        Inder SIngh added a comment - Patch incorporating all review comments from - https://reviews.apache.org/r/4622/
        Inder SIngh made changes -
        Attachment FLUME-1096-7.patch [ 12523923 ]
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Inder!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Inder!
        Arvind Prabhakar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Release Note Adding support to pass headers in AvroCLIClient
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #181 (See https://builds.apache.org/job/flume-trunk/181/)
        FLUME-1096. Support for headers through AvroCLIClient

        (Inder Singh via Arvind Prabhakar) (Revision 1329673)

        Result = SUCCESS
        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1329673
        Files :

        • /incubator/flume/trunk/bin/flume-ng
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
        Show
        Hudson added a comment - Integrated in flume-trunk #181 (See https://builds.apache.org/job/flume-trunk/181/ ) FLUME-1096 . Support for headers through AvroCLIClient (Inder Singh via Arvind Prabhakar) (Revision 1329673) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1329673 Files : /incubator/flume/trunk/bin/flume-ng /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        5d 22h 2m 1 Inder SIngh 09/Apr/12 09:55
        Patch Available Patch Available Resolved Resolved
        15d 2h 35m 1 Arvind Prabhakar 24/Apr/12 12:30

          People

          • Assignee:
            Inder SIngh
            Reporter:
            Inder SIngh
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development