Flume
  1. Flume
  2. FLUME-2718

HTTP Source to support generic Stream Handler

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Flags:
      Patch

      Description

      Currently the HTTP Source supports JSONHandler as the default implementation. A more generic approach will be having a BLOBHandler which accepts any request input stream (that loads the stream as Event payload). Furthermore, this Handler lets you define mandatory request parameters and maps those parameters into Event Headers.

      This way HTTPSource can be used as a generic Data Ingress endpoint for any sink, where one can specify attributes run like basepath, filename & timestamp as request parameters and access those values via HEADER values in sink properties.

      All this can be done without developing any custom Handler code.

      For e.g.

      With the below agent configuration, you can send any type of data (JSON/CSV/TSV) and store it in any sink, HDFS in this case.

      sample command
      curl -v -X POST "http://testHost:8080/?basepath=/data/&filename=test.json&timestamp=1434101498275" --data @test.json
      
      HDFS data path
      /data/2015/06/12/test.json.1434101498275.lzo
      
      agent.conf
      #Agent configuration
      #HTTP Source configuration
      agent.sources = httpSrc
      agent.channels = memChannel
      agent.sources.httpSrc.type = http
      agent.sources.httpSrc.channels = memChannel
      agent.sources.httpSrc.bind = testHost
      agent.sources.httpSrc.port = 8080
      agent.sources.httpSrc.handler = org.apache.flume.source.http.BLOBHandler
      agent.sources.httpSrc.handler.mandatoryParameters = basepath, filename
      
      #Memory channel with default configuration
      agent.channels.memChannel.type = memory
      agent.channels.memChannel.capacity = 100000
      agent.channels.memChannel.transactionCapacity = 1000
      
      #HDFS Sink configuration
      agent.sinks.hdfsSink.type = hdfs
      agent.sinks.hdfsSink.hdfs.path = %{basepath}/%Y/%m/%d
      agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
      agent.sinks.hdfsSink.hdfs.filePrefix = %{filename}
      agent.sinks.hdfsSink.hdfs.fileType = CompressedStream
      agent.sinks.hdfsSink.hdfs.codeC = lzop
      agent.sinks.hdfsSink.channel = memChannel
      
      #Finally, activate.
      agent.channels = memChannel
      agent.sources = httpSrc
      agent.sinks = hdfsSink
      

        Activity

        Hide
        Jarek Jarcec Cecho added a comment -

        Thanks for the patch Hari! Would you mind uploading it to review board for deeper review?

        Show
        Jarek Jarcec Cecho added a comment - Thanks for the patch Hari ! Would you mind uploading it to review board for deeper review?
        Hide
        Hari added a comment - - edited

        Sure. https://reviews.apache.org/r/42302/ created for the same.

        Show
        Hari added a comment - - edited Sure. https://reviews.apache.org/r/42302/ created for the same.
        Hide
        Jarek Jarcec Cecho added a comment -

        I left few comments on the review board.

        Show
        Jarek Jarcec Cecho added a comment - I left few comments on the review board.
        Hide
        ASF subversion and git services added a comment -

        Commit e6416a0707a8be539a163bec9d7b12f7afbc36cf in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e6416a0 ]

        FLUME-2718: HTTP Source to support generic Stream Handler

        (Hari via Jarek Jarcec Cecho)

        Show
        ASF subversion and git services added a comment - Commit e6416a0707a8be539a163bec9d7b12f7afbc36cf in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e6416a0 ] FLUME-2718 : HTTP Source to support generic Stream Handler (Hari via Jarek Jarcec Cecho)
        Hide
        ASF subversion and git services added a comment -

        Commit d6b4053e76024d1f36b43f5c458783ba0849616b in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d6b4053 ]

        FLUME-2718: HTTP Source to support generic Stream Handler

        (Hari via Jarek Jarcec Cecho)

        Show
        ASF subversion and git services added a comment - Commit d6b4053e76024d1f36b43f5c458783ba0849616b in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d6b4053 ] FLUME-2718 : HTTP Source to support generic Stream Handler (Hari via Jarek Jarcec Cecho)
        Hide
        Jarek Jarcec Cecho added a comment -

        The patch is in, thank you for your contribution Hari!

        Show
        Jarek Jarcec Cecho added a comment - The patch is in, thank you for your contribution Hari!
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-1 #146 (See https://builds.apache.org/job/Flume-trunk-hbase-1/146/)
        FLUME-2718: HTTP Source to support generic Stream Handler (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e6416a0707a8be539a163bec9d7b12f7afbc36cf)

        • flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java
        • flume-ng-core/src/test/java/org/apache/flume/source/http/TestBLOBHandler.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-1 #146 (See https://builds.apache.org/job/Flume-trunk-hbase-1/146/ ) FLUME-2718 : HTTP Source to support generic Stream Handler (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e6416a0707a8be539a163bec9d7b12f7afbc36cf ) flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java flume-ng-core/src/test/java/org/apache/flume/source/http/TestBLOBHandler.java

          People

          • Assignee:
            Hari
            Reporter:
            Hari
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development