Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2718

HTTP Source to support generic Stream Handler

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Flags:
      Patch

      Description

      Currently the HTTP Source supports JSONHandler as the default implementation. A more generic approach will be having a BLOBHandler which accepts any request input stream (that loads the stream as Event payload). Furthermore, this Handler lets you define mandatory request parameters and maps those parameters into Event Headers.

      This way HTTPSource can be used as a generic Data Ingress endpoint for any sink, where one can specify attributes run like basepath, filename & timestamp as request parameters and access those values via HEADER values in sink properties.

      All this can be done without developing any custom Handler code.

      For e.g.

      With the below agent configuration, you can send any type of data (JSON/CSV/TSV) and store it in any sink, HDFS in this case.

      sample command
      curl -v -X POST "http://testHost:8080/?basepath=/data/&filename=test.json&timestamp=1434101498275" --data @test.json
      
      HDFS data path
      /data/2015/06/12/test.json.1434101498275.lzo
      
      agent.conf
      #Agent configuration
      #HTTP Source configuration
      agent.sources = httpSrc
      agent.channels = memChannel
      agent.sources.httpSrc.type = http
      agent.sources.httpSrc.channels = memChannel
      agent.sources.httpSrc.bind = testHost
      agent.sources.httpSrc.port = 8080
      agent.sources.httpSrc.handler = org.apache.flume.source.http.BLOBHandler
      agent.sources.httpSrc.handler.mandatoryParameters = basepath, filename
      
      #Memory channel with default configuration
      agent.channels.memChannel.type = memory
      agent.channels.memChannel.capacity = 100000
      agent.channels.memChannel.transactionCapacity = 1000
      
      #HDFS Sink configuration
      agent.sinks.hdfsSink.type = hdfs
      agent.sinks.hdfsSink.hdfs.path = %{basepath}/%Y/%m/%d
      agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
      agent.sinks.hdfsSink.hdfs.filePrefix = %{filename}
      agent.sinks.hdfsSink.hdfs.fileType = CompressedStream
      agent.sinks.hdfsSink.hdfs.codeC = lzop
      agent.sinks.hdfsSink.channel = memChannel
      
      #Finally, activate.
      agent.channels = memChannel
      agent.sources = httpSrc
      agent.sinks = hdfsSink
      

        Activity

        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Thanks for the patch Hari! Would you mind uploading it to review board for deeper review?

        Show
        jarcec Jarek Jarcec Cecho added a comment - Thanks for the patch Hari ! Would you mind uploading it to review board for deeper review?
        Hide
        hariprasad kuppuswamy Hari added a comment - - edited

        Sure. https://reviews.apache.org/r/42302/ created for the same.

        Show
        hariprasad kuppuswamy Hari added a comment - - edited Sure. https://reviews.apache.org/r/42302/ created for the same.
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        I left few comments on the review board.

        Show
        jarcec Jarek Jarcec Cecho added a comment - I left few comments on the review board.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit e6416a0707a8be539a163bec9d7b12f7afbc36cf in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e6416a0 ]

        FLUME-2718: HTTP Source to support generic Stream Handler

        (Hari via Jarek Jarcec Cecho)

        Show
        jira-bot ASF subversion and git services added a comment - Commit e6416a0707a8be539a163bec9d7b12f7afbc36cf in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e6416a0 ] FLUME-2718 : HTTP Source to support generic Stream Handler (Hari via Jarek Jarcec Cecho)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit d6b4053e76024d1f36b43f5c458783ba0849616b in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d6b4053 ]

        FLUME-2718: HTTP Source to support generic Stream Handler

        (Hari via Jarek Jarcec Cecho)

        Show
        jira-bot ASF subversion and git services added a comment - Commit d6b4053e76024d1f36b43f5c458783ba0849616b in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d6b4053 ] FLUME-2718 : HTTP Source to support generic Stream Handler (Hari via Jarek Jarcec Cecho)
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        The patch is in, thank you for your contribution Hari!

        Show
        jarcec Jarek Jarcec Cecho added a comment - The patch is in, thank you for your contribution Hari!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-1 #146 (See https://builds.apache.org/job/Flume-trunk-hbase-1/146/)
        FLUME-2718: HTTP Source to support generic Stream Handler (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e6416a0707a8be539a163bec9d7b12f7afbc36cf)

        • flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java
        • flume-ng-core/src/test/java/org/apache/flume/source/http/TestBLOBHandler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-1 #146 (See https://builds.apache.org/job/Flume-trunk-hbase-1/146/ ) FLUME-2718 : HTTP Source to support generic Stream Handler (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e6416a0707a8be539a163bec9d7b12f7afbc36cf ) flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java flume-ng-core/src/test/java/org/apache/flume/source/http/TestBLOBHandler.java

          People

          • Assignee:
            hariprasad kuppuswamy Hari
            Reporter:
            hariprasad kuppuswamy Hari
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development