Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.1.0
    • Fix Version/s: v1.3.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      From some clients it would be simpler to post an event to a HTTP URL then to make an Avro RPC call. This source would accept inbound HTTP POST requests and ideally would place all the HTTP headers into the headers of the event so they could be used for routing and partitioning.

      1. FLUME-1199-4.patch
        59 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          Mubarak Seyed added a comment - - edited

          For HTTP monitoring, we are planning to embed Jetty 6.26 (with NIO support) for WebUI and monitoring per agent. We can have multiple app context (/agent-status, /http-source1, /http-source2, etc). We can build a REST layer inside agent (as a wrapper) to convert HTTP post request (to Event) and place it in a channel. If there is enough system resource available (Memory, CPU), we can embed REST and Agent in same JVM and run as in-process.

          HBase REST layer runs out of process with RegionServer, meaning it calls HBase client API for each request invocation.

          Show
          Mubarak Seyed added a comment - - edited For HTTP monitoring, we are planning to embed Jetty 6.26 (with NIO support) for WebUI and monitoring per agent. We can have multiple app context (/agent-status, /http-source1, /http-source2, etc). We can build a REST layer inside agent (as a wrapper) to convert HTTP post request (to Event) and place it in a channel. If there is enough system resource available (Memory, CPU), we can embed REST and Agent in same JVM and run as in-process. HBase REST layer runs out of process with RegionServer, meaning it calls HBase client API for each request invocation.
          Hide
          Otis Gospodnetic added a comment -

          Out of curiosity, why such an old version of Jetty? It's up to 8.* now and 7.* versions have been around for well over a year.

          Show
          Otis Gospodnetic added a comment - Out of curiosity, why such an old version of Jetty? It's up to 8.* now and 7.* versions have been around for well over a year.
          Hide
          Mubarak Seyed added a comment -

          I think CDH3 bundled with Jetty 6.1.26.

          Show
          Mubarak Seyed added a comment - I think CDH3 bundled with Jetty 6.1.26.
          Hide
          Mike Percy added a comment -

          This is interesting. I think this could be a very useful piece for system integration purposes.

          Side note: from a design perspective, this is a separate component from the HTTP monitoring support, and as such we should keep them separate.

          Show
          Mike Percy added a comment - This is interesting. I think this could be a very useful piece for system integration purposes. Side note: from a design perspective, this is a separate component from the HTTP monitoring support, and as such we should keep them separate.
          Hide
          Hari Shreedharan added a comment -

          I'm planning to take this up. If anyone is already working on this, let me know.

          Show
          Hari Shreedharan added a comment - I'm planning to take this up. If anyone is already working on this, let me know.
          Hide
          Brock Noland added a comment -

          Hari, I noticed the RB item is not linked to this JIRA.

          Show
          Brock Noland added a comment - Hari, I noticed the RB item is not linked to this JIRA.
          Hide
          Hari Shreedharan added a comment -

          Sorry, missed that. Done now.

          Show
          Hari Shreedharan added a comment - Sorry, missed that. Done now.
          Hide
          Roshan Naik added a comment -

          looks like i got accidentally assigned to this somehow.

          Show
          Roshan Naik added a comment - looks like i got accidentally assigned to this somehow.
          Hide
          Brock Noland added a comment -

          Yeah if you click on anything in this new JIRA version it puts it in edit mode. I have deleted some stuff on accident...

          Show
          Brock Noland added a comment - Yeah if you click on anything in this new JIRA version it puts it in edit mode. I have deleted some stuff on accident...
          Hide
          Hari Shreedharan added a comment - - edited

          Brock - I'd like to get this into v1.3.0. I am hoping that this will get committed soon. So I'm leaving the target version as 1.3.0 itself.

          Show
          Hari Shreedharan added a comment - - edited Brock - I'd like to get this into v1.3.0. I am hoping that this will get committed soon. So I'm leaving the target version as 1.3.0 itself.
          Hide
          Mike Percy added a comment -

          Patch committed. Thanks Hari!

          Show
          Mike Percy added a comment - Patch committed. Thanks Hari!
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #316 (See https://builds.apache.org/job/flume-trunk/316/)
          FLUME-1199. Add HTTP Post Source. (Revision bc1928bc2e23293cb20f4bc2693a3bc262f507b3)

          Result = SUCCESS
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=bc1928bc2e23293cb20f4bc2693a3bc262f507b3
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java
          • flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java
          • flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java
          • flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPBadRequestException.java
          • flume-ng-core/pom.xml
          • flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-sdk/src/main/java/org/apache/flume/event/JSONEvent.java
          • pom.xml
          • flume-ng-core/src/test/java/org/apache/flume/source/http/TestHTTPSource.java
          • flume-ng-core/src/test/java/org/apache/flume/source/http/FlumeHttpServletRequestWrapper.java
          • flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java
          Show
          Hudson added a comment - Integrated in flume-trunk #316 (See https://builds.apache.org/job/flume-trunk/316/ ) FLUME-1199 . Add HTTP Post Source. (Revision bc1928bc2e23293cb20f4bc2693a3bc262f507b3) Result = SUCCESS mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=bc1928bc2e23293cb20f4bc2693a3bc262f507b3 Files : flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPBadRequestException.java flume-ng-core/pom.xml flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sdk/src/main/java/org/apache/flume/event/JSONEvent.java pom.xml flume-ng-core/src/test/java/org/apache/flume/source/http/TestHTTPSource.java flume-ng-core/src/test/java/org/apache/flume/source/http/FlumeHttpServletRequestWrapper.java flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java
          Hide
          Hudson added a comment -

          Integrated in flume-1.3.0 #12 (See https://builds.apache.org/job/flume-1.3.0/12/)
          FLUME-1199. Add HTTP Post Source. (Revision 0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c)

          Result = FAILURE
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java
          • flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPBadRequestException.java
          • flume-ng-core/pom.xml
          • flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java
          • flume-ng-sdk/src/main/java/org/apache/flume/event/JSONEvent.java
          • flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java
          • flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java
          • flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/test/java/org/apache/flume/source/http/TestHTTPSource.java
          • pom.xml
          • flume-ng-core/src/test/java/org/apache/flume/source/http/FlumeHttpServletRequestWrapper.java
          Show
          Hudson added a comment - Integrated in flume-1.3.0 #12 (See https://builds.apache.org/job/flume-1.3.0/12/ ) FLUME-1199 . Add HTTP Post Source. (Revision 0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c) Result = FAILURE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c Files : flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPBadRequestException.java flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java flume-ng-sdk/src/main/java/org/apache/flume/event/JSONEvent.java flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/test/java/org/apache/flume/source/http/TestHTTPSource.java pom.xml flume-ng-core/src/test/java/org/apache/flume/source/http/FlumeHttpServletRequestWrapper.java

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Roko Kruze
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development