Details

    • Type: New Feature New Feature
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v0.9.4
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      For web dashboards, it's useful to have Flume just fire an HTTP POST with event data to a waiting URL.

        Activity

        Hide
        Disabled imported user added a comment -

        I've got a simple implementation up at http://github.com/hammer/flume that works for me, but could use parametrization and some tests.

        Show
        Disabled imported user added a comment - I've got a simple implementation up at http://github.com/hammer/flume that works for me, but could use parametrization and some tests.
        Hide
        Disabled imported user added a comment -

        Migrating some comments over from Github

        • Don't URLEncode "body" every time
        • Open the connection in open()
        • Use rpt.setLongMetric instead of Attributes.setLong

        Also I need to refactor this to allow for parametrized GET parameters and cookies.

        Show
        Disabled imported user added a comment - Migrating some comments over from Github Don't URLEncode "body" every time Open the connection in open() Use rpt.setLongMetric instead of Attributes.setLong Also I need to refactor this to allow for parametrized GET parameters and cookies.
        Hide
        Jonathan Hsieh added a comment -

        Also needs some test cases.

        Show
        Jonathan Hsieh added a comment - Also needs some test cases.
        Hide
        Disabled imported user added a comment -

        I'm adding my general notes to http://www.quora.com/How-can-I-add-a-new-source-or-sink-to-Flume.

        One thing I notice about Jon's RabbitMQ plugin: it doesn't use a logger. Should we figure out a way for plugins to use the Flume logger?

        Show
        Disabled imported user added a comment - I'm adding my general notes to http://www.quora.com/How-can-I-add-a-new-source-or-sink-to-Flume . One thing I notice about Jon's RabbitMQ plugin: it doesn't use a logger. Should we figure out a way for plugins to use the Flume logger?
        Hide
        Disabled imported user added a comment -

        Another difference: in the builder, the plugin throws an IllegalArgumentException rather than a FlumeSpecException.

        Show
        Disabled imported user added a comment - Another difference: in the builder, the plugin throws an IllegalArgumentException rather than a FlumeSpecException.
        Hide
        Disabled imported user added a comment -

        FLUME-83 and FLUME-84 would have been helpful here.

        Show
        Disabled imported user added a comment - FLUME-83 and FLUME-84 would have been helpful here.
        Hide
        Disabled imported user added a comment -

        FLUME-85 too (we really need to allow links in this JIRA)

        Show
        Disabled imported user added a comment - FLUME-85 too (we really need to allow links in this JIRA)
        Hide
        Disabled imported user added a comment -

        It's also a bit odd that we use the same logic to load plugins in SinkFactoryImpl.java and SourceFactoryImpl.java; we should do that only once.

        Lastly, I noticed that Flume doesn't pick up $FLUME_HOME/conf/flume-site.xml; it goes directly to /etc/flume/conf/flume-site.xml. Seems like that should be configurable.

        Show
        Disabled imported user added a comment - It's also a bit odd that we use the same logic to load plugins in SinkFactoryImpl.java and SourceFactoryImpl.java; we should do that only once. Lastly, I noticed that Flume doesn't pick up $FLUME_HOME/conf/flume-site.xml; it goes directly to /etc/flume/conf/flume-site.xml. Seems like that should be configurable.
        Hide
        Disabled imported user added a comment -

        Okay, I figured out why Flume is automagically reading configuration information from /etc:

        export FLUME_CONF_DIR=${FLUME_CONF_DIR:-/etc/flume/conf/}

        happens in bin/flume. I am not sure it's sane to do that?

        Show
        Disabled imported user added a comment - Okay, I figured out why Flume is automagically reading configuration information from /etc: export FLUME_CONF_DIR=${FLUME_CONF_DIR:-/etc/flume/conf/} happens in bin/flume . I am not sure it's sane to do that?
        Hide
        Disabled imported user added a comment -

        Moved to the plugin architecture and made the POST parameter and cookie content configurable. Should be usable for anyone who wants it at http://github.com/hammer/flume/blob/master/plugins/http/src/java/com/cloudera/flume/handlers/http/HttpPostSink.java.

        Show
        Disabled imported user added a comment - Moved to the plugin architecture and made the POST parameter and cookie content configurable. Should be usable for anyone who wants it at http://github.com/hammer/flume/blob/master/plugins/http/src/java/com/cloudera/flume/handlers/http/HttpPostSink.java .
        Hide
        Disabled imported user added a comment -

        Note that I didn't add tests because the current example tests for plugins use a deprecated API. Waiting on FLUME-86 before cleaning up for commit.

        Show
        Disabled imported user added a comment - Note that I didn't add tests because the current example tests for plugins use a deprecated API. Waiting on FLUME-86 before cleaning up for commit.
        Hide
        Patrick Hunt added a comment -

        re FLUME_CONF_DIR we do log which directory we are using.

        also notice in this pending review that bin/flume has been updated to check for dev env (local conf) and use that (prefer) if available
        https://review.cloudera.org/r/391/

        Show
        Patrick Hunt added a comment - re FLUME_CONF_DIR we do log which directory we are using. also notice in this pending review that bin/flume has been updated to check for dev env (local conf) and use that (prefer) if available https://review.cloudera.org/r/391/
        Hide
        Disabled imported user added a comment -

        Well, I just think we have too many places where we might set the path: the bin/flume script sets the conf path if the environment variable is not set, but we also have logic for doing that in FlumeConfiguration.java. We should pick one place to handle an unset environment variable.

        Show
        Disabled imported user added a comment - Well, I just think we have too many places where we might set the path: the bin/flume script sets the conf path if the environment variable is not set, but we also have logic for doing that in FlumeConfiguration.java. We should pick one place to handle an unset environment variable.
        Hide
        Patrick Hunt added a comment -

        but we also have logic for doing that in FlumeConfiguration.java

        ah, ok, I see. Perhaps we should just exit gracefully, with actionable error
        message, if FLUME_CONF_DIR is not set once we get to that code in FlumeConfiguration.java?
        (if you agree could you open a new jira to address?)

        Show
        Patrick Hunt added a comment - but we also have logic for doing that in FlumeConfiguration.java ah, ok, I see. Perhaps we should just exit gracefully, with actionable error message, if FLUME_CONF_DIR is not set once we get to that code in FlumeConfiguration.java? (if you agree could you open a new jira to address?)
        Hide
        Disabled imported user added a comment -

        I actually think handling the missing env variable in the code rather than in the bin scripts is a better idea. Some day we'd like people to manage their cluster in a more scalable fashion than bash scripts so keeping those scripts stupid seems like a good idea to me.

        Show
        Disabled imported user added a comment - I actually think handling the missing env variable in the code rather than in the bin scripts is a better idea. Some day we'd like people to manage their cluster in a more scalable fashion than bash scripts so keeping those scripts stupid seems like a good idea to me.
        Hide
        Alexander Alten-Lorenz added a comment -

        deprecated, flumeNG (1.0x)

        Show
        Alexander Alten-Lorenz added a comment - deprecated, flumeNG (1.0x)

          People

          • Assignee:
            Alexander Alten-Lorenz
            Reporter:
            Disabled imported user
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development