Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: notrack
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Appears to be in progress at http://github.com/tallpsmith/elasticflume

        Activity

        Arvind Prabhakar made changes -
        Fix Version/s notrack [ 12320245 ]
        Mark Thomas made changes -
        Project Import Tue Aug 02 16:57:12 UTC 2011 [ 1312304232406 ]
        Hide
        Jonathan Hsieh added a comment - - edited

        It think his account may have been renamed – my fork points to this as the root now:

        https://github.com/Aconex/elasticflume

        Show
        Jonathan Hsieh added a comment - - edited It think his account may have been renamed – my fork points to this as the root now: https://github.com/Aconex/elasticflume
        Hide
        flume_bobgreen5s added a comment -

        Where did this plugin go? The page is down...

        Show
        flume_bobgreen5s added a comment - Where did this plugin go? The page is down...
        flume_bobgreen5s made changes -
        Comment [ Where did this plugin go? The page is down... ]
        Hide
        Bruce Mitchener added a comment -

        I'd like to see that posted to the flume-dev list ... I have some thoughts / comments on that but it isn't really appropriate for this bug.

        Show
        Bruce Mitchener added a comment - I'd like to see that posted to the flume-dev list ... I have some thoughts / comments on that but it isn't really appropriate for this bug.
        Hide
        Disabled imported user added a comment -

        Very nifty. What is the intended model for distributing sources, sinks, and decorators written by third parties? Should we create some sort of app store for this sort of thing, or are we going to try to get them all into the Flume release? Perhaps I should start this discussion on the Flume mailing list?

        Show
        Disabled imported user added a comment - Very nifty. What is the intended model for distributing sources, sinks, and decorators written by third parties? Should we create some sort of app store for this sort of thing, or are we going to try to get them all into the Flume release? Perhaps I should start this discussion on the Flume mailing list?
        Jonathan Hsieh made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Jonathan Hsieh added a comment -
        Show
        Jonathan Hsieh added a comment - Plugin available here: http://github.com/tallpsmith/elasticflume
        Hide
        Jonathan Hsieh added a comment -

        I've played with any have been able to use this plug in. Nifty nifty.

        Show
        Jonathan Hsieh added a comment - I've played with any have been able to use this plug in. Nifty nifty.
        Hide
        Jonathan Hsieh added a comment -

        Saw this on the twitters and it looks really interesting! Our main use case (writing to HDFS) is scalable so we could add more collectors and write to different files to increase throughput. Looking at the elastic search pages, my impression is that the intended scaling characteristics of the indexing operations is to be scalable as well.

        Show
        Jonathan Hsieh added a comment - Saw this on the twitters and it looks really interesting! Our main use case (writing to HDFS) is scalable so we could add more collectors and write to different files to increase throughput. Looking at the elastic search pages, my impression is that the intended scaling characteristics of the indexing operations is to be scalable as well.
        Hide
        flume_psmith@apache.org added a comment -

        Hi, yeah I'll be further developing this as I gain experience with both Flume and ElasticSearch.

        My biggest concern so far is that it may be possible for ElasticSearch (or any indexing based operation) to be a 'slow consumer' problem. I'm going to need to understand what impact a slow consumer of ElasticSearch may have on the original agent/collector area.

        For my own use case I'm ok with the data not arriving in 'near real time' as long as it gets there eventually (catches up).

        Show
        flume_psmith@apache.org added a comment - Hi, yeah I'll be further developing this as I gain experience with both Flume and ElasticSearch. My biggest concern so far is that it may be possible for ElasticSearch (or any indexing based operation) to be a 'slow consumer' problem. I'm going to need to understand what impact a slow consumer of ElasticSearch may have on the original agent/collector area. For my own use case I'm ok with the data not arriving in 'near real time' as long as it gets there eventually (catches up).
        Disabled imported user created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Disabled imported user
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development