Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-445

real-time buffer and display at Collector

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It would be really cool to have the collector buffer a certain amount of received data, and make it available via the web interface.

        Activity

        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #330 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/330/ )
        Hide
        asrabkin Ari Rabkin added a comment -

        Committed this to both branch and trunk.

        Show
        asrabkin Ari Rabkin added a comment - Committed this to both branch and trunk.
        Hide
        eyang Eric Yang added a comment -

        +1 Looks good.

        Show
        eyang Eric Yang added a comment - +1 Looks good.
        Hide
        asrabkin Ari Rabkin added a comment -

        Last call for objections here; I intend to commit this to both trunk and the 0.4 branch, barring comment.

        Show
        asrabkin Ari Rabkin added a comment - Last call for objections here; I intend to commit this to both trunk and the 0.4 branch, barring comment.
        Hide
        asrabkin Ari Rabkin added a comment -

        I'd like to commit this. Objections? if it turns out not to be very useful, we can deprecate it and nobody will the wiser.

        Show
        asrabkin Ari Rabkin added a comment - I'd like to commit this. Objections? if it turns out not to be very useful, we can deprecate it and nobody will the wiser.
        Hide
        asrabkin Ari Rabkin added a comment -

        I think the tradeoff is whether you think collectors are heavily loaded or mostly idle. At Berkeley, they're mostly idle. So it seemed extravagant to add a whole 'nother process to display logs. If your collectors are heavily loaded, it makes sense to pull this functionality onto the HICC server.

        Show
        asrabkin Ari Rabkin added a comment - I think the tradeoff is whether you think collectors are heavily loaded or mostly idle. At Berkeley, they're mostly idle. So it seemed extravagant to add a whole 'nother process to display logs. If your collectors are heavily loaded, it makes sense to pull this functionality onto the HICC server.
        Hide
        eyang Eric Yang added a comment -

        I also have an implementation of this, but I made a queue buffering on the HICC side to keep the collector simple. This could be accomplished by using:

        new SocketDataLoader("all");

        Either implementation works. Let's think this over. What are the pros/cons of having the buffer in the collector?

        Show
        eyang Eric Yang added a comment - I also have an implementation of this, but I made a queue buffering on the HICC side to keep the collector simple. This could be accomplished by using: new SocketDataLoader("all"); Either implementation works. Let's think this over. What are the pros/cons of having the buffer in the collector?
        Hide
        asrabkin Ari Rabkin added a comment -

        Want to test this at Berkeley before proposing to commit.

        Show
        asrabkin Ari Rabkin added a comment - Want to test this at Berkeley before proposing to commit.
        Hide
        asrabkin Ari Rabkin added a comment -

        A pipeline writer stage and a servlet to display the collected data.

        Show
        asrabkin Ari Rabkin added a comment - A pipeline writer stage and a servlet to display the collected data.
        Hide
        asrabkin Ari Rabkin added a comment -

        In small to midsize deployments, it would be nice for collectors to cache, say, the last megabyte of received log data, and make it visible via the web.

        Intended implementation: Have a web servlet coupled to a collector pipeline stage.
        Keep a total of 1 MB of chunks. Keep an index by stream. Let users browse, via the web, to an arbitrary stream and show all the data in that stream.

        Show
        asrabkin Ari Rabkin added a comment - In small to midsize deployments, it would be nice for collectors to cache, say, the last megabyte of received log data, and make it visible via the web. Intended implementation: Have a web servlet coupled to a collector pipeline stage. Keep a total of 1 MB of chunks. Keep an index by stream. Let users browse, via the web, to an arbitrary stream and show all the data in that stream.

          People

          • Assignee:
            asrabkin Ari Rabkin
            Reporter:
            asrabkin Ari Rabkin
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development