Chukwa
  1. Chukwa
  2. CHUKWA-358

pluggable real-time monitoring at collectors

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.3.0
    • Component/s: Data Collection
    • Labels:
      None
    • Release Note:
      Support for real-time monitoring at collector.

      Description

      Collectors could allow real-time data monitoring

      1. improvedRealtime.patch
        9 kB
        Ari Rabkin
      2. CHUKWA-358.patch
        22 kB
        Ari Rabkin

        Issue Links

          Activity

          Hide
          Ari Rabkin added a comment -

          design is as follows:
          create a new Pipelineable writer, that accepts filters via a very simple socket API. One filter per socket. It then applies all active filters to chunks as they go past; if they match, chunks are written back across the socket.
          Filters expire when the associated socket is closed.

          Show
          Ari Rabkin added a comment - design is as follows: create a new Pipelineable writer, that accepts filters via a very simple socket API. One filter per socket. It then applies all active filters to chunks as they go past; if they match, chunks are written back across the socket. Filters expire when the associated socket is closed.
          Hide
          Ari Rabkin added a comment -

          Developed in response to operational needs here at Berkeley – but should be useful more broadly.

          Show
          Ari Rabkin added a comment - Developed in response to operational needs here at Berkeley – but should be useful more broadly.
          Hide
          Jiaqi Tan added a comment -

          Would this be effectively creating a fork at the collector, one allowing the current path for writing to HDFS, the other allowing a real-time query?

          Show
          Jiaqi Tan added a comment - Would this be effectively creating a fork at the collector, one allowing the current path for writing to HDFS, the other allowing a real-time query?
          Hide
          Ari Rabkin added a comment -

          Yes, that's a fair summary of the intent.

          Show
          Ari Rabkin added a comment - Yes, that's a fair summary of the intent.
          Hide
          Ari Rabkin added a comment -

          Note that reorganizing the filtering in DumpChunks incorporated CHUKWA-357 into this change.

          Show
          Ari Rabkin added a comment - Note that reorganizing the filtering in DumpChunks incorporated CHUKWA-357 into this change.
          Hide
          Jerome Boulon added a comment -

          Just my 0.02 c

          It's inline with one proposal I have at Yahoo! to integrate Chukwa with Yahoo! Real time monitoring stack and building that behind the ChukwaWriter interface it's a good think.
          So it's definitively a +1 on the idea however instead of reimplementing yet another protocol I will prefer to leverage something like Thrift for now and may be Avro when the RPC stack will be there. The reason being that Thrift is well debugged and provide support a large number of language.

          Show
          Jerome Boulon added a comment - Just my 0.02 c It's inline with one proposal I have at Yahoo! to integrate Chukwa with Yahoo! Real time monitoring stack and building that behind the ChukwaWriter interface it's a good think. So it's definitively a +1 on the idea however instead of reimplementing yet another protocol I will prefer to leverage something like Thrift for now and may be Avro when the RPC stack will be there. The reason being that Thrift is well debugged and provide support a large number of language.
          Hide
          Eric Yang added a comment -

          +1 Looks good.

          Show
          Eric Yang added a comment - +1 Looks good.
          Hide
          Ari Rabkin added a comment -

          For now, this was about the most minimal possible protocol. RPC seemed like overkill for this sort of thing. And we really don't want to force clients to use Thrift. It should be possible to integrate Thrift or Avro into this sort of mechanism if the need arises.

          Show
          Ari Rabkin added a comment - For now, this was about the most minimal possible protocol. RPC seemed like overkill for this sort of thing. And we really don't want to force clients to use Thrift. It should be possible to integrate Thrift or Avro into this sort of mechanism if the need arises.
          Hide
          Ari Rabkin added a comment -

          I just committed this, with some modifications to improve robustness and flexibility.

          Show
          Ari Rabkin added a comment - I just committed this, with some modifications to improve robustness and flexibility.
          Hide
          Ari Rabkin added a comment -

          Turns out the previous thread model was too simplistic.

          Show
          Ari Rabkin added a comment - Turns out the previous thread model was too simplistic.
          Hide
          Ari Rabkin added a comment -

          Patch includes modified tests, including small fix for TestCmd.

          Show
          Ari Rabkin added a comment - Patch includes modified tests, including small fix for TestCmd.
          Hide
          Jerome Boulon added a comment -

          Ari could you explain what was too simplistic?
          Thanks.

          Show
          Jerome Boulon added a comment - Ari could you explain what was too simplistic? Thanks.
          Hide
          Ari Rabkin added a comment -

          Previous model was using Jetty threads to push out real-time notification. This is bad, because it means that broken sockets down in the realtime mechanism can gum up the rest of the collector. I'm now introducing a queue and a separate thread per realtime channel.

          Note that if you don't use the SocketTeeWriter, none of this applies.

          Show
          Ari Rabkin added a comment - Previous model was using Jetty threads to push out real-time notification. This is bad, because it means that broken sockets down in the realtime mechanism can gum up the rest of the collector. I'm now introducing a queue and a separate thread per realtime channel. Note that if you don't use the SocketTeeWriter, none of this applies.
          Hide
          Ari Rabkin added a comment -

          barring objections, I'm going to commit this tonight or tomorrow.

          Show
          Ari Rabkin added a comment - barring objections, I'm going to commit this tonight or tomorrow.
          Hide
          Ari Rabkin added a comment -

          Committed.

          Show
          Ari Rabkin added a comment - Committed.

            People

            • Assignee:
              Ari Rabkin
              Reporter:
              Ari Rabkin
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development