Chukwa
  1. Chukwa
  2. CHUKWA-358

pluggable real-time monitoring at collectors

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.3.0
    • Component/s: Data Collection
    • Labels:
      None
    • Release Note:
      Support for real-time monitoring at collector.

      Description

      Collectors could allow real-time data monitoring

      1. CHUKWA-358.patch
        22 kB
        Ari Rabkin
      2. improvedRealtime.patch
        9 kB
        Ari Rabkin

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          5m 6s 1 Ari Rabkin 23/Jul/09 07:40
          Resolved Resolved Reopened Reopened
          5d 2h 29m 1 Ari Rabkin 30/Jul/09 00:48
          Reopened Reopened Patch Available Patch Available
          57s 1 Ari Rabkin 30/Jul/09 00:49
          Patch Available Patch Available Resolved Resolved
          2d 17h 45m 2 Ari Rabkin 31/Jul/09 03:56
          Ari Rabkin made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Ari Rabkin added a comment -

          Committed.

          Show
          Ari Rabkin added a comment - Committed.
          Hide
          Ari Rabkin added a comment -

          barring objections, I'm going to commit this tonight or tomorrow.

          Show
          Ari Rabkin added a comment - barring objections, I'm going to commit this tonight or tomorrow.
          Hide
          Ari Rabkin added a comment -

          Previous model was using Jetty threads to push out real-time notification. This is bad, because it means that broken sockets down in the realtime mechanism can gum up the rest of the collector. I'm now introducing a queue and a separate thread per realtime channel.

          Note that if you don't use the SocketTeeWriter, none of this applies.

          Show
          Ari Rabkin added a comment - Previous model was using Jetty threads to push out real-time notification. This is bad, because it means that broken sockets down in the realtime mechanism can gum up the rest of the collector. I'm now introducing a queue and a separate thread per realtime channel. Note that if you don't use the SocketTeeWriter, none of this applies.
          Hide
          Jerome Boulon added a comment -

          Ari could you explain what was too simplistic?
          Thanks.

          Show
          Jerome Boulon added a comment - Ari could you explain what was too simplistic? Thanks.
          Ari Rabkin made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Ari Rabkin made changes -
          Attachment improvedRealtime.patch [ 12414963 ]
          Hide
          Ari Rabkin added a comment -

          Patch includes modified tests, including small fix for TestCmd.

          Show
          Ari Rabkin added a comment - Patch includes modified tests, including small fix for TestCmd.
          Ari Rabkin made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Ari Rabkin added a comment -

          Turns out the previous thread model was too simplistic.

          Show
          Ari Rabkin added a comment - Turns out the previous thread model was too simplistic.
          Ari Rabkin made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Release Note Support for real-time monitoring at collector.
          Resolution Fixed [ 1 ]
          Hide
          Ari Rabkin added a comment -

          I just committed this, with some modifications to improve robustness and flexibility.

          Show
          Ari Rabkin added a comment - I just committed this, with some modifications to improve robustness and flexibility.
          Hide
          Ari Rabkin added a comment -

          For now, this was about the most minimal possible protocol. RPC seemed like overkill for this sort of thing. And we really don't want to force clients to use Thrift. It should be possible to integrate Thrift or Avro into this sort of mechanism if the need arises.

          Show
          Ari Rabkin added a comment - For now, this was about the most minimal possible protocol. RPC seemed like overkill for this sort of thing. And we really don't want to force clients to use Thrift. It should be possible to integrate Thrift or Avro into this sort of mechanism if the need arises.
          Hide
          Eric Yang added a comment -

          +1 Looks good.

          Show
          Eric Yang added a comment - +1 Looks good.
          Hide
          Jerome Boulon added a comment -

          Just my 0.02 c

          It's inline with one proposal I have at Yahoo! to integrate Chukwa with Yahoo! Real time monitoring stack and building that behind the ChukwaWriter interface it's a good think.
          So it's definitively a +1 on the idea however instead of reimplementing yet another protocol I will prefer to leverage something like Thrift for now and may be Avro when the RPC stack will be there. The reason being that Thrift is well debugged and provide support a large number of language.

          Show
          Jerome Boulon added a comment - Just my 0.02 c It's inline with one proposal I have at Yahoo! to integrate Chukwa with Yahoo! Real time monitoring stack and building that behind the ChukwaWriter interface it's a good think. So it's definitively a +1 on the idea however instead of reimplementing yet another protocol I will prefer to leverage something like Thrift for now and may be Avro when the RPC stack will be there. The reason being that Thrift is well debugged and provide support a large number of language.
          Hide
          Ari Rabkin added a comment -

          Note that reorganizing the filtering in DumpChunks incorporated CHUKWA-357 into this change.

          Show
          Ari Rabkin added a comment - Note that reorganizing the filtering in DumpChunks incorporated CHUKWA-357 into this change.
          Ari Rabkin made changes -
          Link This issue incorporates CHUKWA-357 [ CHUKWA-357 ]
          Hide
          Ari Rabkin added a comment -

          Yes, that's a fair summary of the intent.

          Show
          Ari Rabkin added a comment - Yes, that's a fair summary of the intent.
          Hide
          Jiaqi Tan added a comment -

          Would this be effectively creating a fork at the collector, one allowing the current path for writing to HDFS, the other allowing a real-time query?

          Show
          Jiaqi Tan added a comment - Would this be effectively creating a fork at the collector, one allowing the current path for writing to HDFS, the other allowing a real-time query?
          Ari Rabkin made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Ari Rabkin added a comment -

          Developed in response to operational needs here at Berkeley – but should be useful more broadly.

          Show
          Ari Rabkin added a comment - Developed in response to operational needs here at Berkeley – but should be useful more broadly.
          Ari Rabkin made changes -
          Field Original Value New Value
          Attachment CHUKWA-358.patch [ 12414316 ]
          Hide
          Ari Rabkin added a comment -

          design is as follows:
          create a new Pipelineable writer, that accepts filters via a very simple socket API. One filter per socket. It then applies all active filters to chunks as they go past; if they match, chunks are written back across the socket.
          Filters expire when the associated socket is closed.

          Show
          Ari Rabkin added a comment - design is as follows: create a new Pipelineable writer, that accepts filters via a very simple socket API. One filter per socket. It then applies all active filters to chunks as they go past; if they match, chunks are written back across the socket. Filters expire when the associated socket is closed.
          Ari Rabkin created issue -

            People

            • Assignee:
              Ari Rabkin
              Reporter:
              Ari Rabkin
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development