[CHUKWA-441] Real time hadoop activity monitoring capabilities via Client Trace Logs

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Component/s: Data Collection, User Interface

Environment

Redhat EL 5.1, Java 6

• Assignee: Eric Yang
• Reporter: Eric Yang
• Created: 30/Dec/09 00:01
• Updated: 03/Jan/10 15:52
• Resolved: 02/Jan/10 18:35

Description

Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.

The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:

Adaptor > Agent -> Collector -> SocketTeeWriter < Socket Data Loader <- HICC