[CHUKWA-678] Make use of ChukwaWriter in agent

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Component/s: Data Collection

Environment

MacOSX, Java 6

• Assignee: shreyas subramanya
• Reporter: shreyas subramanya
• Created: 03/Jan/13 03:30
• Updated: 30/Jun/14 02:14
• Resolved: 30/Jun/14 02:14

Description

The chukwa agent sends out data chunks to various destinations through the combination of Connector and ChukwaSender interfaces. For sending chunks to collector, we have http implementation of these interfaces. The collector writes out the received chunks to various destinations through classes implementing ChukwaWriter interface. Optionally, multiple destinations can be chosen by specifying PipelineStageWriter.

The proposal is to:
1. Use ChukwaWriter to send out data chunks to multiple destinations from the agent. Further, PipelinestageWriter can be made default and pipeline configuration specified in the agent config file
2. Implement (or modify) Pipelineable writers for HBase, Http, Hdfs and WebHdfs
3. Do away with the Connector interface and have a single non configurable connector object as part of the agent. This class initiates the configured writer, waits for data chunks and passes the chunks to Writer.add()/send(). Connection protocol for each destination is handled by the init() of the individual writers.

Considerations:
1. In case of Pipelineable writers, we need a way to merge the results of each pipeline stage before the agent commits the chunk.
2. Handling pipeline failure