[#HADOOP-4709] Update chukwa source code and develop chukwa from public repository

Hadoop Common

Update chukwa source code and develop chukwa from public repository

Created: 21/Nov/08 10:15 PM Updated: 08/Jul/09 04:40 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

None

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
HADOOP-4709-3.patch	2008-12-01 12:14 AM	Eric Yang	1.85 MB
HADOOP-4709-binary.tar.gz	2008-11-26 08:12 AM	Eric Yang	60 kB

Environment:

Redhat EL 4.5, Java 1.6, Hadoop trunk

Issue Links:

Incorporates

This issue incorporates:

~~HADOOP-4710~~ Chukwa - Add duplicate detection, and implement virtual offset of the log file to checkpoint file

~~HADOOP-4811~~ Import style sheet images for Chukwa

~~HADOOP-4529~~ Support CHUKWA_CONF_DIR environment property for configuration storage

~~HADOOP-4462~~ Rolling mechanism for demux output

~~HADOOP-4463~~ Utility classes for Archive and ChukwaRecord files

~~HADOOP-4461~~ Update Chukwa parsers

~~HADOOP-4460~~ Update Chukwa Demux process

~~HADOOP-4459~~ - Improve Chukwa Collector

~~HADOOP-4434~~ Improve log rotation and reduce communication between agent and logging process, make agent more fault tolerant

Release Note:

« Hide

What is new in ~~HADOOP-4709~~:

  Dashboard for visual Hadoop metrics and system metrics.
  Database loader for loading chukwa records to jdbc connected database.
  FileAdaptor for streaming small file as one record for data processing.
  Scripts for capturing tcpdump output and stream to Chukwa for archive.

Improvement

  New load blanace algorithm for ChukwaAgent to distribute data to Chukwa Collectors more evenly.
  Unit test and validation for agent, collector, and demux map reduce job.

Contributed by Jerome Boulon, Andy Konwinski, Ariel Rabkin, Eric Yang

Resolution Date:

05/Dec/08 08:30 PM

Description

« Hide

New features that were added to Chukwa source code since last time chukwa was committed to SVN repository:

Added Hadoop Infrastructure Care Center (UI for visualize data collected by Chukwa)
Added FileAdaptor for streaming small file in one chunk
Added compression to archive and demux output
Added unit tests and validation for agent, collector, and demux map reduce job
Added database loader for loading demux output (sequence file) to jdbc connected database
Added algorithm to distribute collector load more evenly

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

No work has yet been logged on this issue.