[HADOOP-4462] Rolling mechanism for demux output

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Jerome Boulon
• Reporter: Jerome Boulon
• Created: 20/Oct/08 16:23
• Updated: 08/Jul/09 16:40
• Resolved: 20/Dec/08 00:34

Description

In order to reduce the number of file on HDFS we need to have a rolling mechanism for the demux output

• avoid immediate merging if there's already file for the same time range, create a spill file instead
• merge all raw files every hours
• merge all hourly files every days

Issue Links

• is part of: HADOOP-4709 Update chukwa source code and develop chukwa from public repository