[CHUKWA-78] Track map reduce cluster usage through through job history and jobconf

• Type: New Feature
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Component/s: Data Collection, Data Processors

Environment

Redhat 5.1, Java 6

• Assignee: Eric Yang
• Reporter: Eric Yang
• Created: 01/Apr/09 05:14
• Updated: 08/Apr/09 20:56
• Resolved: 08/Apr/09 20:56

Description

Job history data and jobconf contains valuable information to determine the usage of the map reduce job cluster. The request feature is to create the following components:

• ChukwaJobTrackerInstrumentation API to stream job history and jobconf file to Chukwa.
• parsers to collect job data.
• MDL dictionary to load key/value pair for Job data to database.
• Usage report of the cluster base on user, and job queues.

Issue Links

• depends upon: CHUKWA-55 Collect and aggregate slot-hours usage by user, and by scheduler queues.
• depends upon: CHUKWA-61 Provide usage reporting widget(s)
• is depended upon by: CHUKWA-50 Create parser for parsing job history log file and job conf file in demux