[MAPREDUCE-212] want InputFormat for task logs

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Assignee: Unassigned
• Reporter: Doug Cutting
• Created: 03/Apr/07 18:00
• Updated: 18/Jul/14 05:06
• Resolved: 18/Jul/14 05:06

Description

We should provide an InputFormat implementation that includes all the task logs from a job. Folks should be able to do something like:

job = new JobConf();
job.setInputFormatClass(TaskLogInputFormat.class);
TaskLogInputFormat.setJobId(jobId);
...

Tasks should ideally be localized to the node that each log is on.

Examining logs should be as lightweight as possible, to facilitate debugging. It should not require a copy to HDFS. A faster debug loop is like a faster search engine: it makes people more productive. The sooner one can find that, e.g., most tasks failed with a NullPointerException on line 723, the better.