[MAPREDUCE-1773] streaming doesn't support jobclient.output.filter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.22.0
• Component/s: contrib/streaming

Environment

Linux

• Hadoop Flags: Reviewed
• Release Note: Improved console messaging for streaming jobs by using the generic JobClient API itself instead of the existing streaming-specific code.
• Assignee: Amareshwari Sriramadasu
• Reporter: Alok Singh
• Created: 11/Jul/09 00:17
• Updated: 12/Dec/11 06:20
• Resolved: 01/Jun/10 11:40

Description

the streaming Jobclient implementation i.e contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java is significantly different than the core hadoop mapred/org/apache/hadoop/mapred/JobClient.java.

for example unlike StreamJob.java, JobClient.java it gets tasks log when jobclient.output.filter=ALL is specified .

With hod-logs going away in hadoop 0.20 (due to new scheduler) user has no good way of programmitically getting logs

We should have intermediate adaptor class to implement Tools for the purpose of submitting jobs via m/r, streaming, pipes so that we don't miss some core functionality.

GenericJobClient implements Tools and then StreamJob extends GenericJobClient, JobClient extends GenericJobClient

Alok