Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

• Key: MAPREDUCE-381
• Type: Sub-task
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Devaraj Das
• Reporter: Devaraj Das

Description

Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

Comments

[Devaraj Das]

Attached patch provides the functionality. The patch adds a new class TaskInProgressStatus that is passed along in the status report (like when the method JobSubmissionProtocol.getMapTaskReports() is invoked) to the client. Also, the ClusterStatus now has the list of active trackers.
Provided the following new command line options in the JobClient:
1) -list-trackers : display the list of active trackers in the cluster
2) -list-attempt-ids <jobId> <task-type> <task-state> : displays the list of tasks for a given job of a give type (like map or reduce) currently in a particular state (running or completed).

[Arun C Murthy]

Minor comments:

1. I'd suggest we deprecate both the current ClusterStatus constructors and use only the new one - this helps with the complication in write and readFields too.
2. Rather than introduce {set|is}{Failed|Pending|Completed|Running|Killed} methods in TaskInProgressStatus I'd suggest a single {get|set}Status which takes/returns the enum which is called with appropriate values from TaskInProgress.generateSingleReport.
3. There are some debugging statements left over.

Long term: It would be really great to use the TaskInProgressStatus.TaskInProgressStatusType to maintain the TIP's state rather than all the booleans...

[Arun C Murthy]

Super minor nit: I'd prefer to rename TaskInProgressStatusType to just Status! smile

[Sharad Agarwal]

Clients may be interested in names of blacklisted trackers as well. I think -list-trackers should list all by default and perhaps take an argument to list blacklisted/active ones ?

[Devaraj Das]

Thanks Arun/Sharad for looking at this. I have incorporated the comments.
1) Made TaskInProgressStatusType a separate class all by itself and called it TIPStatus
2) Factored TaskInProgressStatus functionality into TaskReport since that is where it is needed
3) Added a new getClusterStatus method that takes a boolean argument and depending on whether it is true, sets the task tracker names as well. If false, only counts of blacklisted/active trackers are set (as is the behavior today). I did this since I wanted to avoid iterations over the tasktracker list for every call to assignTasks in CapacityScheduler, for example.

[Sharad Agarwal]

looks good.
Please take a note to document the new commands to the commands manual.

[Devaraj Das]

All test cases and test-patch passed. I committed this.