[HADOOP-4149] JobQueueJobInProgressListener.jobUpdated() might not work as expected

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.19.0
• Fix Version/s: 0.19.0
• Hadoop Flags: Reviewed
• Assignee: Amar Kamat Assign to me
• Reporter: Amar Kamat
• Created: 10/Sep/08 14:32
• Updated: 08/Jul/09 16:53
• Resolved: 20/Oct/08 23:53

Description

JobQueueJobInProgressListener uses a TreeSet to store the sorted collection of JobInProgress objects. The comparator used to sort the JIPs follow the following order

• priority (>=)
• start time (<=)
• job id [jt-identifier, job-index] (<=)

If any JIP object is changed w.r.t priority or start-time, then the TreeSet will be inconsistent. Hence doing a delete might not work. Consider the following
1) jobs are submitted in the following order

number	jobid	priority
1	j1	NORMAL
2	j2	LOW
3	j3	NORMAL

2) The sorted collection will be in the order : j1,j3,j2

3) If job3's priority is changed to LOW then the collection wont change but delete will bail out on j1 itself as the comparator will return a -ve number. TreeSet uses the comparator both for sorting and deleting. If i indicates the index in the collection and obj represents the object under consideration, then looks like TreeSet.remove(obj) follows something like :

• continue to search if the compare(i, obj) is -ve
• bail out if the compare(i, obj) is +ve
• delete the obj of compare(i,obj) == 0

Issue Links

• is blocked by: HADOOP-4053 Schedulers need to know when a job has completed