|
The information disparity comes in because, when a task is scheduled to be executed the Capacity Scheduler increases the number of the tasks running in a queue and then it passes the actual list of the tasks to the task tracker. So there is a disparity between what is actually running and what is scheduled to run on the task tracker.
So I think we should be changing the Number of Running <task> to represent Running <task> Scheduled and <Task> To be scheduled. Would that make sense to the end user? Once the number of tasks is incremented in the scheduler, we have assigned it to the TaskTracker. So, when it comes back in the next heartbeat, this is going to be incremented on the JT. Is this understanding correct ?
If yes, when the TTs come in for the next heartbeat, the difference in maps should not be as high as shown in the bug comment. Is this happening ? I had an offline discussion with Karam about this issue, as mentioned in his comment, this indeed happens for a short duration between heartbeats when the task slots are actually getting assigned for a particular job. And the cluster information and scheduling information get synched up in the next heartbeat.
In that case, I am thinking if it makes sense to not show this information at all in the scheduler's UI. Essentially it is a measure of how many tasks are scheduled, which mostly would become equal to number of tasks run in the next heartbeat. By removing this information, the user does not get to see how much is scheduled for a while. I think that is ok. So I would vote for removing this info from the scheduler UI. Others ?
+1 for removing it from the scheduler information displayed, given that there is nothing else we can do. But I think this should be done only after fixing
I think, we should be remove the redundant information which is being displayed by the task scheduler as, we are already displaying the number of maps and reduces running cluster wide and job wise split in the job details table.
I am marking
I am marking
The number of Waiting {Maps|Reduces} reduces is derived from the running counts which are not synchronized with the exact counts all the time. So, they won't be correct all the time, even though at the point of their actual usage in the scheduler we make sure that they are correct. Hence, the waiting counts should be removed from the scheduler information.
The correct waiting counts can be maintained by JT(just like running counts) and stored in the cluster-summary; will file a JIRA to get that information there. Hemanth and I looked at what's going on here. Essentially, there are two sources of truth, regarding the number of running tasks in the system. Each JobInProgress object maintains counts of running map and reduce tasks. These counts are incremented when a task is assigned to a TT (in obtainNewMapTask() or obtainNewReduceTask()). These counts are used the by the CapacityScheduler . The cluster summary, represented by the ClusterStatus object, also contains counts of the total number of maps and reduce tasks. These are incremented by the JT using the TT status. The counts maintained by the JobInProgress objects and the ClusterStatus object, are off by a heartbeat. The former increments its counts when a task is assigned. Once the task runs on a TT, its running status is conveyed to the JT in the TT's next heartbeat. During startup, a lot of TTs approach the JT for tasks to run. As a result, the counts of running tasks across all JobInProgress objects are much higher than the cluster count, since the cluster count is updated only when the TTs report their status in their next hearbeat. That explains the discrepancy reported in this Jira. In steady state, these two counts are mostly identical, or off by a little bit, as TTs finish their tasks at different times.
This is not really a bug, as it's not clear which count is 'correct'. We're reporting from two different sources: the cluster summary and the Scheduler (which gets it info from the JobInProgress objects). But different numbers do get reflected in the UI. So the best fix is to probably indicate in the Scheduler part of the UI that its computation is off from the cluster summary by a heartbeat. Maybe a little explanation in the bottom that says something like: "This info varies from that of the cluster summary by a heartbeat". I don't think we should change anything in the scheduler or the cluster summary. They're both doing the right thing their own way. An alternate solution is to have the cluster summary use the counts from the JobInProgress objects, but this is performance-intensive, and was presumably the reason why the cluster summary maintains its own count. You do want to the leave the rest of the UI as is. The cluster summary is useful, as is the per-queue information of running tasks (reported by the Scheduler) as it lets users know whether the queue is running above/at/below its guaranteed capacity.
The scheduler maintains a partial waiting count of map/reduce tasks. It doesn't need to know the total number of pending tasks if this total is larger than the cluster capacity. So, for performance reasons, it only counts up to the cluster capacity. Vivek's comment
 makes sense. I guess the confusion is because of the lack of information provided along with the counts. If we fix on the naming and say
running task : The task that is currenly getting executed on some tracker scheduled task : The task that is scheduled by the scheduler/jt for running and which might not be actually running then we can show running-tasks info in cluster-summary and scheduled-tasks info in the scheduler's info and get away with the issue. Thoughts?
I personally don't think you want to introduce such fine-grained differences. Having the user think of 'running' tasks and 'scheduled' tasks separately seems unnecessary as it doesn't buy them much. If the numbers are off by a bit, that doesn't seem so bad, as long as you can maybe add a little disclaimer, or provide an explanation in a FAQ. My guess is, users will really be concerned about the stats for their queue. As long as they can see whether a queue is fully occupied or not, and where they are in the waiting list, they should be fine.
At the max the numbers can be off by # nodes in the cluster. There are a few more issues with this information display. I had an offline discussion with Vivek, and we came up with a few observations and ideas.
There are some options to address this:
The advantage with the last two approaches is that we could deterministically say how far off the scheduling info would be, as compared to the cluster status. For e.g. if the updater thread runs once every 30 seconds, we could say the information would be off by 30 seconds. Since in any case it appears that the information cannot be completely in sync, maybe we should leave it simple for now, mark that the information is not synchronized with the cluster status, and see if in steady state the information is way off. If that happens, we could fix it using one of the methods I've stated above. Thoughts ? Instead of relying on freshness of the queue information objects which are used by scheduler to maintain the scheduling information, we can compute the number of running tasks when the information is requested for by the user.
Following can be done for the same:
We also should be fixing HADOOP-4576 Only downside of this approach would be that, whenever there is a UI request we would be iterating thro' running jobs in order to compute the number of running task, which might be a performance problem. Any comments on the approach described above? We can probably check what happens with a reasonable size of running jobs per queue - say 50 jobs. If the performance isn't a concern, this may be a reasonable approach.
We should also check that if this info is periodically checked, say once every 30 seconds or so (equal to a refresh interval on the web page), the performance is still fine.
In case this is turning out to be an issue, caching the values maybe an option. We can update it periodically like I suggested above. Checked with 80 jobs in a queue, the maximum time taken for getting running and pending tasks was 2 milliseconds(inclusive of scheduler.getJobs()) by refreshing the job tracker and queue details page every 15 seconds.
So if we have 10 queues, we would have a maximum delay of 20 milliseconds when user requests for information. This can be further reduced if we get in HADOOP-4576 Marking
Attaching a patch fixing this issue:
The patch does the following: 1. in SchedulingInfo.toString() it does JobQueueManager.getRunningJobs(queue) and iterates thro' running job list and keeps adding number of running tasks in the queue. We need not worry about the jobs which are initialized but not scheduled as those jobs would have running maps and reduces as zero. I have moved the comment from within testSchedulingInfo() to outside. I have removed the FakeJobInProgress.fail() and I am using taskTrackerManager.finalizeJob api instead as the latter raises events which is required for cleaning up the running job queues. Few comments :
TestCapacityScheduler 1) - String[] infoStrings = schedulingInfo.split("\n"); - + String[] infoStrings = schedulingInfo.split("\n"); Unnecessary diff. 2) I feel we should break testSchedulingInformation() into 3 sub testcases namely
Had an offline discussion with Sreekanth and it seems that we can hold the test case splitting for now. Also we should add a disclaimer that cluster-status and scheduler-info will be off. I personally dont think its worth mentioning the number by which it will be off.
Attaching screen shot so that can show what would be displayed to the user.
Incorporating Amar's comment in the patch.
I am not splitting test case in this patch as, I am testing default scheduling information plus running and waiting count side by side. Even if we are splitting the test case then logical division would be running,waiting and default scheduling information. Because running and waiting information can be tested side by side. The disclaimer which I was testing has been removed as I have changed the disclaimer. Can you please check the same? After offline discussion with Hemanth and Amar, it was decided that we would show used maps and reduce in terms of percentage instead of running maps and reduce. Attaching screenshot with changes made to display percentages.
Changes made to display maps and reduces as Used Maps (%) and Used Reduces (%), made appropriate changes in Test case
Had an offline discussion with Amar, I have incorporated changes which were suggested by him. The screenshot attached displays the UI which would be presented to the user.
I feel Used Maps doesnt convey the sense that the maps are scheduled. Scheduled Maps or Running Maps might make more sense. Also there is no clear cut distinction as to what information is config and what is dynamic. I feel we should have 2 sections namely Config/Constants and Runtime Status. Disclaimers might actually confuse the users.
Attaching latest screenshot incorporating all changes
Attaching patch incorporating comments from above
Output of test-patch:
[exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. Ant test-contrib passed without errors. The patch only touches capacity scheduler and nothing in core. So, I did not run core tests.
I just committed this. Thanks, Sreekanth !
Integrated in Hadoop-trunk #680 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/680/
)
Edit release note for publication.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Closed
Major
Wrong number of running map/reduce tasks are displayed in queue information.
When job starts getting slots, JobTracker UI displayed following information about "Cluster Summary" and "Scheduling Information" -:
Maps=9, Reduces=0, Total Submissions=5, Nodes =106, Map Task Capacity=212 Reduce Task Capacity =212, Avg. Tasks/Node=4.00
Scheduling Information shows -:
Queue Name=default
Guaranteed Capacity Maps (%) :100.0
Guaranteed Capacity Reduces (%) :100.0
Current Capacity Maps : 212
Current Capacity Reduces : 212
User Limit : 25
Reclaim Time limit : 300
Number of Running Maps : 105
Number of Running Reduces : 8
Number of Waiting Maps : 107
Number of Waiting Reduces : 204
Priority Supported : YES
This type of information is displayed only when job is started getting slots.