Details
Bug
Issue Links
- is broken by
-
MAPREDUCE-2494
Make the distributed cache delete entires using LRU priority
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
I forgot to add in that I tested this on 0.23, and mrv2 does not have this issue at all. I added in a dist cache entry that takes 30 min to download and the job succeeded.
Moved it to MAPREDUCE. I don't know how it ended up being created in HDFS, I guess I just put it in the wrong spot.
This patch changes the locking on CacheStatus. I have not added in any new tests because it is a complicated bug, that is difficult to reproduce outside of a full cluster. branch-1 seems to have a build issue so I have tested it against 1.0.2. The commit tests pass, and the smoke tests are still running. I will also verify that it fixes the issue manually once the other tests finish.
I have "verified" with some code changes that this fixes the issue. It is very difficult to reproduce the error. I had to modify the code to cause the issue to show up more often. I had to make it so the dist cache always was trying to delete things, and it never actually deleted them, I also had to modify it so that all dist cache entries take a long time to download, and on top of that I had to hit it just right so that a job was finishing at the same time a dist cache entry was being deleted. ugg.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542634/MR-4576.txt
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2784//console
This message is automatically generated.
+1. Thanks Bobby.
Note the findbugs exist without this patch.
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
[exec]
[exec]
Closed upon release of Hadoop 1.2.0.
MAPREDUCE-2494introduced a new lock when releasing a dist cache entry that introduced this problem. Thanks to Koji for finding and debugging this.Essentially the heartbeat thread holds a lock on the TaskTracker object.
So does the job cleanup thread. Which also holds a lock on the TrackerDistributedCacheMenager's big list lock (this is the lock that
MAPREDUCE-2494added in).The thread that deletes things from the dist cache also grabs that big lock, and at the same time grabs locks in turn for every entry in the dist cache.
While an entry in the dist cache is being downloaded it also holds the lock for the dist cache entry.
So this can result in
Downloading thread holds dist cache lock which blocks the dist cache delete thread which holds the full dist cache map lock that blocks the job cleanup thread that holds that Task Tracker lock which blocks the heartbeat thread. This can be seen below.
I think it is probably best to change the DistCache entries' locks so that when we go to delete them if the lock is held we skip that entry instead of having it block.