Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
-
HideIf the running job is retired, then Job url is redirected to the history page. To construct the history url, JobTracker maintains the mapping of job id to history file names. The entries from mapping is purged for jobs older than mapreduce.jobtracker.jobhistory.maxage configured value.ShowIf the running job is retired, then Job url is redirected to the history page. To construct the history url, JobTracker maintains the mapping of job id to history file names. The entries from mapping is purged for jobs older than mapreduce.jobtracker.jobhistory.maxage configured value.
Description
The tracking url for running jobs and the jobs which are retired is different. This creates problem for clients which caches the job running url because soon it becomes invalid when job is retired.
Attachments
Attachments
- patch-1185-ydist.txt
- 8 kB
- Amareshwari Sriramadasu
- patch-1185-3-ydist.txt
- 8 kB
- Amareshwari Sriramadasu
- patch-1185-2-ydist.txt
- 9 kB
- Amareshwari Sriramadasu
- patch-1185-1-ydist.txt
- 9 kB
- Amareshwari Sriramadasu
- 1185_v7.patch
- 7 kB
- Amareshwari Sriramadasu
- 1185_v6.patch
- 7 kB
- Amareshwari Sriramadasu
- 1185_v5.patch
- 8 kB
- Sharad Agarwal
- 1185_v4.patch
- 6 kB
- Sharad Agarwal
- 1185_v3.patch
- 6 kB
- Sharad Agarwal
- 1185_v2.patch
- 6 kB
- Sharad Agarwal
- 1185_v1.patch
- 6 kB
- Sharad Agarwal
Activity
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12424566/1185_v1.patch
against trunk revision 834284.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/236/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/236/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/236/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/236/console
This message is automatically generated.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12424596/1185_v2.patch
against trunk revision 834284.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/134/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/134/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/134/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/134/console
This message is automatically generated.
In JSPUtil, instead of creating the url from jobid and user, we can get it from status.getTrackingUrl()
In JSPUtil, instead of creating the url from jobid and user, we can get it from status.getTrackingUrl()
Thanks Amareshwari. Attaching the new patch with this fix.
In the test case, does it also make sense to add one more check – do not set conn.setInstanceFollowRedirects(false) and then ensure that conn.connect() is successful. If not the contents, we would at least verify that the redirection works as expected, no?
Added the test case to test redirected url validating response code of 200.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12425183/1185_v4.patch
against trunk revision 881095.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/247/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/247/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/247/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/247/console
This message is automatically generated.
I think the approach of including the job history file name in the URL since the beginning will cause more headaches, since the job history file name includes some things that are unparseable by humans. It may be easier and more human-friendly to translate the job id internally to the history file name, and return the content of job history. This will require a map between job ids and the file name to be kept inside the jobtracker, but that should not be too big, since the entries can be removed when job history is purged periodically. Makes sense ?
In any case, Hadoop 0.21 will have a different human-friendly file naming scheme, when this can go away.
I think the approach of including the job history file name in the URL since the beginning will cause more headaches, since the job history file name includes some things that are unparseable by humans. This will require a map between job ids and the file name to be kept inside the jobtracker,
Since URL is given by the JobClient API, the thinking was that users don't really need to bother about the contents and map based thing will not survive JT restarts. The long term solution being done by MAPREDUCE-323. Saying that it looks like (also based on off line discussion with Arun) that map based solution is acceptable in medium term until MAPREDUCE-323 is done.
but that should not be too big, since the entries can be removed when job history is purged periodically. Makes sense ?
Yes, that should work. The only problem we also need to keep in mind that when history cleaner thread runs, it does a listing of the files in history folder and deletes the ones which are older than 30 days (default). But in the meanwhile operations could have manually purged the files. So those entries won't be purged from the map. To address this timestamp can be maintained along with the history file name.
Patch for review. This maintains a map of {JobID, {historyFileName, timestamp}} in JobHistory class. The entries are purged in HistoryCleaner thread.
Earlier patch had a findbugs warning. Attaching patch after suppressing it.
test-patch and ant test passed on Y!20 patch, except TestHdfsProxy.
The test for redirection was missing in earlier Y!20 patch. Added the test in attached patch.
Trunk patch.
Since you are already doing a remove of entries from the jobHistoryFileMap while iterating through it to handle the manual move scenario, I think the jobHistoryFileMap.remove() while deleting the history file is redundant and can be removed.
Y!20 Patch incorporating Jothi's comment. Also fixed a bug in jobdetails.jsp
Testing completed with new patch. All testcases pased:
+1 for QA.
Scenarios tested:
Change the mapred.job.tracker.history.completed.location to point to a local file system directory
1) Run some jobs like sleep, randomwriter, streaming job and see if their job URI gets redirected to the done folder
after the jobs are retired.
2) Run multiple jobs and see if still the jobs are getting redirected properly.
3) Do a JT restart and then type the jobid URL (not the done folder URL) of the job before the JT restart and see if
it gives proper error.
4) Change the mapred.job.tracker.history.completed.location to point to a dfs directory and then run steps 1,2,3 and
see if it works properly.
Uploaded the patch for trunk with Jothi's comment incorporated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12426531/1185_v6.patch
against trunk revision 885530.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/157/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/157/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/157/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/157/console
This message is automatically generated.
A couple of minor nits on v6:
- The comment before removing the MovedFileInfo has to be fixed
- The method getJobId is no longer required
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12426631/1185_v7.patch
against trunk revision 886059.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/285/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/285/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/285/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/285/console
This message is automatically generated.
Integrated in Hadoop-Mapreduce-trunk-Commit #145 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/145/)
. Redirect running job url to history url if job is already retired. Contributed by Amareshwari Sriramadasu and Sharad Agarwal.
Simple fix would be: JT to redirect to history url in case job is already retired.