Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2351

mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 1-win, 1.3.0
    • Fix Version/s: 0.23.0, 1.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, mapred.job.tracker.history.completed.location is resolved relative to the default filesystem. If not set it defaults to history/done in the local log directory. There is no way to set it to another local filesystem location (with a file:// URI) or an arbitrary Hadoop filesystem.

      1. MAPREDUCE-2351.branch-1-win.patch
        6 kB
        Ivan Mitic
      2. HADOOP-472.branch-1-win.3.patch
        6 kB
        Chelsey Chang
      3. MAPREDUCE-2351.patch
        3 kB
        Tom White

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        4h 18m 1 Tom White 02/Mar/11 05:10
        Patch Available Patch Available Resolved Resolved
        1d 16h 25m 1 Tom White 03/Mar/11 21:36
        Resolved Resolved Closed Closed
        256d 3h 12m 1 Arun C Murthy 15/Nov/11 00:48
        Hide
        Ivan Mitic added a comment -

        Could someone please backport the newly attached patch to branch-1-win?

        Chelsey, I committed the patch to branch-1 only, as we'll be merging all branch-1 changes to branch-1-win in a day or so, and your patch will be picked up.

        Show
        Ivan Mitic added a comment - Could someone please backport the newly attached patch to branch-1-win? Chelsey, I committed the patch to branch-1 only, as we'll be merging all branch-1 changes to branch-1-win in a day or so, and your patch will be picked up.
        Ivan Mitic made changes -
        Fix Version/s 1.3.0 [ 12324153 ]
        Hide
        Ivan Mitic added a comment -

        I committed the backport patch to branch-1. Thank you Chelsey for contribution!

        Show
        Ivan Mitic added a comment - I committed the backport patch to branch-1. Thank you Chelsey for contribution!
        Ivan Mitic made changes -
        Affects Version/s 0.23.0 [ 12315570 ]
        Affects Version/s 1-win [ 12321744 ]
        Affects Version/s 1.3.0 [ 12324153 ]
        Ivan Mitic made changes -
        Attachment MAPREDUCE-2351.branch-1-win.patch [ 12588955 ]
        Hide
        Ivan Mitic added a comment -

        Chelsey, the name of your patch does not seem valid. You should name it based on the Apache Jira id. Attaching the same page with the right name.

        Show
        Ivan Mitic added a comment - Chelsey, the name of your patch does not seem valid. You should name it based on the Apache Jira id. Attaching the same page with the right name.
        Hide
        Ivan Mitic added a comment -

        Thanks Chelsey for doing the backport. I verified that the new test passes on both Windows and Linux. +1 on the patch. Will commit shortly.

        Show
        Ivan Mitic added a comment - Thanks Chelsey for doing the backport. I verified that the new test passes on both Windows and Linux. +1 on the patch. Will commit shortly.
        Chelsey Chang made changes -
        Attachment HADOOP-472.branch-1-win.3.patch [ 12588711 ]
        Hide
        Chelsey Chang added a comment -

        Hi,

        Could someone please backport the newly attached patch to branch-1-win?
        It has pretty much the same changes as MAPREDUCE-2351.patch
        with a small difference.
        The new testcase testDoneFolderOnNotDefaultFileSystem fails on Windows because Windows default permission is not 755. So in addition to the changes in MAPREDUCE-2341.patch, the permission check for history_done_folder's "sub-directories" on Windows is skipped, given that users can still access the information via the Web-UI.

        Show
        Chelsey Chang added a comment - Hi, Could someone please backport the newly attached patch to branch-1-win? It has pretty much the same changes as MAPREDUCE-2351 .patch with a small difference. The new testcase testDoneFolderOnNotDefaultFileSystem fails on Windows because Windows default permission is not 755. So in addition to the changes in MAPREDUCE-2341 .patch, the permission check for history_done_folder's "sub-directories" on Windows is skipped, given that users can still access the information via the Web-UI.
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/ )
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.23.0 [ 12315570 ]
        Resolution Fixed [ 1 ]
        Hide
        Tom White added a comment -

        I've just committed this.

        Show
        Tom White added a comment - I've just committed this.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472392/MAPREDUCE-2351.patch
        against trunk revision 1076411.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/114//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/114//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/114//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472392/MAPREDUCE-2351.patch against trunk revision 1076411. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/114//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/114//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/114//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        +1 lgtm.

        Contingent on not causing a delta in the test results.

        Show
        Eli Collins added a comment - +1 lgtm. Contingent on not causing a delta in the test results.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472392/MAPREDUCE-2351.patch
        against trunk revision 1075670.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:

        -1 contrib tests. The patch failed contrib unit tests.

        -1 system test framework. The patch failed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/111//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/111//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/111//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472392/MAPREDUCE-2351.patch against trunk revision 1075670. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/111//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/111//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/111//console This message is automatically generated.
        Tom White made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tom White made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-2351.patch [ 12472392 ]
        Hide
        Tom White added a comment -

        This patch allows any filesystem to be specified for the completed.location property.

        Show
        Tom White added a comment - This patch allows any filesystem to be specified for the completed.location property.
        Tom White created issue -

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development