Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1850

Include job submit host information (name and ip) in jobconf and jobdetails display

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Enhancement to identify the source (submit host and ip) of a job request.

      1. mapred-1850-5.patch
        7 kB
        Krishna Ramachandran
      2. mapred-1850-4.patch
        6 kB
        Krishna Ramachandran
      3. mapred-1850-3.patch
        6 kB
        Krishna Ramachandran
      4. mapred-1850-2.patch
        7 kB
        Krishna Ramachandran
      5. mapred-1850-1.patch
        7 kB
        Krishna Ramachandran
      6. mapred-1850.patch
        6 kB
        Krishna Ramachandran
      7. mapred-1850.patch
        7 kB
        Krishna Ramachandran

        Activity

        Hide
        Krishna Ramachandran added a comment -

        This is a forward port from a patch for an earlier release

        Fix for deprecated APIs
        For trunk still need to fix Job.java and Configuration

        Show
        Krishna Ramachandran added a comment - This is a forward port from a patch for an earlier release Fix for deprecated APIs For trunk still need to fix Job.java and Configuration
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12447083/mapred-1850.patch
        against trunk revision 954364.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/571/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/571/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/571/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/571/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447083/mapred-1850.patch against trunk revision 954364. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/571/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/571/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/571/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/571/console This message is automatically generated.
        Hide
        Krishna Ramachandran added a comment -

        revised earlier patch. Modified JobSubmitter to update submit Host and IP

        Show
        Krishna Ramachandran added a comment - revised earlier patch. Modified JobSubmitter to update submit Host and IP
        Hide
        Krishna Ramachandran added a comment -

        revised and renamed

        Show
        Krishna Ramachandran added a comment - revised and renamed
        Hide
        Krishna Ramachandran added a comment -

        revised JobSubmitter after a svn sync

        Show
        Krishna Ramachandran added a comment - revised JobSubmitter after a svn sync
        Hide
        Krishna Ramachandran added a comment -

        submitting a revision

        Show
        Krishna Ramachandran added a comment - submitting a revision
        Hide
        Krishna Ramachandran added a comment -

        revised patch

        Show
        Krishna Ramachandran added a comment - revised patch
        Hide
        Arun C Murthy added a comment -

        Krishna - I don't see why you need to change both JobClient and JobSubmitter, shouldn't adding this in JobSubmitter suffice?

        Show
        Arun C Murthy added a comment - Krishna - I don't see why you need to change both JobClient and JobSubmitter, shouldn't adding this in JobSubmitter suffice?
        Hide
        Krishna Ramachandran added a comment -

        I missed that. Sorry - tested with only JobSubmitter.java setting the submit host and submit IP

        Show
        Krishna Ramachandran added a comment - I missed that. Sorry - tested with only JobSubmitter.java setting the submit host and submit IP
        Hide
        Krishna Ramachandran added a comment -

        revised patch with only JobSubmitter setting these

        Show
        Krishna Ramachandran added a comment - revised patch with only JobSubmitter setting these
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12447875/mapred-1850-3.patch
        against trunk revision 957283.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/263/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/263/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/263/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/263/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447875/mapred-1850-3.patch against trunk revision 957283. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/263/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/263/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/263/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/263/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Some comments on the patch:
        1. Can you add the new configuration names to MRJobConfig and use String constants every where?
        2. I don't think links given in javadoc in JobConf methods will work, as there are no such classes. I don't know why javadoc is not complaining. Can you change them to plain text instead of links?
        3. Minor : Can you remove the comment saying "// auto generated stub" from JobInProgress getters?

        Show
        Amareshwari Sriramadasu added a comment - Some comments on the patch: 1. Can you add the new configuration names to MRJobConfig and use String constants every where? 2. I don't think links given in javadoc in JobConf methods will work, as there are no such classes. I don't know why javadoc is not complaining. Can you change them to plain text instead of links? 3. Minor : Can you remove the comment saying "// auto generated stub" from JobInProgress getters?
        Hide
        Krishna Ramachandran added a comment -

        I think I got all suggested changes (previous comment)

        Show
        Krishna Ramachandran added a comment - I think I got all suggested changes (previous comment)
        Hide
        Amareshwari Sriramadasu added a comment -

        JobConf still uses the configuration names, not String constants. Also, Javadoc for JobConf.getJobSubmitHostAddress() still has links.
        The new configuration should be added in MRJobConfig.java, not MRConfig.

        Show
        Amareshwari Sriramadasu added a comment - JobConf still uses the configuration names, not String constants. Also, Javadoc for JobConf.getJobSubmitHostAddress() still has links. The new configuration should be added in MRJobConfig.java, not MRConfig.
        Hide
        Krishna Ramachandran added a comment -

        ah! another inadvertent omission. Sorry. Think I got them all.
        Use of String constant has not been consistent though I have done my part.

        Show
        Krishna Ramachandran added a comment - ah! another inadvertent omission. Sorry. Think I got them all. Use of String constant has not been consistent though I have done my part.
        Hide
        Amareshwari Sriramadasu added a comment -

        Latest patch looks fine

        Show
        Amareshwari Sriramadasu added a comment - Latest patch looks fine
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12448021/mapred-1850-5.patch
        against trunk revision 957897.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448021/mapred-1850-5.patch against trunk revision 957897. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/console This message is automatically generated.
        Hide
        Krishna Ramachandran added a comment -

        can you commit and resolve?

        Show
        Krishna Ramachandran added a comment - can you commit and resolve?
        Hide
        Amareshwari Sriramadasu added a comment -

        Couldn't view the test report. Re-running through hudson.

        Show
        Amareshwari Sriramadasu added a comment - Couldn't view the test report. Re-running through hudson.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12448021/mapred-1850-5.patch
        against trunk revision 958279.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/271/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/271/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/271/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/271/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448021/mapred-1850-5.patch against trunk revision 958279. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/271/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/271/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/271/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/271/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Test failure is because of MAPREDUCE-1834.

        I just committed this. Thanks Krishna !

        Show
        Amareshwari Sriramadasu added a comment - Test failure is because of MAPREDUCE-1834 . I just committed this. Thanks Krishna !
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

          People

          • Assignee:
            Krishna Ramachandran
            Reporter:
            Krishna Ramachandran
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development