Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1898

[Herriot] Implement a functionality for getting the job summary information of a job.

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.1
    • Fix Version/s: None
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      herriot

      Description

      Implement a method for getting the job summary details of a job. The job summary should be.
      jobId, startTime, launchTime, finishTime, numMaps, numSlotsPerMap, numReduces, numSlotsPerReduce, user, queue, status, mapSlotSeconds, reduceSlotSeconds, clusterMapCapacity,clusterReduceCapacity.

      1. MAPREDUCE-1898.patch
        15 kB
        Vinay Kumar Thota
      2. MAPREDUCE-1898.patch
        15 kB
        Vinay Kumar Thota
      3. 1898-ydist-security.patch
        15 kB
        Vinay Kumar Thota
      4. MAPREDUCE-1898.patch
        14 kB
        Vinay Kumar Thota
      5. 1898-ydist-security.patch
        15 kB
        Vinay Kumar Thota
      6. 1898-ydist-security.patch
        15 kB
        Vinay Kumar Thota
      7. 1898-ydist-security.patch
        16 kB
        Vinay Kumar Thota

        Issue Links

          Activity

          Hide
          Vinay Kumar Thota added a comment -

          Initial patch for yahoo dist security branch.

          Show
          Vinay Kumar Thota added a comment - Initial patch for yahoo dist security branch.
          Hide
          Iyappan Srinivasan added a comment -

          1)numSoltsPerMap - typo. Correct it to numSlotsPerMap

          2) numSoltsPerReduce - typo. Correct it to numSlotsPerMap

          3) JobTracker.getJobSummaryFromLogs - In @param section, add the second parameter.

          Show
          Iyappan Srinivasan added a comment - 1)numSoltsPerMap - typo. Correct it to numSlotsPerMap 2) numSoltsPerReduce - typo. Correct it to numSlotsPerMap 3) JobTracker.getJobSummaryFromLogs - In @param section, add the second parameter.
          Hide
          Vinay Kumar Thota added a comment -

          Addressed Iyappan's comments.

          Show
          Vinay Kumar Thota added a comment - Addressed Iyappan's comments.
          Hide
          Balaji Rajagopalan added a comment -

          Serializing command seperated string does not look good, very error prone. Please check if you can serialize hashmap, last time I talked to someone I was told to write my own writable object.

          Show
          Balaji Rajagopalan added a comment - Serializing command seperated string does not look good, very error prone. Please check if you can serialize hashmap, last time I talked to someone I was told to write my own writable object.
          Hide
          Konstantin Boudnik added a comment -

          Actually, I don't like the fact that we use aspects to modify framework classes. JobInfoImpl is Herriot class and can be modified at the source code level if needed. Using aspects for this is overkill.

          Show
          Konstantin Boudnik added a comment - Actually, I don't like the fact that we use aspects to modify framework classes. JobInfoImpl is Herriot class and can be modified at the source code level if needed. Using aspects for this is overkill.
          Hide
          Vinay Kumar Thota added a comment -

          Cos, you are absolutely correct and it was my mistake.Some how I was in under impression that JobInfoImpl class belongs to mapreduce, so that I have implemented the functionality by using aspectJ. BTW, I will modify the code and re-submit the new patch.

          Show
          Vinay Kumar Thota added a comment - Cos, you are absolutely correct and it was my mistake.Some how I was in under impression that JobInfoImpl class belongs to mapreduce, so that I have implemented the functionality by using aspectJ. BTW, I will modify the code and re-submit the new patch.
          Hide
          Vinay Kumar Thota added a comment -

          Addressed the comments and uploaded the new patch.

          Show
          Vinay Kumar Thota added a comment - Addressed the comments and uploaded the new patch.
          Hide
          Vinay Kumar Thota added a comment -

          Patch for trunk.

          Show
          Vinay Kumar Thota added a comment - Patch for trunk.
          Hide
          Konstantin Boudnik added a comment -

          Patch looks good. The only nit: would it be better to replace string concatenation in JobTracker.getJobSummaryInfo with StringBuffer.append() operations ?

          Show
          Konstantin Boudnik added a comment - Patch looks good. The only nit: would it be better to replace string concatenation in JobTracker.getJobSummaryInfo with StringBuffer.append() operations ?
          Hide
          Vinay Kumar Thota added a comment -

          Addressed the comments.

          Show
          Vinay Kumar Thota added a comment - Addressed the comments.
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good. Let's re-verify before commit. Also, since current test-patch doesn't verify Herriot tests it'd be awesome to have a comment from the author on how it works.

          Show
          Konstantin Boudnik added a comment - +1 patch looks good. Let's re-verify before commit. Also, since current test-patch doesn't verify Herriot tests it'd be awesome to have a comment from the author on how it works.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12450168/MAPREDUCE-1898.patch
          against trunk revision 1074251.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 21 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/77//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/77//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/77//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450168/MAPREDUCE-1898.patch against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/77//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/77//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/77//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.

          Given this is not an issue with MRv2 should we still commit this? I'm happy to, but not sure it's useful. Thanks.

          Show
          Arun C Murthy added a comment - Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks. Given this is not an issue with MRv2 should we still commit this? I'm happy to, but not sure it's useful. Thanks.

            People

            • Assignee:
              Vinay Kumar Thota
              Reporter:
              Vinay Kumar Thota
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development