Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This information can be reported back to jobtracker to help profiling jobs and scheduling tasks.

      1. MAPREDUCE-1201-v3-modified.patch
        17 kB
        Vinod Kumar Vavilapalli
      2. MAPREDUCE-1201-v3.patch
        16 kB
        Scott Chen
      3. MAPREDUCE-1201-v2.patch
        15 kB
        Scott Chen
      4. MAPREDUCE-1201.patch
        14 kB
        Scott Chen

        Issue Links

          Activity

          Hide
          Scott Chen added a comment -

          Add getCumulativeCpuTime in ProcfsBasedProcessTree.
          cumulativeCpuTime =
          total of (utime + stime) for all current subprocesses +
          total of (final utime + final stime) for all finished subprocess.

          This is done by parsing utime and stime in /proc/<pid>/stat.

          Show
          Scott Chen added a comment - Add getCumulativeCpuTime in ProcfsBasedProcessTree. cumulativeCpuTime = total of (utime + stime) for all current subprocesses + total of (final utime + final stime) for all finished subprocess. This is done by parsing utime and stime in /proc/<pid>/stat.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The patch conflicts with changes in MAPREDUCE-1167. Making that as a blocker for this issue.

          Show
          Vinod Kumar Vavilapalli added a comment - The patch conflicts with changes in MAPREDUCE-1167 . Making that as a blocker for this issue.
          Hide
          Scott Chen added a comment -

          Thanks for marking the issue to blocked, Vinod

          I am submitting this again with the update in MAPREDUCE-1167

          This patch does the following things:
          1. Parse utime and stime in /proc/<pid>/stat for all the subprocesses under the assigned PID (see "man proc" for the definition of utime and stime)
          2. Compute the CPU time gained for each subprocess (utime + stime) - (last_utime + last_stime). If this is the first time the subprocess is seen, last_utime = 0 and last_stime = 0.
          3. Sum the CPU time gained and update the cumulative CPU time of the process tree

          This may seem confusing. Note that we cannot simply sum up (utime + stime) for all the subprocesses. This is because some of the subprocesses may finish before the process tree finishes. So if we just sum up the CPU time for the existing subprocesses, we may lose the CPU used by the finished subprocesses. So our approach is computing the CPU gain each update and cumulate them. In this way, even if the subprocess finished, we still get the CPU usage of it.

          Show
          Scott Chen added a comment - Thanks for marking the issue to blocked, Vinod I am submitting this again with the update in MAPREDUCE-1167 This patch does the following things: 1. Parse utime and stime in /proc/<pid>/stat for all the subprocesses under the assigned PID (see "man proc" for the definition of utime and stime) 2. Compute the CPU time gained for each subprocess (utime + stime) - (last_utime + last_stime). If this is the first time the subprocess is seen, last_utime = 0 and last_stime = 0. 3. Sum the CPU time gained and update the cumulative CPU time of the process tree This may seem confusing. Note that we cannot simply sum up (utime + stime) for all the subprocesses. This is because some of the subprocesses may finish before the process tree finishes. So if we just sum up the CPU time for the existing subprocesses, we may lose the CPU used by the finished subprocesses. So our approach is computing the CPU gain each update and cumulate them. In this way, even if the subprocess finished, we still get the CPU usage of it.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12425694/MAPREDUCE-1201-v2.patch
          against trunk revision 882790.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/148/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/148/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/148/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/148/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12425694/MAPREDUCE-1201-v2.patch against trunk revision 882790. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/148/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/148/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/148/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/148/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I'm starting to review this patch.

          Show
          Vinod Kumar Vavilapalli added a comment - I'm starting to review this patch.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The patch looks good overall. Only minor comments will improve maintainability:

          • Rename JIFFY_LENGTH to JIFFY_LENGTH_IN_MILLIs?
          • Tree-dump
            • Change UTIME to USER_MOD_TIME and STIME to SYSTEM_TIME ?
            • Change the tree-dump header to include the units, i.e. "UTIME(MILLIS) STIME(MILLIS)"
          Show
          Vinod Kumar Vavilapalli added a comment - The patch looks good overall. Only minor comments will improve maintainability: Rename JIFFY_LENGTH to JIFFY_LENGTH_IN_MILLIs? Tree-dump Change UTIME to USER_MOD_TIME and STIME to SYSTEM_TIME ? Change the tree-dump header to include the units, i.e. "UTIME(MILLIS) STIME(MILLIS)"
          Hide
          Scott Chen added a comment -

          Thanks. Will make the necessary changes and submit again.

          Show
          Scott Chen added a comment - Thanks. Will make the necessary changes and submit again.
          Hide
          Scott Chen added a comment -

          Have done the suggested changes by Vinod and uploaded it again.

          Show
          Scott Chen added a comment - Have done the suggested changes by Vinod and uploaded it again.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Patch with very minor changes:

          • Changed USER_MOD_TIME to USER_MODE_TIME (sorry, that was my mistake earlier)
          • Wrapped around line(s) longer than 80 chars
          • Removed usage of Long.parseLong() in favor of using long values directly
          Show
          Vinod Kumar Vavilapalli added a comment - Patch with very minor changes: Changed USER_MOD_TIME to USER_MODE_TIME (sorry, that was my mistake earlier) Wrapped around line(s) longer than 80 chars Removed usage of Long.parseLong() in favor of using long values directly
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Submitting the latest patch to Hudson.

          Show
          Vinod Kumar Vavilapalli added a comment - Submitting the latest patch to Hudson.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427292/MAPREDUCE-1201-v3-modified.patch
          against trunk revision 888269.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/172/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/172/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/172/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/172/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427292/MAPREDUCE-1201-v3-modified.patch against trunk revision 888269. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/172/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/172/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/172/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/172/console This message is automatically generated.
          Hide
          Scott Chen added a comment -

          Thanks for the help, Vinod. Now the patch looks better.
          I think the test failures are not related to this patch because the previous submit had no problem.

          Show
          Scott Chen added a comment - Thanks for the help, Vinod. Now the patch looks better. I think the test failures are not related to this patch because the previous submit had no problem.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Hudson's report is suspicious, the two test failed because of NoClassFoundException, no way related to the patch. Looks similar to https://issues.apache.org/jira/browse/MAPREDUCE-952?focusedCommentId=12787361&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12787361.

          Will resbumit again. We should investigate the Hudson's erroneous behaviour though. Guess it used some invalid common jar.

          Show
          Vinod Kumar Vavilapalli added a comment - Hudson's report is suspicious, the two test failed because of NoClassFoundException , no way related to the patch. Looks similar to https://issues.apache.org/jira/browse/MAPREDUCE-952?focusedCommentId=12787361&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12787361 . Will resbumit again. We should investigate the Hudson's erroneous behaviour though. Guess it used some invalid common jar.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427292/MAPREDUCE-1201-v3-modified.patch
          against trunk revision 888431.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/306/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/306/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/306/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/306/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427292/MAPREDUCE-1201-v3-modified.patch against trunk revision 888431. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/306/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/306/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/306/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/306/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427292/MAPREDUCE-1201-v3-modified.patch
          against trunk revision 891111.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/206/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/206/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/206/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/206/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427292/MAPREDUCE-1201-v3-modified.patch against trunk revision 891111. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/206/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/206/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/206/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/206/console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          The failed contrib tests are all related to Streaming and are not related to this patch.

          Show
          dhruba borthakur added a comment - The failed contrib tests are all related to Streaming and are not related to this patch.
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Scott and Vinod!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Scott and Vinod!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )

            People

            • Assignee:
              Scott Chen
              Reporter:
              Scott Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development