Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Removes JNI calls to get jvm current/max heap usage in ClusterStatus. Any instances of ClusterStatus serialized in a prior version will not be correctly deserialized using the updated class.

      Description

      The ClusterStatus constructor makes two JNI calls to the Runtime to fetch memory information. ClusterStatus instances are often created inside the JobTracker to obtain other, unrelated metrics (sometimes from schedulers' inner loops). Given that this information is related to the JobTracker process and not the cluster, the metrics are also available via JvmMetrics, and the jsps can gather this information for themselves: these fields can be removed from ClusterStatus

      1. mr-1683-trunk-v1.patch
        3 kB
        Luke Lu
      2. MAPREDUCE-1683_yhadoop_20_9.patch
        7 kB
        Arun C Murthy
      3. M1683-1.patch
        4 kB
        Chris Douglas
      4. ASF.LICENSE.NOT.GRANTED--MAPREDUCE-1683_yhadoop_20_S.patch
        7 kB
        Arun C Murthy
      5. ASF.LICENSE.NOT.GRANTED--MAPREDUCE-1683_part2_yhadoop_20_10.patch
        0.7 kB
        Arun C Murthy

        Issue Links

          Activity

          Hide
          Arun C Murthy added a comment -

          Rather than remove these fields from ClusterStatus, I've made a slightly less invasive change: I've made the memory-related info a part of 'detailed' information alongwith tracker-names and such. Thus users can call JobClient.getClusterStatus(true) to get this info.

          Nevertheless, the issue is important enough to make an incompatible change for better performance on the JT. This is called from various critical pieces of framework code (e.g. JobInProgress) to matter.

          Show
          Arun C Murthy added a comment - Rather than remove these fields from ClusterStatus, I've made a slightly less invasive change: I've made the memory-related info a part of 'detailed' information alongwith tracker-names and such. Thus users can call JobClient.getClusterStatus(true) to get this info. Nevertheless, the issue is important enough to make an incompatible change for better performance on the JT. This is called from various critical pieces of framework code (e.g. JobInProgress) to matter.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for leaving the memory vals in the detailed info.

          Patch for 20 looks good too.

          When working on the trunk patch, we should cleanup/remove all the useless constructors - constructors are all package private so no harm in doing a cleanup here, I think.

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for leaving the memory vals in the detailed info. Patch for 20 looks good too. When working on the trunk patch, we should cleanup/remove all the useless constructors - constructors are all package private so no harm in doing a cleanup here, I think.
          Hide
          Arun C Murthy added a comment -

          Patch for the yahoo security branch, not to be committed.

          Show
          Arun C Murthy added a comment - Patch for the yahoo security branch, not to be committed.
          Hide
          Arun C Murthy added a comment -

          We missed a part of the patch to fix the jsps.

          Show
          Arun C Murthy added a comment - We missed a part of the patch to fix the jsps.
          Hide
          Luke Lu added a comment -

          Looks like JobClient.getClusterStatus(boolean detailed) won't behave as expected, as there is no way to get jobtracker ClusterStatus with jobtracker memory info from the new ClientProtocol due to MAPREDUCE-777.

          Show
          Luke Lu added a comment - Looks like JobClient.getClusterStatus(boolean detailed) won't behave as expected, as there is no way to get jobtracker ClusterStatus with jobtracker memory info from the new ClientProtocol due to MAPREDUCE-777 .
          Hide
          Amareshwari Sriramadasu added a comment -

          Looks like JobClient.getClusterStatus(boolean detailed) won't behave as expected, as there is no way to get jobtracker ClusterStatus with jobtracker memory info from the new ClientProtocol due to MAPREDUCE-777.

          I raised MAPREDUCE-1779 for this.

          Show
          Amareshwari Sriramadasu added a comment - Looks like JobClient.getClusterStatus(boolean detailed) won't behave as expected, as there is no way to get jobtracker ClusterStatus with jobtracker memory info from the new ClientProtocol due to MAPREDUCE-777 . I raised MAPREDUCE-1779 for this.
          Hide
          Luke Lu added a comment -

          The JobTracker memory usage fields in ClusterStatus are not really used by any one other than the JT jsp page. Let's just remove them all together.

          Show
          Luke Lu added a comment - The JobTracker memory usage fields in ClusterStatus are not really used by any one other than the JT jsp page. Let's just remove them all together.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12444160/mr-1683-trunk-v1.patch
          against trunk revision 942764.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/179/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/179/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444160/mr-1683-trunk-v1.patch against trunk revision 942764. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/179/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/179/console This message is automatically generated.
          Hide
          Luke Lu added a comment -

          No tests are need for this patch, as it's strictly removing a feature and existing tests should suffice.

          Show
          Luke Lu added a comment - No tests are need for this patch, as it's strictly removing a feature and existing tests should suffice.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch functionally looks fine. But lets wait for the resolution of MAPREDUCE-1779 before this.

          Show
          Amareshwari Sriramadasu added a comment - Patch functionally looks fine. But lets wait for the resolution of MAPREDUCE-1779 before this.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch looks fine to me.

          Show
          Amareshwari Sriramadasu added a comment - Patch looks fine to me.
          Hide
          Amar Kamat added a comment -

          One suggestion : Instead of printing the total memory, which will remain fixed/clamped, shouldn't we print the currently used heap memory i.e Runtime.totalMemory() - Runtime.freeMemory()?

          Show
          Amar Kamat added a comment - One suggestion : Instead of printing the total memory, which will remain fixed/clamped, shouldn't we print the currently used heap memory i.e Runtime.totalMemory() - Runtime.freeMemory()?
          Hide
          Chris Douglas added a comment -

          That's a good idea. Updated patch

          Show
          Chris Douglas added a comment - That's a good idea. Updated patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445141/M1683-1.patch
          against trunk revision 946833.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/542/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/542/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/542/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/542/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445141/M1683-1.patch against trunk revision 946833. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/542/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/542/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/542/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/542/console This message is automatically generated.
          Hide
          Chris Douglas added a comment -

          Both TestChild and TestPipes pass on my machine; neither should be affected by this patch.

          +1

          I committed this. Thanks, Luke and Arun!

          Show
          Chris Douglas added a comment - Both TestChild and TestPipes pass on my machine; neither should be affected by this patch. +1 I committed this. Thanks, Luke and Arun!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #327 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/327/)
          MAPREDUCE-1683. Remove JNI calls from ClusterStatus cstr. Contributed by Arun Murthy and Luke Lu

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #327 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/327/ ) MAPREDUCE-1683 . Remove JNI calls from ClusterStatus cstr. Contributed by Arun Murthy and Luke Lu

            People

            • Assignee:
              Luke Lu
              Reporter:
              Chris Douglas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development