Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-850

Display more memory details on the web ui

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changes the format of the message with Heap usage on the NameNode web page.

      Description

      With the HDFS-94 being commited, the namenode will use JMX memory beans to get information about heap usage.
      They provide us with additional information such as NonHeap memory usage and Heap Commited and Initialized memory in addition to Used and Max.
      It will be useful to see that additional information on the NameNode web ui.

      1. HDFS-850.patch
        2 kB
        Dmytro Molkov
      2. screenshot-2.jpg
        65 kB
        Dmytro Molkov
      3. HDFS-850.patch
        2 kB
        Dmytro Molkov
      4. screenshot-1.jpg
        60 kB
        Dmytro Molkov
      5. HDFS-850.patch
        2 kB
        Dmytro Molkov

        Issue Links

          Activity

          Dmytro Molkov created issue -
          Dmytro Molkov made changes -
          Field Original Value New Value
          Link This issue is blocked by HDFS-94 [ HDFS-94 ]
          dhruba borthakur made changes -
          Assignee Dmytro Molkov [ dms ]
          Dmytro Molkov made changes -
          Attachment HDFS-850.patch [ 12429972 ]
          Hide
          Dmytro Molkov added a comment -

          What it looks like.

          Show
          Dmytro Molkov added a comment - What it looks like.
          Dmytro Molkov made changes -
          Attachment screenshot-1.jpg [ 12429973 ]
          Hide
          Suresh Srinivas added a comment -

          This information will be useful.

          1. Can you print the memory usage in separate lines from files and block counts.
          2. Current format is: Heap Size <used> / <max>. Committed Heap is <committed>. Using the following format would better explain the information: Heap memory used <used> is <%> of committed memory <commited>. Maximum heap memory is <max>. This also calculates the % usage from the committed memory (which is what is currently available) instead of maximum size the process can grow to.
          Show
          Suresh Srinivas added a comment - This information will be useful. Can you print the memory usage in separate lines from files and block counts. Current format is: Heap Size <used> / <max>. Committed Heap is <committed>. Using the following format would better explain the information: Heap memory used <used> is <%> of committed memory <commited>. Maximum heap memory is <max>. This also calculates the % usage from the committed memory (which is what is currently available) instead of maximum size the process can grow to.
          Hide
          Dmytro Molkov added a comment -

          Updated patch addressing comments by Suresh

          Show
          Dmytro Molkov added a comment - Updated patch addressing comments by Suresh
          Dmytro Molkov made changes -
          Attachment HDFS-850.patch [ 12429989 ]
          Hide
          Dmytro Molkov added a comment -

          Updated view

          Show
          Dmytro Molkov added a comment - Updated view
          Dmytro Molkov made changes -
          Attachment screenshot-2.jpg [ 12429990 ]
          Hide
          Allen Wittenauer added a comment -

          This same code path should be added to the JobTracker. Is there an equivalent JIRA for it?

          Show
          Allen Wittenauer added a comment - This same code path should be added to the JobTracker. Is there an equivalent JIRA for it?
          Dmytro Molkov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Changes the format of the message with Heap usage on the NameNode web page.
          Affects Version/s 0.22.0 [ 12314241 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429990/screenshot-2.jpg
          against trunk revision 898573.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/185/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429990/screenshot-2.jpg against trunk revision 898573. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/185/console This message is automatically generated.
          Hide
          Dmytro Molkov added a comment -

          Resubmitting the patch, because the system tried to apply the screenshot

          Show
          Dmytro Molkov added a comment - Resubmitting the patch, because the system tried to apply the screenshot
          Dmytro Molkov made changes -
          Attachment HDFS-850.patch [ 12430071 ]
          Dmytro Molkov made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Dmytro Molkov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12430071/HDFS-850.patch
          against trunk revision 898573.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/186/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/186/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/186/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12430071/HDFS-850.patch against trunk revision 898573. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/186/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/186/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/186/console This message is automatically generated.
          Hide
          Dmytro Molkov added a comment -

          This is a UI change so no test included with it

          Show
          Dmytro Molkov added a comment - This is a UI change so no test included with it
          Hide
          Suresh Srinivas added a comment -

          This patch does not apply to the trunk. Looks like there are conflicting changes. Could you please upload a new patch.

          Show
          Suresh Srinivas added a comment - This patch does not apply to the trunk. Looks like there are conflicting changes. Could you please upload a new patch.
          Hide
          Hemanth Yamijala added a comment -

          This same code path should be added to the JobTracker. Is there an equivalent JIRA for it?

          I filed MAPREDUCE-1396 for this.

          Show
          Hemanth Yamijala added a comment - This same code path should be added to the JobTracker. Is there an equivalent JIRA for it? I filed MAPREDUCE-1396 for this.
          Hide
          dhruba borthakur added a comment -

          +1 Code looks good. I will commit it.

          Show
          dhruba borthakur added a comment - +1 Code looks good. I will commit it.
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Dmytro!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Dmytro!
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314241 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #213 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/213/)
          . The WebUI displays more details about namenode memory usage.
          (Dmytro Molkov via dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #213 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/213/ ) . The WebUI displays more details about namenode memory usage. (Dmytro Molkov via dhruba)
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/ )
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314046 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          35m 49s 1 Dmytro Molkov 13/Jan/10 00:18
          Open Open Patch Available Patch Available
          20d 20h 5m 2 Dmytro Molkov 13/Jan/10 00:18
          Patch Available Patch Available Resolved Resolved
          57d 5h 48m 1 dhruba borthakur 11/Mar/10 06:06
          Resolved Resolved Closed Closed
          166d 14h 44m 1 Tom White 24/Aug/10 21:50

            People

            • Assignee:
              Dmytro Molkov
              Reporter:
              Dmytro Molkov
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development