Issue Details (XML | Word | Printable)

Key: HDFS-453
Type: New Feature New Feature
Status: Patch Available Patch Available
Priority: Major Major
Assignee: Aaron Kimball
Reporter: Aaron Kimball
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
Hadoop HDFS

XML-based metrics as JSP servlet for NameNode

Created: 30/Jun/09 12:02 AM   Updated: 20/Oct/09 06:32 AM
Return to search
Component/s: None
Affects Version/s: None
Fix Version/s: None

Time Tracking:
Not Specified

File Attachments:
  Size
File Licensed for inclusion in ASF works dfshealth.xml.jspx 2009-08-13 03:53 PM Steve Loughran 3 kB
XML File example-dfshealth.xml 2009-08-21 12:42 AM Aaron Kimball 1 kB
Text File Licensed for inclusion in ASF works HDFS-453.2.patch 2009-07-03 11:59 PM Aaron Kimball 12 kB
Text File Licensed for inclusion in ASF works HDFS-453.3.patch 2009-08-21 12:43 AM Aaron Kimball 18 kB
Text File Licensed for inclusion in ASF works HDFS-453.4.patch 2009-08-21 09:53 PM Aaron Kimball 18 kB
Text File Licensed for inclusion in ASF works HDFS-453.5.patch 2009-08-21 09:54 PM Aaron Kimball 18 kB
Text File Licensed for inclusion in ASF works HDFS-453.6.patch 2009-09-18 10:27 PM Aaron Kimball 18 kB
Text File Licensed for inclusion in ASF works HDFS-453.patch 2009-06-30 12:02 AM Aaron Kimball 7 kB
Issue Links:
Reference


 Description  « Hide
In HADOOP-4559, a general REST API for reporting metrics was proposed but work seems to have stalled. In the interim, we have a simple XML translation of the existing NameNode status page which provides the same metrics as the human-readable page. This is a relatively lightweight addition to provide some machine-understandable metrics reporting.

 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Steve Loughran added a comment - 02/Jul/09 04:38 PM - edited
The NN JSP helper is chucking out HTML from some of its methods, not well-formed XML. There's a risk that the content pushed out here is not valid XML.
  1. the tests that have been left out need to take the XML and try and parse it. This isn't that hard, you could do it in Ant using <get> then <xmlvalidate>, ideally against a schema that comes with the patch.
  2. All content that goes out as XML has to be passed through XMLenc. I actually think this is a good opportunity to knock up a successor to xmlenc, one that is namespace aware, and so can be used to push XHTML in the right namespace.

Steve Loughran added a comment - 02/Jul/09 04:45 PM
actually, I'm not so sure about the schema, I just remembered how badly xsd sucks

also,

  1. use .jspx rather than .jsp for XML JSP pages.
  2. consider moving the functions into their own .java code for better integration with the rest of the source (and the IDE, etc)

Aaron Kimball added a comment - 04/Jul/09 12:00 AM
Attaching new patch. I performed suggestion (2) above. Renaming the page to a .jspx doesn't seem to work. Jasper considers .jspx files via a completely separate parser than .jsp ones; one that seems to have trouble with the presence of <% .. %> and other JSP-isms. This patch also includes a test that starts a MiniDFSCluster, retrieves /dfshealth.xml.jsp, uses a SAX parser to parse the page, and check that the main <cluster>..</cluster> element is present.

Hadoop QA added a comment - 04/Jul/09 03:19 AM
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12412528/HDFS-453.2.patch
against trunk revision 790733.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/3/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/3/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/3/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/3/console

This message is automatically generated.


Aaron Kimball added a comment - 04/Jul/09 08:00 AM
The test failures are unrelated. Confusingly, it looks from that test report that only the hdfsproxy component is tested – not HDFS itself?

Steve Loughran added a comment - 13/Aug/09 03:52 PM
  • I like the tests here; nice trick of handing off to the XML parser the task of retrieving the XML; no need for any other work on your part.
  • I'm not sure that dfshealth.xml.jsp is valid XML though, as there is a </br> at the end of <inode_limit>, which worries me
  • regarding JSP vs JSPX, yes, the .jspx files are parsed differently - they have to be well-formed XML, as defined by the JSP 2.0 specification: http://java.sun.com/products/jsp/reference/api/index.html

JSPX pages stop you accidentallly getting too much stuff in front of the <? xml ?> page, and encourage you to create well-formed XML as for all the bits that aren't scriptlet-generated text, your IDE will tell you off in advance, as will the JSP compiler.

I'm attaching the .jspx equivalent, which

  1. moves everything to the jspx syntax
  2. adds a todo item where the page doesn't handle the risk that the application isn't bonded to a namenode. Is this a risk?
  3. turns off cache control

I've already mentioned in HDFS-91 that JSP content doesn't expire, which will burn anyone monitoring through a proxy server. Ideally we should have a JSP tag library that does this for every page, so there's no need to do it by hand in every JSP page.


Steve Loughran added a comment - 13/Aug/09 03:53 PM
.jspx equivalent. Not tested, the IDE isn't too unhappy with it though. It is well-formed XML.

Aaron Kimball added a comment - 21/Aug/09 12:42 AM
attaching example output from /dfshealth.jspx

Aaron Kimball added a comment - 21/Aug/09 12:43 AM
Attaching updated patch
  • Moves page to dfshealth.jspx; validates.
  • Fixes malformed XHTML in NameNodeJspHelper
  • Fixes bug in build.xml that caused new webapp compilation to fail
  • Updates test to hit the new URL.

Hadoop QA added a comment - 21/Aug/09 07:00 AM
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12417201/HDFS-453.3.patch
against trunk revision 805652.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings).

-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/76/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/76/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/76/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/76/console

This message is automatically generated.


Aaron Kimball added a comment - 21/Aug/09 09:53 PM
The two new findbugs warnings are introduced by code generated by the JspC compiler.

I accidentally introduced two new uses of deprecated methods; these have been fixed. The third javac warning is the lack of serialUid in an auto-generated class. So this will continue to get a -1 javac warning.

JspC introduces a further warning now that we've added a jspx file:

[jsp-compile] 09/08/21 14:30:00 WARN compiler.TldLocationsCache: Internal Error: File /WEB-INF/web.xml not found

Given that JspC generates the correct web.xml file, is this a problem?

The core unit test failure is unrelated.


Aaron Kimball added a comment - 21/Aug/09 09:54 PM
patch #4 was generated incorrectly and won't apply. Here we go.

Hadoop QA added a comment - 22/Aug/09 07:43 AM
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12417320/HDFS-453.5.patch
against trunk revision 806746.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 96 javac compiler warnings (more than the trunk's current 95 warnings).

-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/79/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/79/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/79/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/79/console

This message is automatically generated.


Aaron Kimball added a comment - 18/Sep/09 10:27 PM
New patch resync'd with trunk

Hadoop QA added a comment - 19/Sep/09 12:51 AM
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420092/HDFS-453.6.patch
against trunk revision 816785.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 111 javac compiler warnings (more than the trunk's current 110 warnings).

-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/36/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/36/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/36/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/36/console

This message is automatically generated.


Aaron Kimball added a comment - 19/Sep/09 01:06 AM
unrelated test failure.

Aaron Kimball added a comment - 20/Oct/09 06:32 AM
Is there a reason this is still waiting to commit? MAPREDUCE-679 went in weeks ago.