Issue Details (XML | Word | Printable)

Key: HADOOP-4618
Type: Improvement Improvement
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Konstantin Shvachko
Reporter: Konstantin Shvachko
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

Move http server from FSNamesystem into NameNode.

Created: 07/Nov/08 10:20 PM   Updated: 08/Jul/09 04:43 PM
Return to search
Component/s: None
Affects Version/s: 0.1.0
Fix Version/s: 0.20.0

Time Tracking:
Not Specified

File Attachments:
  Size
Text File Licensed for inclusion in ASF works MoveHttpServer.patch 2008-11-10 10:38 PM Konstantin Shvachko 24 kB
Text File Licensed for inclusion in ASF works MoveHttpServer.patch 2008-11-08 02:11 AM Konstantin Shvachko 24 kB
Text File Licensed for inclusion in ASF works MoveHttpServer.patch 2008-11-07 11:52 PM Konstantin Shvachko 24 kB

Hadoop Flags: Reviewed, Incompatible change
Release Note:
Moved HTTP server from FSNameSystem to NameNode. Removed FSNamesystem.getNameNodeInfoPort(). Replaced FSNamesystem.getDFSNameNodeMachine() and FSNamesystem.getDFSNameNodePort() with new method FSNamesystem.getDFSNameNodeAddress(). Removed constructor NameNode(bindAddress, conf).
Resolution Date: 11/Nov/08 02:46 AM


 Description  « Hide
NameNode is responsible now for starting its RPC server. The (web UI) http server is started inside FSNamesystem class.
I think it should be the NameNode's responsibility to deal with all issues intended for exposing information it holds to the outside world. This should include the RCP server as well as the http server.
And FSNamesystem class should be a logical container of internal namespace data-structures and in general should not be accessed directly from the outside.

 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Konstantin Shvachko added a comment - 07/Nov/08 10:30 PM
I came to this conclusion trying to implement a standby node. SNN and NN should be represented by the same class, since one can turn into another and vice versa during their life time. I realized that with current implementation the node startup logic will be spread between two classes (NameNode and FSNamesystem), which is inconvenient, complex, and error prone.

One of the consequences of the current design is that we expose FSNamesystem to jsp-s and servlets through a static public method getFSNamesystem(), which returns this. This seems to be an unnecessary shortcut, because all jsp-s and servlets have a reference to the NameNode instance and can access FSNamesystem data via name-node.
This also creates redundant fields in FSNamesystem like name-node's host, port and infoPort.


Konstantin Shvachko added a comment - 07/Nov/08 11:52 PM
This patch moves the http server to the NameNode class.
Additionally I removed one redundant NN constructor and redundant fields handlerCount, startTime.
I changed dfshealth.jsp and nn_browsedfscontent.jsp to not use getFSNamesystem() method.
I could not get rid of the namenodeAddress field in FSNamesystem because it would require changing JSPHelper constructor, which is used in too many if not all other jsps.

Konstantin Shvachko added a comment - 07/Nov/08 11:56 PM
One more change is that I moved field supportAppends from NameNode to FSNamesystem where all such fields are used to be.

Tsz Wo (Nicholas), SZE added a comment - 08/Nov/08 12:29 AM
The patch does not apply to trunk.

Konstantin Shvachko added a comment - 08/Nov/08 02:11 AM
Patch synchronized with trunk.

Hadoop QA added a comment - 08/Nov/08 07:49 PM
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393549/MoveHttpServer.patch
against trunk revision 712344.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

-1 javadoc. The javadoc tool appears to have generated 1 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3558/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3558/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3558/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3558/console

This message is automatically generated.


Steve Loughran added a comment - 10/Nov/08 04:25 PM
Even though this will be tricky for me to merge in with my lifecycle stuff, I think it is good, hence the : +1

HADOOP-4383 would benefit from it, as it would not need to rummage in the namesystem to get the relevant web port.

FSNamesystem.getNameNodeInfoPort() is public -if anyone is using it and its sibling methods their app will break. And as NameNode.fsnamesystem is public, this is effectively a bit of a public API that is being broken. I know my code will break, and I'm happy with that, but I don't know who else is using it.


Tsz Wo (Nicholas), SZE added a comment - 10/Nov/08 06:22 PM
In FSNamesystem.close(), it is better not to take dir.close() out from finally{}. If pendingReplications.stop() throws a RuntimeException, e.g. NPE, then dir won't be closed and, as a result, the image won't be closed.

Konstantin Shvachko added a comment - 10/Nov/08 10:38 PM
Thanks for the comments.
  • Good point Steve, I will mark this as an incompatible change. The right way would have been to deprecate getNameNodeInfoPort(), but there is no way to implement the method in FSNamesystem as the http server is not there anymore.
  • Nicholas, I agree I should have left the "final" section in place.
  • JavaDoc warning is related to HADOOP-4621.

In this version I restored the final section and deprecated FSNamesystem.getDFSNameNodeAddress() in order to be able to remove it later.


Tsz Wo (Nicholas), SZE added a comment - 10/Nov/08 10:55 PM
+1 new patch looks good.

Hadoop QA added a comment - 11/Nov/08 02:42 AM
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393663/MoveHttpServer.patch
against trunk revision 712729.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3571/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3571/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3571/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3571/console

This message is automatically generated.


Konstantin Shvachko added a comment - 11/Nov/08 02:46 AM
I just committed this.

Steve Loughran added a comment - 12/Nov/08 11:29 AM
OK, My code is up in sync with this; I even have a test for MiniDFSCluster that gets the port number and asserts that it is open

Robert Chansler added a comment - 03/Mar/09 01:41 AM
Edit release note for publication.