This is the design document for review.

Here is the missing image.

Cool docs! One quick question: Does this design allow for multiple simultaneous Stabdby-NameNode?

Yes you should be able to run multiple standby nodes by design. NN will stream the journal to all of them.

Comments organized by section:
General comments:

• More description on how the NN and SNN startup and communicate with each other would be good to have.
• Design will only work for one standby namenode. For example if a standby node starts spooling, followed by another standby node how does flow work?

Streaming:

• Proposal is for SNN to treat each received record as name-node command and to call respective name-node method to update namespace. This is a not a good idea. For example modification time etc. which are based on NN will be different on SNN. SNN should get indication of what changed and apply it to its copy of namespace.
• How is the reliability of edit changes ensured? That is, when NN sends records to SNN how are the following ensured?
• Did SNN application really receive the record?
• Did SNN apply that record to its namespace copy?
• Did SNN store the the record in secondary storage reliably?

Journal Spool:

• Clarify that JS is stored as a file on the harddisk

Admin command saveCheckpoint
Not very clear on the following sentence. Does NN keep running if all the storage directories fail? Why should it then enter safe mode?

• NN should keep running even if the last storage directory fails. It should enter safe mode in this case.

Most answers are in the document. I'll try to clarify.

1. Multiple standby nodes can be connected to the same active name-node.
   Journal spool is an internal SNN mechanism that helps to delay applying journal records to the SNN's memory namespace while SNN starts up or creates a checkpoint. So if you have multiple standbys each of them will support its own journal spool. Don't see a problem here.
   The question is how the name-node handles rollEdits from the second SNN when the checkpointing for the first one is still in progress. There are 2 ways to handle this.
   1. Declare that NN can tolerate only one checkpoint at a time and refuse checkpoint requests from other standbys until the current is complete. This is the simplest approach, and I plan to implement it at least initially.
   2. NN can create multiple edits.new for each subsequent rollEdits request. That is, when SNN1 asks NN to rollEdits NN creates edits.new.1
      When SNN2 asks NN to rollEdits and edits.new.1 still exists NN creates edits.new.2 and so on. This is more tricky but is doable too.
2. SNN treats each received journal record as a name-node command.
   Journal records in hdfs are self-contained, that is they do not depend on the current state of the name-server. Say, if a modification time should be changed then the time to which it should be changed to is specified in the record, it is not taken from SNN.system.currentTime().
3. When SNN receives a record it applies it to the namespace in its memory AND if any storage directories are specified for SNN it will write the record to the edits file(s), just as NN would do, AND THEN will reply to the name-node that operation is complete.
4. Journal spool persistently stores records on a local disk.
5. NN should keep running even if the last storage directory fails. It should enter safe mode in this case.
   When all journal streams fail on the name-node it (NN) has no more way to persist namespace modifications. Currently we treat that by shutting down the name-node. Shutting down NN when the last hard drive fails does not work well since we loose both the image file and the in-memory state. I think we should keep NN running, but prevent it from name-space mutations. The safe-mode prevents it from changes. Keeping it in running condition reserves a chance to save the in-memory image to some other device manually mounted before calling saveCheckpoint.

This patch introduces two new types of name-nodes: a Checkpoint node and a Backup node.

• The role of the Checkpoint node to checkpoint name-node meta-data by merging image and edits files.
• The Backup node extends functionality of the Checkpointer by that it can receive online updates of the file system meta-data, apply them to its memory state and persist them on disks just like the name-node does. Thus at any time the Backup node contains an up-to-date image of the namespace both in memory and on local disk(s).
  This also results in much more efficient checkpointing because backup node does not need to transfer files from the active name-node and does not need to replay (merge) edits.
• Term Standby node is reserved for further extension of the backup node functionality, when cluster will be able to switch over to the new name-node if the active dies.
  This is mentioned in the "Warm standby provision" section of the design document.

Typical use cases:

1. Run Checkpoint node only to create checkpoints. This should be used instead of the current SecondaryNameNode, which is depricated by the patch. I reused a lot of the SecondaryNameNode code so this effort was not wasted, it just evolved.
2. Run Backup node to support online streaming of edits and efficient checkpointing.
   This particularly targets eliminating NFS as a remote storage for edits.
3. Run NameNode without persistent storage at all and delegate all "persisting" functionality to the Backup node. The trick here is to start name-node with -importCheckpoint option and then run the Backup node.

In the near term I plan to

• attach an updated design document with all modifications and clarifications to the initial design.
• provide more test cases in TestBackupNode unit test;
• and perform large scale testing.

Hi Konstantin, this is very interesting. I will wait for your design doc. A few short questions in the meantime:

1. Will this support having multiple Backup nodes for a single namenode?
2. What are the pros-and-cons of this effort vs the Bookkeeper integration (HADOOP-5189)?

Links Edit/Image related issues here: HADOOP-3347, HADOOP-5119

(Dhruba-1) Current implementation supports only one Backup node. It will just not let the other register once one is already registered. This is the implementation restriction only. We can extend it by providing a Collection of registered backup nodes on the active node. Something to work on in the future. You can have multiple checkpointers though as long as there are no any backups.
(Dhruba-2) Bookkeeper integration is a separate issue but related. Separate because BK provides only the journalling functionality. You cannot keep an up-to-date namespace in BK, and even if you do it is not useful for the name-node. Separate because fast journalling is important and that is what BK provides, while backup nodes are still bounded by the local disk speed. Related since both should use the same abstractions for journalling - abstract edit streams under the common journalling framework.

This is the updated document.

I will send comments on the design doc later.

Comments (lot of them are nits...):
Design related:
1. Design uses BackupNode as a subclass of NameNode. In future where Backup can transition to Active or Active can transition to Backup, this relationship seems stringent. Capturing active, backup as the state of Namenode might be a better idea.
2. If EditLogBackupOutputStream fails to write to the backup, how does active and backup recover from this condition. Looks like the current mechanism to recover a failed stream to disk is not workable for stream to backup.

Coding related:
1. General comments:
1.1. For consistently use NameNode and DataNode intead of name-node and data-node
1.2. Many class contructors call default constructor of the super class super(). This is unnecessary.
1.3. Many if statements are not enclosed in curly brackets. Do we follow this java coding convention?

2. UnregisterNodeException.java
2.1. is shown as changed file. It should be new add and UnregisteredDatanodeException.java should be deleted.
2.2. Class comment still only talks about Datanode. It should be made generic to all types of node
2.3. Constructor can be change to take

and

instead of

and

3. NodeRegistration.java
3.1. Class comment seems to indicate this is NameNode specific. Comment should indicate that it is generic.

4. CheckpointCommand.java
4.1. Additional description for the class to indicate that this is sent in response to startCheckpoint NamenodeProtocol command would be good.

5. NamenodeProtocol.java
5.1. In class comments why are the changes made for previous protocol version removed (in this case version 2). Isn't retaining this history a good idea for better documentation?
5.2. getBlocks() comment suggests that it i used by Balancer. Why have this comment if this could be used by things other than Balancer in the future.
5.3. Description for methods versionRequest, errorReport, endCheckpoint,
5.4. startCheckpoint() good to indicate direction of the message (from backup to active)
5.5. Move static variables for Journal pipeline action to the beginning of the class
5.6. journal() better documentation on what it does and the direction of this message.

6. ServerCommand.java
6.1. Should the action static ints be moved to NamenodeProtocol.java? This is similar to Journal actions defined in NamenodeProtocol.java

7. HdfsConstants.java
7.1. It is better to be explicit about printing NamenodeRole as "Backup NameNode" and "Checkpoint NameNode" instead of just "Backup Node" and "Checkpoint Node".

8. BackupNode.java
8.1. In class comments </li> and </ol> are missing.
8.2. No need to defined LOG as it is inherited from NameNode
8.3. Using dfs.backup.address and dfs.backup.http.address means backup is defined at the install time. To change backup to active namenode, configuration change needs to be made. This could be restrictive in the future when a backup node can automatically transtion to become active.
8.4. When starting nodes, we may need to check the address read from configuration with the local node address for http.address etc.
8.5. Comment "NameNodeCommon methods" should change to "Common NameNode methods"
8.6. import org.apa/che.commons.logging.*; is not used
8.7. Replace import java.io.; import java.net. to import specific ones IOException, InetSocketAddress, SocketTimeoutException
8.8. handleShake() use NamenodeProtocol.NOTIFY instead of DatanodeProtocol.NOTIFY in errorReport
8.9. Should BackupNode override Namenode.errorReport(), in order to not do the processing of releasing backup node?
8.10. Should BackupNode throw exception for NamenodeProtocol methods NameNode.journalSize()?
8.11. stop() - is there a need for RPC.stopProxy(namenode). Does super.stop() take care of it?

9. Checkpointer.java
9.1. doCheckPoint() calls backupNode.setCheckpointState(CheckpointStates.ROLLED_EDITS). Should we do this in startJournalSpool(). That way for some reason startCheckpoint() to active fails, the checkpointer state remains consistent. doCheckPoint() should wait for state to reach CheckpointStates.ROLLED_EDITS to make sure the active has rolled edits before proceeding further with checkpoint. This helps in any boundary conditions where checkpoint could miss edits that might still come from active to previous edits log.
9.2. doCheckpoint() should call convergeJournalSpool() for BackupNode of type BACKUP only
9.3. Given that Checkpointer is looking into internals of BackupNode so much, should it be made private inner class of BackupNode?
9.4. Should downloadCheckpoint() and uploadCheckpoint() have registrationID or some other token that is ephemeral perhaps for validating the access. Otherwise these URLs could be misused to mess with the Namenode state. This could be done in another jira.
9.5. run() Use either period or checkpointPeriod in the while loop. Also we can keep 1000 * period calculated outside the while loop.

10. NameNode.java
10.1. For better readability NameNode could support instead of node.getRole() == NamenodeRole.BACKUP, method isRole(NamenodeRole.BACKUP) or isActive(), isBackup(), isCheckpointer().
10.2. Not sure what the comment // trash in NameNode constructor is far...

11. CheckpointSignature.java
11.1. Need to create a separate jira to enhance this to include additional information that could help in catching error conditions such as backup node FSImage number of files not matching the active node etc. Currently any condition where backup does not match the active goes undetected.

12 EditLogBackupInputStream.java
12.1. Should constructor be using parameter name to set the member address?
12.2. getType() should return JournalType.BACKUP
12.3. Class comments need to be updated

13. FSNamesystem.java
13.1. getStorageDirs() FSNamesystem.LOG can just be LOG()

14. FSImage.java
14.1. minor nit - startCheckpoint() comment has name-name.
14.2. startCheckpoint() should some of the if conditions be else if where msg is set
14.3. Adding parentheses to the if condition that combines || with && will make it more readable

I am going to comment on Suresh's items that are not reflected in the new patch.

Design related:

1. Active node does not turn into Backup node in current design. We plan to have a possibility to turn a Standby node into Active, yes but this is done by just changing the role of the node. I think it makes sense to keep BackupNode as a subclass of NameNode because some commands, those that don't modify the meta-data, will work on the backup node exactly as they do on the active node. Say you can do lsr from the backup, but if you try to create a file it will throw SafeModeException.

2. EditLogBackupOutputStream failing to write to the backup node is handled by calling processIOError(), which removes the failed BackupOutputStream from the list of EditLogOutputStream(s) and increments the checkpointTime for the remaining streams to make the failed image outdated.

Coding related:

1.1. name-node and data-node vs NameNode and DataNode is actually consistent if you look at other comments. Lets say NameNode and DataNode refer to classes and name-node and data-node to the abstractions behind them.
1.2. Calling super() in class constructors is unnecessary, but helps me to debug and analyze the code. You can set a breakpoint, step into, and easily find the code of the super constructor.

2.3. UnregisteredNodeException Constructor takes DatanodeID and DatanodeInfo as parameters because this is a constructor specific for data-nodes. We may later need name-node, balancer, etc specific constructors, which will have their special parameters.

5.1. We do not keep comments for previous versions in the code, they can be retrieved from svn history. I copied the respective comment from ClientProtocol.

6.1. I think that ServerCommand actions (static ints) should belong to the commands that use them. I plan to file an issue to move DatanodeCommand actions into the respective commands.

7.1. My reasoning is that we have NameNode, DataNode, and now we will have Backup Node and Checkpoint node. I first thought about naming them "Backup NameNode" and "Checkpoint NameNode" connotative to SecondaryNameNode.
But "Backup node" sounds better to me and besides this is what they are - nodes.

8.3. dfs.backup.address and dfs.backup.http.address define the addresses which the backup node starts on its rpc and http servers. You do not need to change the configuration when the node transitions from backup to active.
Example. If you start DataNode on port 0, which means on any free port, you do not change the config file after startup, right? Same with backup/active nodes.
8.9. Backup node does not have backup streams now, so errorReport() is just harmless. If we decide to implement chaining edits streams then yes we will need to override errorReport() and probably other methods too.
8.10. BackupNode.journalSize() will report the correct size of its edits file. I don't think it should throw an exception.
8.11. BackupNode.stop() should call RPC.stopProxy() because it needs to stop the name-node proxy it is connected to as a client.
And the NameNode.stop() will not take care of that because NameNode does not act as its own client.

9.1. Checkpointer.doCheckPoint() first calls NameNode.startCheckpoint() which in turn calls startJournalSpool() from the NameNode itself and does not return until the BackupNode actually rolls its edits.
9.2. convergeJournalSpool() contains common logic for Backup and Checkpoint nodes. I agree the name does not make much sense for the checkpoint node since it does not converge no journals as it doesn't have any.
But for the backup node its reflects the semantics of the operation.
9.3. Checkpointer should not be an inner class of the BackupNode, imo. We have been through this with FSNamesystem class, had to factor out most of the stuff out of it.
9.4. downloadCheckpoint() and uploadCheckpoint() should evelve in the future. Right now they use CheckpointSignature for validation, but we may need to include registration as well.

11.1. CheckpointSignature currently includes the name-nodes's edits file modification time only for identification purposes. I planned to create an issue to include there the number of files in the namepsace at the time when the checkpoint starts.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12401793/BackupNode.patch
against trunk revision 752292.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 20 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 findbugs. The patch appears to introduce 9 new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/67/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/67/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/67/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/67/console

This message is automatically generated.

I fixed all find bugs except for one. It is complaining about writing to a static field NameNode.myMetrics. This was introduced before. I don't see a reason why findBugs treats it differently now.
In general I don't also see a reason why NameNode.myMetrics should be a static member. I'll file a bug to change this.

Forgot to mention that I fixed 3 old find bugs, so the overall count went down by 2.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402109/BackupNode.patch
against trunk revision 753052.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 20 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/59/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/59/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/59/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/59/console

This message is automatically generated.

1. I think it might be a good idea to abstract the active/backup/checkpointer as states of Namenode. This helps in transition of states back and forth. In current design an active node cannot become backup. This can be done in a separate jira.

Coding related:

6.1. I think that ServerCommand actions (static ints) should belong to the commands that use them. I plan to file an issue to move DatanodeCommand actions into the respective commands.

I am not clear on this. Since ACT_SHUTDOWN is part of protocol itself, it should be in NamenodeProtocol.java?

8.3. dfs.backup.address and dfs.backup.http.address define the addresses which the backup node starts on its rpc and http servers. You do not need to change the configuration when the node transitions from backup to active.

Example. If you start DataNode on port 0, which means on any free port, you do not change the config file after startup, right? Same with backup/active nodes.

My example is more like: Say you configure address X as active and address Y as backup during install time. Now for some reason you want to start active node on Y. At this time the addresses that Y tries to bind to address X. Similarly starting backup on X will result in X trying to bind to address Y. So irrespective of local address, the services start on configured addresses. I am not sure this can be solved until we have node management as a part of HA that can assign roles to the nodes. Nodes always bind to their local address and start or transition to the role they are assigned.

> In general I don't also see a reason why NameNode.myMetrics should be a static member.

Static variables sometimes are misused as global variables. This is one example. Another one is FSNamesystem.fsNamesystemObject (see HADOOP-2413).

BTW, could you not using the deprecated API FSNamesystem.getFSNamesystem() in BackupStorage?

+  private FSNamesystem getFSNamesystem() {
+    // HADOOP-5119 should get rid of this.
+    return FSNamesystem.getFSNamesystem();
+  }

I suggest passing a FSNamesystem object in the constructor BackupStorage(). Then, pass the FSNamesystem object created in BackupNode.loadNamesystem(..), which is the only place creating a BackupStorage.

S> ACT_SHUTDOWN is part of protocol itself, it should be in NamenodeProtocol.java

Makes sense I moved all action codes into NamenodeProtocol.

About your address concerns: you in fact propose a change to how configuration works.
You are saying that configuration should contain addresses rcp.address and http.address which would be used by the server starting on this node no matter what this server is. This is different from how things are in hadoop now. With your approach you will have to create a separate configuration for each server. Say for the name-node you will need to set local rcp.address = X and http.address = Y, while for data-nodes you will have to move this values to other variables namenode.rcp.address = X and namenode.http.address = Y because rcp.address and http.address should reflect the data-nodes' addresses.
We on the contrary were trying so far (as much as possible) to keep one configuration for all servers: set everything in one config file and ssh it to all servers and clients unmodified.
Well this task is not for this issue anyway.

N> Static variables sometimes are misused as global variables.

Exactly!

N> I suggest passing a FSNamesystem object in the constructor BackupStorage().

It's the other way around: Backup node constructs BackupStorage() first then creates FSNamesystem with the BackupStorage as a parameter.
My idea with the comment you cited above was that while fixing HADOOP-5119 we will make a member in FSImage referencing FSNamesystem, and a getter getFSNamesystem(), which returns the reference. Then BackupStorage() will require only one change - removing the method you cited.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402164/BackupNode.patch
against trunk revision 753346.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 20 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/86/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/86/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/86/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/86/console

This message is automatically generated.

+1 for the patch.
Minor nits that need not be fixed:

1. HdfsConstants.java two spaces in "Standby Node"
2. Findbugs warning can be removed by initializing myMetrics in NameNode.createNameNode() method

Synced with trunk and removed space in "Standby Node".

Testing comments.

• I tested this on a relatively small cluster at a rather heavy load.
• The test plan is the same as for the SecondaryNameNode, but
  1. there are two nodes to test BackupNode and ChekpointNode
  2. the registration procedure, which is new for BackupNode is tested by the unit test.

Thanks everybody for comments and help.

I just committed this.

Integrated in Hadoop-trunk #779 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/779/)
. Introduce backup node and checkpoint node. Contributed by Konstantin Shvachko.

• This is an incompatible change
• Need release notes
• Need to updated DatanodeProtocol version since DatanodeCommand is changed.

DatanodeCommand and DatanodeRegistration got modified, but their serialization did not change so it will work with the old versions.

BackupNode may be a good feature but it just cannot pass its own unit test and no one is able to fix it in 6 months. It is still failing in my local machine recently. See HDFS-192.

Editorial pass over all release notes prior to publication of 0.21.

I'm out of the office, and unlikely to read email on this account until Tues 8th June.

– Jonathan

The original of this email was scanned for viruses by the Government Secure Intranet virus scanning service supplied by Cable&Wireless Worldwide in partnership with MessageLabs. (CCTM Certificate Number 2009/09/0052.) On leaving the GSi this email was certified virus free.
Communications via the GSi may be automatically logged, monitored and/or recorded for legal purposes.

Hi,
I am pretty new to Hadoop and HDFS. I was curious what happens to an incomplete file write.
1. Namenode Active and Standby running
2. From a client, i started copying 1 GB file
3. Around 500 MB is copied, while Primary Namenode goes down.

Some how, if i make the standby name node to active and make the client to resumbit the request to the new Active name node, whether file copying will continue from where it is left or it will restart ?

I am seeing some issues with acquiring of Lease in case of client requesting to the new Active name node. Can anyone please help me with this ?