Need a mechanism for data nodes to update generation stamps.

• Key: HADOOP-3283
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Tsz Wo (Nicholas), SZE
• Reporter: Tsz Wo (Nicholas), SZE
• Fix Version/s: 0.18.0

Description

For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.

Comments

[Tsz Wo (Nicholas), SZE]

One way is to an inter-DataNode protocol for block manipulation:

interface BlockProtocol extends VersionedProtocol {
  /**
   * Update the GenerationStamp of a block.
   * @return true iff update was required and done successfully. 
   */
  public boolean updateGenerationStamp(Block block,
      GenerationStamp generationstamp) throws IOException;

  /**
   * @return the BlockMetaDataInfo of the block. 
   */
  public BlockMetaDataInfo getBlockMetaDataInfo(Block block) throws IOException;
}

[dhruba borthakur]

+1.

[dhruba borthakur]

Maybe we can call it InterDatanodeProtocol to make it more explicit.

[Tsz Wo (Nicholas), SZE]

For adding a Datanode RPC server, we need to store the RPC server port and distribute it. One way is to store the port in DatanodeID. I am not sure whether it is the right way to go.

[Tsz Wo (Nicholas), SZE]

3283_20080423c.patch: preview, it does not pass the tests yet.

[dhruba borthakur]

1. The two new methods in the InterDatanodeProtocol should take a Block object as the first parameter.
2. The MiniDFSCluster should have a method named getDatanode.getDataNode() that returns an object of type InterDatanodeProtocol. This will allow tests to easily invoke RPCs in the InterDatanodeProtocol.

[Tsz Wo (Nicholas), SZE]

3283_20080424.patch: fixed the bugs and update the patch with Dhruba's comments.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380882/3283_20080424.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 15 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs -1. The patch appears to introduce 1 new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/console

This message is automatically generated.

[Tsz Wo (Nicholas), SZE]

The new Findbugs warning says that class doesn't override equals in the superclass. In our case, we have BlockMetaDataInfo extending Block and we don't need to define equals in BlockMetaDataInfo.

[Tsz Wo (Nicholas), SZE]

3283_20080425versionID.patch: update the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol

[dhruba borthakur]

I just committed this. Thanks Nicholas!

[Hudson]

Integrated in Hadoop-trunk #471 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/471/)

[Hudson]

Integrated in Hadoop-trunk #483 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/483/)