Hadoop Common
  1. Hadoop Common
  2. HADOOP-3283

Need a mechanism for data nodes to update generation stamps.

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Added an IPC server in DataNode and a new IPC protocol InterDatanodeProtocol. Added conf properties dfs.datanode.ipc.address and dfs.datanode.handler.count with defaults "0.0.0.0:50020" and 3, respectively.
      Changed the serialization in DatanodeRegistration and DatanodeInfo, and therefore, updated the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol.
      Show
      Added an IPC server in DataNode and a new IPC protocol InterDatanodeProtocol. Added conf properties dfs.datanode.ipc.address and dfs.datanode.handler.count with defaults "0.0.0.0:50020" and 3, respectively. Changed the serialization in DatanodeRegistration and DatanodeInfo, and therefore, updated the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol.

      Description

      For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.

      1. 3283_20080423c.patch
        23 kB
        Tsz Wo Nicholas Sze
      2. 3283_20080424.patch
        28 kB
        Tsz Wo Nicholas Sze
      3. 3283_20080425versionID.patch
        2 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #483 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/483/ )
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #471 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/471/ )
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Nicholas!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Nicholas!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          3283_20080425versionID.patch: update the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol

          Show
          Tsz Wo Nicholas Sze added a comment - 3283_20080425versionID.patch: update the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The new Findbugs warning says that class doesn't override equals in the superclass. In our case, we have BlockMetaDataInfo extending Block and we don't need to define equals in BlockMetaDataInfo.

          Show
          Tsz Wo Nicholas Sze added a comment - The new Findbugs warning says that class doesn't override equals in the superclass. In our case, we have BlockMetaDataInfo extending Block and we don't need to define equals in BlockMetaDataInfo.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12380882/3283_20080424.patch
          against trunk revision 645773.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 15 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to introduce 1 new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12380882/3283_20080424.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 15 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 1 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          3283_20080424.patch: fixed the bugs and update the patch with Dhruba's comments.

          Show
          Tsz Wo Nicholas Sze added a comment - 3283_20080424.patch: fixed the bugs and update the patch with Dhruba's comments.
          Hide
          dhruba borthakur added a comment -

          1. The two new methods in the InterDatanodeProtocol should take a Block object as the first parameter.
          2. The MiniDFSCluster should have a method named getDatanode.getDataNode() that returns an object of type InterDatanodeProtocol. This will allow tests to easily invoke RPCs in the InterDatanodeProtocol.

          Show
          dhruba borthakur added a comment - 1. The two new methods in the InterDatanodeProtocol should take a Block object as the first parameter. 2. The MiniDFSCluster should have a method named getDatanode.getDataNode() that returns an object of type InterDatanodeProtocol. This will allow tests to easily invoke RPCs in the InterDatanodeProtocol.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          3283_20080423c.patch: preview, it does not pass the tests yet.

          Show
          Tsz Wo Nicholas Sze added a comment - 3283_20080423c.patch: preview, it does not pass the tests yet.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          For adding a Datanode RPC server, we need to store the RPC server port and distribute it. One way is to store the port in DatanodeID. I am not sure whether it is the right way to go.

          Show
          Tsz Wo Nicholas Sze added a comment - For adding a Datanode RPC server, we need to store the RPC server port and distribute it. One way is to store the port in DatanodeID. I am not sure whether it is the right way to go.
          Hide
          dhruba borthakur added a comment -

          Maybe we can call it InterDatanodeProtocol to make it more explicit.

          Show
          dhruba borthakur added a comment - Maybe we can call it InterDatanodeProtocol to make it more explicit.
          Hide
          dhruba borthakur added a comment -

          +1.

          Show
          dhruba borthakur added a comment - +1.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          One way is to an inter-DataNode protocol for block manipulation:

          interface BlockProtocol extends VersionedProtocol {
            /**
             * Update the GenerationStamp of a block.
             * @return true iff update was required and done successfully. 
             */
            public boolean updateGenerationStamp(Block block,
                GenerationStamp generationstamp) throws IOException;
          
            /**
             * @return the BlockMetaDataInfo of the block. 
             */
            public BlockMetaDataInfo getBlockMetaDataInfo(Block block) throws IOException;
          }
          
          Show
          Tsz Wo Nicholas Sze added a comment - One way is to an inter-DataNode protocol for block manipulation: interface BlockProtocol extends VersionedProtocol { /** * Update the GenerationStamp of a block. * @ return true iff update was required and done successfully. */ public boolean updateGenerationStamp(Block block, GenerationStamp generationstamp) throws IOException; /** * @ return the BlockMetaDataInfo of the block. */ public BlockMetaDataInfo getBlockMetaDataInfo(Block block) throws IOException; }

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development