namenode should be able to retrieve block metadata from a datanode

• Key: HADOOP-3201
• Type: New Feature
• Status: Closed
• Resolution: Duplicate
• Priority: Major
• Assignee: Tsz Wo (Nicholas), SZE
• Reporter: dhruba borthakur

Description

The Append design (HADOOP-1700) requires that the namenode be able to contact the datanode when a lease expires.

When a lease expires, the namenode has to fix up the size of the last block of the file that was being written to. The namenode contacts the datanodes, retrieves the block generation stamp and the length of the lastblock of all known replicas, determines which replicas are good and which one are to be deleted, stamps the winning replicas with a new generation stamp and deletes losing replicas. Once this process is complete, the namenode can allow a new writer to append to this file. Details of this design are in HADOOP-1700.

A few options available to us:

1. The namenode sends the request as a response to the next heartbeat RPC from the datanode. This methodology is currently used for requesting block reports from datanodes.

2. The datanode has an RPC server. A pool of threads in the namenode can be used to make RPCs to the datanodes.

Comments

[Doug Cutting]

Unless is proves unfeasable, it seems preferable to not add a new RPC protocol, daemon and namenode thread pool, but rather to transmit this in heartbeat responses. Lease recovery does not seem to be performance sensitive.

[Tsz Wo (Nicholas), SZE]

It seems that option 1 should work fine.

[dhruba borthakur]

+1 for Option 1.

[Tsz Wo (Nicholas), SZE]

FSNamesystem.gotHeartbeat(..., Object[] xferResults, Object[] deleteList) uses Object[] as pass-by-reference parameters. It is better to define a class and use it as the return type. For example,

class HreatbeatResponse {
    boolean registration;

    Block[] xferBlocks;
    DatanodeInfo[][] xferDatanodeInfo;

    Block[] deleteList;
  }

//the new header would be
HreatbeatResponse FSNamesystem.gotHeartbeat(DatanodeID nodeID,
                              long capacity,
                              long dfsUsed,
                              long remaining,
                              int xceiverCount,
                              int xmitsInProgress)

[dhruba borthakur]

I like the idea of HreatbeatResponse. But it would be better if we change the name to HeartbeatResponse.

[dhruba borthakur]

1. The namenode asks the primary datanode (through the heartbeat response mechanism) to do a lease recovery. It specifies the blockid in the heartbeat response.

2. The primary datanode receives the blocks for which a lease recovery is needed. It contacts the specified datanode(s) in the pipeline, retrieves blockids and genstamps from them. It then makes a RPC to the namenode specifying the generation stamps of the replicas. The namenode then decides which replicas to keep and which to delete and make necessary midifications to the blocksMap. It also removes the lease associated with the file, thereby indicating the lease recovery is complete.

3. Each datanode has an RPC server. The primary datanode makes an RPC to the secondary datanode(s) in the pipeline to retrieve generation stamp of a block(s).

[Tsz Wo (Nicholas), SZE]

This issue is fixed as a part of HADOOP-3283.