Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3201

namenode should be able to retrieve block metadata from a datanode

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The Append design (HADOOP-1700) requires that the namenode be able to contact the datanode when a lease expires.

      When a lease expires, the namenode has to fix up the size of the last block of the file that was being written to. The namenode contacts the datanodes, retrieves the block generation stamp and the length of the lastblock of all known replicas, determines which replicas are good and which one are to be deleted, stamps the winning replicas with a new generation stamp and deletes losing replicas. Once this process is complete, the namenode can allow a new writer to append to this file. Details of this design are in HADOOP-1700.

      A few options available to us:

      1. The namenode sends the request as a response to the next heartbeat RPC from the datanode. This methodology is currently used for requesting block reports from datanodes.

      2. The datanode has an RPC server. A pool of threads in the namenode can be used to make RPCs to the datanodes.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                dhruba Dhruba Borthakur
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: