Hadoop Common
  1. Hadoop Common
  2. HADOOP-1306

DFS Scalability: Reduce the number of getAdditionalBlock RPCs on the namenode


    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      One of the most-frequently-invoked RPCs in the namenode is the addBlock() RPC. The DFSClient uses this RPC to allocate one more block for a file that it is currently operating upon. The scalability of the namenode will improve if we can decrease the number of addBlock() RPCs. One idea that we want to discuss here is to make addBlock() return more than one block. This proposal came out of a discussion I had with Ben Reed.

      Let's say that addBlock() returns n blocks for the file. The namenode already tracks these blocks using the pendingCreates data structure. The client guarantees that these n blocks will be used in order. The client also guarantees that if it cannot use a block (dues to whatever reason), it will inform the namenode using the abandonBlock() RPC. These RPCs are already supported.

      Another possible optimization : since the namenode has to allocate n blocks for a file, should it use the same set of datanodes for this set of blocks? My proposal is that if n is a small number (e.g. 3), it is prudent to allocate the same set of datanodes to host all replicas for this set of blocks. This will reduce the CPU spent in chooseTargets().

      1. fineGrainLocks3.patch
        42 kB
        dhruba borthakur


        dhruba borthakur created issue -
        dhruba borthakur made changes -
        Field Original Value New Value
        Attachment fineGrainLocks2.patch [ 12357297 ]
        dhruba borthakur made changes -
        Attachment fineGrainLocks3.patch [ 12357483 ]
        dhruba borthakur made changes -
        Attachment fineGrainLocks2.patch [ 12357297 ]
        dhruba borthakur made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Duplicate [ 3 ]
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s dfs [ 12310710 ]


          • Assignee:
            dhruba borthakur
          • Votes:
            0 Vote for this issue
            0 Start watching this issue


            • Created: