Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9710

Change DN to send block receipt IBRs in batches

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When a DN has received a block, it immediately sends a block receipt IBR RPC to NN for reporting the block. Even if a DN has received multiple blocks at the same time, it still sends multiple RPCs. It does not scale well since NN has to process a huge number of RPCs when many DNs receiving many blocks at the same time.

        Attachments

        1. HDFS-9710-branch-2.7.01.patch
          40 kB
          Konstantin Shvachko
        2. h9710_20160225.patch
          41 kB
          Tsz Wo Nicholas Sze
        3. h9710_20160224.patch
          39 kB
          Tsz Wo Nicholas Sze
        4. h9710_20160219.patch
          31 kB
          Tsz Wo Nicholas Sze
        5. h9710_20160217.patch
          28 kB
          Tsz Wo Nicholas Sze
        6. h9710_20160216b.patch
          28 kB
          Tsz Wo Nicholas Sze
        7. h9710_20160216.patch
          28 kB
          Tsz Wo Nicholas Sze
        8. h9710_20160205.patch
          28 kB
          Tsz Wo Nicholas Sze
        9. h9710_20160201.patch
          50 kB
          Tsz Wo Nicholas Sze

          Issue Links

            Activity

              People

              • Assignee:
                szetszwo Tsz Wo Nicholas Sze
                Reporter:
                szetszwo Tsz Wo Nicholas Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: