Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9710

Change DN to send block receipt IBRs in batches

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When a DN has received a block, it immediately sends a block receipt IBR RPC to NN for reporting the block. Even if a DN has received multiple blocks at the same time, it still sends multiple RPCs. It does not scale well since NN has to process a huge number of RPCs when many DNs receiving many blocks at the same time.

        Attachments

        1. h9710_20160201.patch
          50 kB
          Tsz-wo Sze
        2. h9710_20160205.patch
          28 kB
          Tsz-wo Sze
        3. h9710_20160216.patch
          28 kB
          Tsz-wo Sze
        4. h9710_20160216b.patch
          28 kB
          Tsz-wo Sze
        5. h9710_20160217.patch
          28 kB
          Tsz-wo Sze
        6. h9710_20160219.patch
          31 kB
          Tsz-wo Sze
        7. h9710_20160224.patch
          39 kB
          Tsz-wo Sze
        8. h9710_20160225.patch
          41 kB
          Tsz-wo Sze
        9. HDFS-9710-branch-2.7.01.patch
          40 kB
          Konstantin Shvachko

          Issue Links

          incorporates

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-9729 Use null to replace DataNode.EMPTY_DEL_HINT

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10810 Setreplication removing block from underconstrcution temporarily when batch IBR is enabled.

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-8999 Allow a file to be closed with COMMITTED but not yet COMPLETE blocks.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-11837 Backport HDFS-9710 to branch-2.7: Change DN to send block receipt IBRs in batches

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-9726 Refactor IBR code to a new class

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                szetszwo Tsz-wo Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: