Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9710

Change DN to send block receipt IBRs in batches

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 2.7.4, 3.0.0-alpha1
    • datanode
    • None
    • Reviewed

    Description

      When a DN has received a block, it immediately sends a block receipt IBR RPC to NN for reporting the block. Even if a DN has received multiple blocks at the same time, it still sends multiple RPCs. It does not scale well since NN has to process a huge number of RPCs when many DNs receiving many blocks at the same time.

      Attachments

        1. h9710_20160201.patch
          50 kB
          Tsz-wo Sze
        2. h9710_20160205.patch
          28 kB
          Tsz-wo Sze
        3. h9710_20160216.patch
          28 kB
          Tsz-wo Sze
        4. h9710_20160216b.patch
          28 kB
          Tsz-wo Sze
        5. h9710_20160217.patch
          28 kB
          Tsz-wo Sze
        6. h9710_20160219.patch
          31 kB
          Tsz-wo Sze
        7. h9710_20160224.patch
          39 kB
          Tsz-wo Sze
        8. h9710_20160225.patch
          41 kB
          Tsz-wo Sze
        9. HDFS-9710-branch-2.7.01.patch
          40 kB
          Konstantin Shvachko

        Issue Links

          incorporates

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-9729 Use null to replace DataNode.EMPTY_DEL_HINT

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10810 Setreplication removing block from underconstrcution temporarily when batch IBR is enabled.

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-8999 Allow a file to be closed with COMMITTED but not yet COMPLETE blocks.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-11837 Backport HDFS-9710 to branch-2.7: Change DN to send block receipt IBRs in batches

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-9726 Refactor IBR code to a new class

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: