Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9710

Change DN to send block receipt IBRs in batches

          Details

          • Type: Improvement
          • Status: Resolved
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: None
          • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
          • Component/s: datanode
          • Labels:
            None
          • Target Version/s:
          • Hadoop Flags:
            Reviewed

            Description

            When a DN has received a block, it immediately sends a block receipt IBR RPC to NN for reporting the block. Even if a DN has received multiple blocks at the same time, it still sends multiple RPCs. It does not scale well since NN has to process a huge number of RPCs when many DNs receiving many blocks at the same time.

              Attachments

              1. HDFS-9710-branch-2.7.01.patch
                40 kB
                Konstantin Shvachko
              2. h9710_20160225.patch
                41 kB
                Tsz Wo Nicholas Sze
              3. h9710_20160224.patch
                39 kB
                Tsz Wo Nicholas Sze
              4. h9710_20160219.patch
                31 kB
                Tsz Wo Nicholas Sze
              5. h9710_20160217.patch
                28 kB
                Tsz Wo Nicholas Sze
              6. h9710_20160216b.patch
                28 kB
                Tsz Wo Nicholas Sze
              7. h9710_20160216.patch
                28 kB
                Tsz Wo Nicholas Sze
              8. h9710_20160205.patch
                28 kB
                Tsz Wo Nicholas Sze
              9. h9710_20160201.patch
                50 kB
                Tsz Wo Nicholas Sze

                Issue Links

                incorporates

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-9729 Use null to replace DataNode.EMPTY_DEL_HINT

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved
                is related to

                Bug - A problem which impairs or prevents the functions of the product. HDFS-10810 Setreplication removing block from underconstrcution temporarily when batch IBR is enabled.

                • Major - Major loss of function.
                • Resolved
                relates to

                Bug - A problem which impairs or prevents the functions of the product. HDFS-8999 Allow a file to be closed with COMMITTED but not yet COMPLETE blocks.

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-11837 Backport HDFS-9710 to branch-2.7: Change DN to send block receipt IBRs in batches

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-9726 Refactor IBR code to a new class

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      szetszwo Tsz Wo Nicholas Sze
                      Reporter:
                      szetszwo Tsz Wo Nicholas Sze
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      23 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: