Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11313

Segmented Block Reports

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.6.2
    • None
    • datanode, namenode
    • None

    Description

      Block reports from a single DataNode can be currently split into multiple RPCs each reporting a single DataNode storage (disk). The reports are still large since disks are getting bigger. Splitting blockReport RPCs into multiple smaller calls would improve NameNode performance and overall HDFS stability.
      This was discussed in multiple jiras. Here the approach is to let NameNode divide blockID space into segments and then ask DataNodes to report replicas in a particular range of IDs.

      Attachments

        1. SegmentedBlockReports.pdf
          122 kB
          Vinitha Reddy Gankidi

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16452 Increase ipc.maximum.data.length default from 64MB to 128MB

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-9011 Support splitting BlockReport of a storage into multiple RPC

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-14657 Refine NameSystem lock usage during processing FBR

          • Major - Major loss of function.
          • Patch Available

          Activity

            People

              redvine Vinitha Reddy Gankidi
              shv Konstantin Shvachko
              Votes:
              0 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

                Created:
                Updated: