Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9011

Support splitting BlockReport of a storage into multiple RPC

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Currently if a DataNode has too many blocks (more than 1m by default), it sends multiple RPC to the NameNode for the block report, each RPC contains report for a single storage. However, in practice we've seen sometimes even a single storage can contains large amount of blocks and the report even exceeds the max RPC data length. It may be helpful to support sending multiple RPC for the block report of a storage.

      Attachments

        1. HDFS-9011.002.patch
          50 kB
          Jing Zhao
        2. HDFS-9011.001.patch
          48 kB
          Jing Zhao
        3. HDFS-9011.000.patch
          34 kB
          Jing Zhao

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-11313 Segmented Block Reports

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-8574 When block count for a volume exceeds dfs.blockreport.split.threshold, block report causes exception

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              jingzhao Jing Zhao
              jingzhao Jing Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated: