Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7435

PB encoding of block reports is very inefficient

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.0.0-alpha, 3.0.0-alpha1
    • 2.7.0
    • datanode, namenode
    • None
    • Reviewed

    Description

      Block reports are encoded as a PB repeating long. Repeating fields use an ArrayList with default capacity of 10. A block report containing tens or hundreds of thousand of longs (3 for each replica) is extremely expensive since the ArrayList must realloc many times. Also, decoding repeating fields will box the primitive longs which must then be unboxed.

      Attachments

        1. HDFS-7435.patch
          16 kB
          Daryn Sharp
        2. HDFS-7435.000.patch
          59 kB
          Jing Zhao
        3. HDFS-7435.001.patch
          59 kB
          Jing Zhao
        4. HDFS-7435.002.patch
          63 kB
          Jing Zhao
        5. HDFS-7435.patch
          54 kB
          Daryn Sharp
        6. HDFS-7435.patch
          64 kB
          Daryn Sharp
        7. HDFS-7435.patch
          67 kB
          Daryn Sharp
        8. HDFS-7435.patch
          67 kB
          Daryn Sharp
        9. HDFS-7435.patch
          75 kB
          Daryn Sharp
        10. HDFS-7435.patch
          74 kB
          Daryn Sharp
        11. HDFS-7435.patch
          75 kB
          Daryn Sharp
        12. HDFS-7435.patch
          75 kB
          Daryn Sharp

        Issue Links

          breaks

          Sub-task - The sub-task of the issue HDFS-9484 NNThroughputBenchmark$BlockReportStats should not send empty block reports

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10569 A bug causes OutOfIndex error in BlockListAsLongs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          contains

          Bug - A problem which impairs or prevents the functions of the product. HDFS-7906 BlockReport of a RUR can have a FINALIZED original replica from Truncate

          • Major - Major loss of function.
          • Closed
          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16284 Unboxing long value in old version's block report cause full GC

          • Major - Major loss of function.
          • Open
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-11339 Reuse buffer for Hadoop RPC

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-9221 HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array

          • Major - Major loss of function.
          • Closed

          Activity

            People

              daryn Daryn Sharp
              daryn Daryn Sharp
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: