Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7435

PB encoding of block reports is very inefficient

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0-alpha1
    • Fix Version/s: 2.7.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Block reports are encoded as a PB repeating long. Repeating fields use an ArrayList with default capacity of 10. A block report containing tens or hundreds of thousand of longs (3 for each replica) is extremely expensive since the ArrayList must realloc many times. Also, decoding repeating fields will box the primitive longs which must then be unboxed.

        Attachments

        1. HDFS-7435.000.patch
          59 kB
          Jing Zhao
        2. HDFS-7435.001.patch
          59 kB
          Jing Zhao
        3. HDFS-7435.002.patch
          63 kB
          Jing Zhao
        4. HDFS-7435.patch
          75 kB
          Daryn Sharp
        5. HDFS-7435.patch
          75 kB
          Daryn Sharp
        6. HDFS-7435.patch
          74 kB
          Daryn Sharp
        7. HDFS-7435.patch
          75 kB
          Daryn Sharp
        8. HDFS-7435.patch
          67 kB
          Daryn Sharp
        9. HDFS-7435.patch
          67 kB
          Daryn Sharp
        10. HDFS-7435.patch
          64 kB
          Daryn Sharp
        11. HDFS-7435.patch
          54 kB
          Daryn Sharp
        12. HDFS-7435.patch
          16 kB
          Daryn Sharp

          Issue Links

            Activity

              People

              • Assignee:
                daryn Daryn Sharp
                Reporter:
                daryn Daryn Sharp
              • Votes:
                0 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: