Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6949

Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.22.0, 0.23.0
    • io
    • None
    • Incompatible change, Reviewed
    • Hide
      Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
      Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
      Show
      Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5. Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
    • Writable, primitive arrays, RPC

    Description

      Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n

      It would not be needed to specify each element types for primitive arrays.

      Attachments

        1. ObjectWritable.diff
          5 kB
          Navis Ryu
        2. arrayprim.patch
          20 kB
          Matthew Foley
        3. arrayprim_v4.patch
          23 kB
          Matthew Foley
        4. arrayprim_v5.patch
          24 kB
          Matthew Foley
        5. arrayprim_v6.patch
          24 kB
          Matthew Foley
        6. arrayprim_v7.patch
          24 kB
          Matthew Foley

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1583 Improve backup-node sync performance by wrapping RPC parameters

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-2126 Improve Namenode startup time [umbrella task]

          • Major - Major loss of function.
          • Open
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-7158 Reduce RPC packet size for homogeneous arrays, such as the array responses to listStatus() and getBlockLocations()

          • Major - Major loss of function.
          • Open

          Activity

            People

              mattf Matthew Foley
              navis Navis Ryu
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 10m
                  10m
                  Remaining:
                  Remaining Estimate - 10m
                  10m
                  Logged:
                  Time Spent - Not Specified
                  Not Specified