Hadoop Common
  1. Hadoop Common
  2. HADOOP-6949

Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
      Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
      Show
      Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5. Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
    • Tags:
      Writable, primitive arrays, RPC

      Description

      Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n

      It would not be needed to specify each element types for primitive arrays.

      1. arrayprim_v4.patch
        23 kB
        Matt Foley
      2. arrayprim_v5.patch
        24 kB
        Matt Foley
      3. arrayprim_v6.patch
        24 kB
        Matt Foley
      4. arrayprim_v7.patch
        24 kB
        Matt Foley
      5. arrayprim.patch
        20 kB
        Matt Foley
      6. ObjectWritable.diff
        5 kB
        Navis

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Matt Foley
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 10m
                10m
                Remaining:
                Remaining Estimate - 10m
                10m
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development