Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6949

Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
      Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
      Show
      Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5. Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
    • Tags:
      Writable, primitive arrays, RPC

      Description

      Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n

      It would not be needed to specify each element types for primitive arrays.

        Attachments

        1. arrayprim_v4.patch
          23 kB
          Matt Foley
        2. arrayprim_v5.patch
          24 kB
          Matt Foley
        3. arrayprim_v6.patch
          24 kB
          Matt Foley
        4. arrayprim_v7.patch
          24 kB
          Matt Foley
        5. arrayprim.patch
          20 kB
          Matt Foley
        6. ObjectWritable.diff
          5 kB
          Navis

          Issue Links

            Activity

              People

              • Assignee:
                mattf Matt Foley
                Reporter:
                navis Navis
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 10m
                  10m
                  Remaining:
                  Remaining Estimate - 10m
                  10m
                  Logged:
                  Time Spent - Not Specified
                  Not Specified