[HADOOP-7158] Reduce RPC packet size for homogeneous arrays, such as the array responses to listStatus() and getBlockLocations()

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.22.0
• Component/s: io
• Hadoop Flags: Incompatible change
• Tags: Writable, homogeneous arrays, RPC
• Assignee: Unassigned
• Reporter: Matthew Foley
• Created: 28/Feb/11 09:22
• Updated: 03/Jan/19 19:50

Description

While commenting on HADOOP-6949, which proposes a big improvement in the RPC wire format for arrays of primitives, Konstantin Shvachko said:
"Can/should we extend this to arrays of non-primitive types? This should benefit return types for calls like listStatus() and getBlockLocations() on a large directory."

The improvement for primitive arrays is based on not type-labeling every element in the array, so the array in question must be strictly homogenous; it cannot have subtypes of the assignable type. For instance, it could not be applied to heartbeat responses of DatanodeCommand[], whose array elements carry subtypes of DatanodeCommand, each of which must be type-labeled independently. However, as Konstantin points out, it could really help lengthy response arrays for things like listStatus() and getBlockLocations().

I will attach a prototype implementation to this Jira, for discussion. However, since it can't be automatically applied to all arrays passing through RPC, I'll just providing the wrapper type. By using it, a caller is asserting that the array is strictly homogeneous in the above sense.

Issue Links

• relates to: HADOOP-6949 Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting