[PIG-2359] Support more efficient Tuples when schemas are known

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11

Release Note

newTupleForSchema(Schema s) and newTupleForSchema(byte... types) methods are introduced to TupleFactory, allowing Pig to choose optimized Tuple implementations for specific schemas, when those are available.

Two optimized implementations are introduced:
- single-value Tuples (tuples which only hold a single int, double, String, etc
- primitive-value Tuples (tuples which only hold values that can be represented by a fixed-width datatype in Java: int, long, double, boolean, float).

Using such tuples can result in significant memory utilization improvements.

• Assignee: Dmitriy V. Ryaboy
• Reporter: Dmitriy V. Ryaboy
• Created: 13/Nov/11 23:05
• Updated: 22/Feb/13 04:53
• Resolved: 17/Jan/12 01:40

Description

Pig Tuples have significant overhead due to the fact that all the fields are Objects.
When a Tuple only contains primitive fields (ints, longs, etc), it's possible to avoid this overhead, which would result in significant memory savings.

Issue Links

• blocks: PIG-2454 Make use of primitive tuples in builtin UDFs and operators
• relates to: PIG-2550 Custom tuple results in "Unexpected datatype 110 while reading tuplefrom binary file" while spilling