Pig
  1. Pig
  2. PIG-2359

Support more efficient Tuples when schemas are known

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      newTupleForSchema(Schema s) and newTupleForSchema(byte... types) methods are introduced to TupleFactory, allowing Pig to choose optimized Tuple implementations for specific schemas, when those are available.

      Two optimized implementations are introduced:
      - single-value Tuples (tuples which only hold a single int, double, String, etc
      - primitive-value Tuples (tuples which only hold values that can be represented by a fixed-width datatype in Java: int, long, double, boolean, float).

      Using such tuples can result in significant memory utilization improvements.
      Show
      newTupleForSchema(Schema s) and newTupleForSchema(byte... types) methods are introduced to TupleFactory, allowing Pig to choose optimized Tuple implementations for specific schemas, when those are available. Two optimized implementations are introduced: - single-value Tuples (tuples which only hold a single int, double, String, etc - primitive-value Tuples (tuples which only hold values that can be represented by a fixed-width datatype in Java: int, long, double, boolean, float). Using such tuples can result in significant memory utilization improvements.

      Description

      Pig Tuples have significant overhead due to the fact that all the fields are Objects.
      When a Tuple only contains primitive fields (ints, longs, etc), it's possible to avoid this overhead, which would result in significant memory savings.

      1. PIG-2359.1.patch
        64 kB
        Dmitriy V. Ryaboy
      2. PIG-2359.2.patch
        78 kB
        Dmitriy V. Ryaboy
      3. PIG-2359.3.patch
        98 kB
        Dmitriy V. Ryaboy
      4. PIG-2359.4.patch
        100 kB
        Dmitriy V. Ryaboy

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Dmitriy V. Ryaboy
              Reporter:
              Dmitriy V. Ryaboy
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development