Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2359

Support more efficient Tuples when schemas are known

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      newTupleForSchema(Schema s) and newTupleForSchema(byte... types) methods are introduced to TupleFactory, allowing Pig to choose optimized Tuple implementations for specific schemas, when those are available.

      Two optimized implementations are introduced:
      - single-value Tuples (tuples which only hold a single int, double, String, etc
      - primitive-value Tuples (tuples which only hold values that can be represented by a fixed-width datatype in Java: int, long, double, boolean, float).

      Using such tuples can result in significant memory utilization improvements.
      Show
      newTupleForSchema(Schema s) and newTupleForSchema(byte... types) methods are introduced to TupleFactory, allowing Pig to choose optimized Tuple implementations for specific schemas, when those are available. Two optimized implementations are introduced: - single-value Tuples (tuples which only hold a single int, double, String, etc - primitive-value Tuples (tuples which only hold values that can be represented by a fixed-width datatype in Java: int, long, double, boolean, float). Using such tuples can result in significant memory utilization improvements.

      Description

      Pig Tuples have significant overhead due to the fact that all the fields are Objects.
      When a Tuple only contains primitive fields (ints, longs, etc), it's possible to avoid this overhead, which would result in significant memory savings.

        Attachments

        1. PIG-2359.4.patch
          100 kB
          Dmitriy V. Ryaboy
        2. PIG-2359.3.patch
          98 kB
          Dmitriy V. Ryaboy
        3. PIG-2359.2.patch
          78 kB
          Dmitriy V. Ryaboy
        4. PIG-2359.1.patch
          64 kB
          Dmitriy V. Ryaboy

          Issue Links

            Activity

              People

              • Assignee:
                dvryaboy Dmitriy V. Ryaboy
                Reporter:
                dvryaboy Dmitriy V. Ryaboy
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: