Pig
  1. Pig
  2. PIG-688

PERFORMANCE: Vectorize operators

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      By Vectorization, I mean passing multiple (/vector of) records at a time between operators (and potentially other functions like udfs)

      Vectorization of pig operators can improve performance by
      1. improving locality and cache utilization
      2. Reducing number of function calls. Many functions calls are likely to be dynamically resolved. There may be some checks in each function that we might be able to do once for several recs.
      3. Potentially benefit from cpu pipeline architecture. ( But I don't know how good java VM is at that ..)

      To do vectorization in map stage, we need to use MapRunner - see PIG-687.

        Activity

        Thejas M Nair created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Thejas M Nair
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development