Pig
  1. Pig
  2. PIG-2668

projection pushdown should work through Split

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      A = load ...
      B1 = filter A by cond1;
      B2 = filter A by cond2;
      B3 = filter A by cond3;

      B = union B1, B2, B3; does not push projection.

      Load (A)

      Split

      ---------------------------

       

      Filter(B1) Filter(B2) ...

      Because of the split operator introduced between the filter conditions and load, the filter does not get pushed into the load function.

      A simple way to fix this in pig would be to not share the load across the filter operators. Another option is to push the condition (B1 or B2 or B3) into Load operator and retain rest of the current plan (split and filters following the split).

      You can of course achieve the same effect by having a separate load statement as input for each of the filters.

        Activity

          People

          • Assignee:
            Aniket Mokashi
            Reporter:
            Aniket Mokashi
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development