Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3594

Pig 0.11 with HCatLoader partition filter does not push down valid conditions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.11.1
    • Fix Version/s: 0.11.2
    • Component/s: None
    • Labels:
      None

      Description

      When running the following pig query (script.pig) :

      A = LOAD ' my_data' USING org.apache.hcatalog.pig.HCatLoader();
      B = FILTER A by part1=='$val1' and part2 matches '$val2';
      C = FILTER B by map#'key' matches '(val3|val4)';
      DUMP C;

      > pig -useHCatalog -p val1=2013_01_01 val2=XYZ script.pig

      This results in Pig scanning all partitions inspite of filter condition. part1=='$val1' should have been pushed to HCatLoader even though there are other conditions that cannot be pushed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                chitnis Mona Chitnis
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: