Pig
  1. Pig
  2. PIG-3173

Partition filter push down does not happen partition keys condition include a AND and OR construct

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.1
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      A = load 'db.table' using org.apache.hcatalog.pig.HCatLoader();
      B = filter A by (region=='usa' AND dt=='201302051800') OR (region=='uk' AND dt=='201302051800');
      C = foreach B generate name, age;
      DUMP C;

      gives the below warning and scans the whole table.

      2013-02-06 22:22:16,233 [main] WARN org.apache.pig.newplan.PColFilterExtractor - No partition filter push down: You have an partition column (region ) in a construction like: (pcond and ...) or (pcond and ...) where pcond is a condition on a partition column.
      2013-02-06 22:22:16,233 [main] WARN org.apache.pig.newplan.PColFilterExtractor - No partition filter push down: You have an partition column (datestamp ) in a construction like: (pcond and ...) or (pcond and ...) where pcond is a condition on a partition column.

      1. PIG-3173-2.patch
        9 kB
        Rohini Palaniswamy
      2. PIG-3173-1.patch
        13 kB
        Rohini Palaniswamy

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          29d 21h 40m 1 Alan Gates 18/Apr/13 22:58
          Open Open Patch Available Patch Available
          50d 6h 26m 2 Rohini Palaniswamy 29/Apr/13 21:23
          Patch Available Patch Available Resolved Resolved
          57m 23s 1 Rohini Palaniswamy 29/Apr/13 22:20
          Resolved Resolved Closed Closed
          167d 19h 25m 1 Daniel Dai 14/Oct/13 17:46
          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Cheolsoo Park made changes -
          Link This issue relates to PIG-3395 [ PIG-3395 ]
          Rohini Palaniswamy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Rohini Palaniswamy added a comment -

          Checked into trunk (0.12). Thanks Dmitriy and Cheolsoo

          Show
          Rohini Palaniswamy added a comment - Checked into trunk (0.12). Thanks Dmitriy and Cheolsoo
          Hide
          Cheolsoo Park added a comment -

          +1 to PIG-3173-2.patch.

          Show
          Cheolsoo Park added a comment - +1 to PIG-3173 -2.patch.
          Rohini Palaniswamy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rohini Palaniswamy made changes -
          Attachment PIG-3173-2.patch [ 12581032 ]
          Rohini Palaniswamy made changes -
          Link This issue is related to PIG-3300 [ PIG-3300 ]
          Hide
          Rohini Palaniswamy added a comment -

          > But this patch at least lets Pig push down expressions when A, B, C, and D are ALL partition conditions. IMO, this alone is a big win. Can we get this patch in and do further optimization on the '(A and B) or (C and D)' case in a separate jira?
          Sure. I had started on the optimization patch but I did not complete it before leaving for vacation. Wanted to be careful as there was lot of change and I had to almost evaluate the whole tree and ensure it works for all combinations as we are extracting partial conditions. I will create a separate jira for that and put the patch later. I will update this patch (first one had a bug) with just pushing down all partition conditions.

          Show
          Rohini Palaniswamy added a comment - > But this patch at least lets Pig push down expressions when A, B, C, and D are ALL partition conditions. IMO, this alone is a big win. Can we get this patch in and do further optimization on the '(A and B) or (C and D)' case in a separate jira? Sure. I had started on the optimization patch but I did not complete it before leaving for vacation. Wanted to be careful as there was lot of change and I had to almost evaluate the whole tree and ensure it works for all combinations as we are extracting partial conditions. I will create a separate jira for that and put the patch later. I will update this patch (first one had a bug) with just pushing down all partition conditions.
          Hide
          Cheolsoo Park added a comment -

          If I understand comments on RB, there is no real issue with the patch other than that we can do better on the '(A and B) or (C and D)' case.

          Currently, Pig rejects all of the following expressions even if A, B, C, and D are all partition conditions:

          • (A and B) or (C and D)
          • (A and B) or C
          • A or (C and D)

          But this patch at least lets Pig push down expressions when A, B, C, and D are ALL partition conditions. IMO, this alone is a big win. Can we get this patch in and do further optimization on the '(A and B) or (C and D)' case in a separate jira?

          Thanks!

          Show
          Cheolsoo Park added a comment - If I understand comments on RB, there is no real issue with the patch other than that we can do better on the '(A and B) or (C and D)' case. Currently, Pig rejects all of the following expressions even if A, B, C, and D are all partition conditions: (A and B) or (C and D) (A and B) or C A or (C and D) But this patch at least lets Pig push down expressions when A, B, C, and D are ALL partition conditions. IMO, this alone is a big win. Can we get this patch in and do further optimization on the '(A and B) or (C and D)' case in a separate jira? Thanks!
          Alan Gates made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Alan Gates added a comment -

          Canceling patch until feedback from Dmitriy is addressed.

          Show
          Alan Gates added a comment - Canceling patch until feedback from Dmitriy is addressed.
          Rohini Palaniswamy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.10.1 [ 12320547 ]
          Fix Version/s 0.12 [ 12323380 ]
          Rohini Palaniswamy made changes -
          Field Original Value New Value
          Attachment PIG-3173-1.patch [ 12574459 ]
          Show
          Rohini Palaniswamy added a comment - https://reviews.apache.org/r/10035/
          Rohini Palaniswamy created issue -

            People

            • Assignee:
              Rohini Palaniswamy
              Reporter:
              Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development