Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4551

Partition filter is not pushed down in case of SPLIT

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.11.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      The below query with implicit split will not push down the partition filters and will scan the whole table.

      A  = LOAD 'db1.table1'        USING org.apache.hive.hcatalog.pig.HCatLoader();
      B = FILTER A BY ( ((date=='20150501' AND pk2 =='1')) and pk3 == '127' );
      C  = FILTER A BY ( ((date=='20150501' AND pk2=='1') OR (date=='20150430' AND pk2=='1')) and pk3 == '127' );
      

      The workaround now is to write two separate LOAD statements for each FILTER. We should do that behind the scenes while planning instead of user having to do that.

        Attachments

        1. pig-4551_v01_notestyet.patch
          10 kB
          Koji Noguchi
        2. pig-4551_v02_notestyet.patch
          11 kB
          Koji Noguchi
        3. pig-4551_v03.patch
          44 kB
          Koji Noguchi

          Activity

            People

            • Assignee:
              knoguchi Koji Noguchi
              Reporter:
              rohini Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: