Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6318

Push down limit past flatten is incorrect

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.13.0
    • Fix Version/s: 1.14.0
    • Component/s: None
    • Labels:
    • Environment:

      Ubuntu Mate 18.04

      Apache Drill 1.14.0-SNAPSHOT

      Description

      Sample data: sample.json

      Result of

      select flatten(data) as d from dfs.root.`sample.json`
      

      is

      d
      Honored Boy Scout
      Yawning Wolf
      Closed Queen
      Innocent Volunteer
      Junior Wing
      Lame Mantis
      Old Master
      Numb Pawn
      Hollow Guardian
      Twin Hurricane
      Helpless Avalange

      Let's try to get first 3 rows:

      select flatten(data) as d from dfs.root.`sample.json` limit 3
      

      Result has only 2 rows:

      d
      Honored Boy Scout
      Yawning Wolf

      Reason: Limit was pushed down below flatten and only 3 top rows from json was selected. In this 3 rows only 2nd has items in "data" field.

      Let's try to get 3 rows from the middle:

      select flatten(data) as d from dfs.root.`sample.json` limit 3 offset 5
      

      Result is empty.

      Reason: Limit and offset was pushed down below flatten and only 6, 7 and 8 row from json was selected. This 3 rows contains only 3 items in "data" field. After flatten limit and offset applies second time and reject all select items.

      Error in org/apache/drill/exec/planner/logical/DrillPushLimitToScanRule.java

        Attachments

        1. sample.json
          0.6 kB
          Oleg Zinoviev

          Activity

            People

            • Assignee:
              le.louch Oleg Zinoviev
              Reporter:
              le.louch Oleg Zinoviev
              Reviewer:
              Aman Sinha
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: