Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6318

Push down limit past flatten is incorrect

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.13.0
    • 1.14.0
    • None
    • Ubuntu Mate 18.04

      Apache Drill 1.14.0-SNAPSHOT

    Description

      Sample data: sample.json

      Result of

      select flatten(data) as d from dfs.root.`sample.json`
      

      is

      d
      Honored Boy Scout
      Yawning Wolf
      Closed Queen
      Innocent Volunteer
      Junior Wing
      Lame Mantis
      Old Master
      Numb Pawn
      Hollow Guardian
      Twin Hurricane
      Helpless Avalange

      Let's try to get first 3 rows:

      select flatten(data) as d from dfs.root.`sample.json` limit 3
      

      Result has only 2 rows:

      d
      Honored Boy Scout
      Yawning Wolf

      Reason: Limit was pushed down below flatten and only 3 top rows from json was selected. In this 3 rows only 2nd has items in "data" field.

      Let's try to get 3 rows from the middle:

      select flatten(data) as d from dfs.root.`sample.json` limit 3 offset 5
      

      Result is empty.

      Reason: Limit and offset was pushed down below flatten and only 6, 7 and 8 row from json was selected. This 3 rows contains only 3 items in "data" field. After flatten limit and offset applies second time and reject all select items.

      Error in org/apache/drill/exec/planner/logical/DrillPushLimitToScanRule.java

      Attachments

        1. sample.json
          0.6 kB
          Oleg Zinoviev

        Activity

          People

            le.louch Oleg Zinoviev
            le.louch Oleg Zinoviev
            Aman Sinha Aman Sinha
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: