[DRILL-6318] Push down limit past flatten is incorrect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.14.0
• Labels: ready-to-commit

Environment

Ubuntu Mate 18.04

Apache Drill 1.14.0-SNAPSHOT

• Assignee: Oleg Zinoviev
• Reporter: Oleg Zinoviev
• Reviewer: aj_09
• Created: 10/Apr/18 06:50
• Updated: 05/May/18 19:15
• Resolved: 05/May/18 19:15

Description

Sample data: sample.json

Result of

select flatten(data) as d from dfs.root.`sample.json`

is

d
Honored Boy Scout
Yawning Wolf
Closed Queen
Innocent Volunteer
Junior Wing
Lame Mantis
Old Master
Numb Pawn
Hollow Guardian
Twin Hurricane
Helpless Avalange

Let's try to get first 3 rows:

select flatten(data) as d from dfs.root.`sample.json` limit 3

Result has only 2 rows:

d
Honored Boy Scout
Yawning Wolf

Reason: Limit was pushed down below flatten and only 3 top rows from json was selected. In this 3 rows only 2nd has items in "data" field.

Let's try to get 3 rows from the middle:

select flatten(data) as d from dfs.root.`sample.json` limit 3 offset 5

Result is empty.

Reason: Limit and offset was pushed down below flatten and only 6, 7 and 8 row from json was selected. This 3 rows contains only 3 items in "data" field. After flatten limit and offset applies second time and reject all select items.

Error in org/apache/drill/exec/planner/logical/DrillPushLimitToScanRule.java

Issue Links

• links to: GitHub Pull Request #1204
• links to: GitHub Pull Request #1204