[DRILL-6857] Limit is not being pushed into scan when selecting from a parquet file with multiple row groups.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Labels: ready-to-commit
• Assignee: Arina Ielchiieva
• Reporter: Anton Gozhiy
• Reviewer: Vova Vysotskyi
• Created: 15/Nov/18 17:39
• Updated: 28/Nov/18 11:19
• Resolved: 26/Nov/18 16:33

Description

Data:
A parquet file that contains more than one row group. Example is attached.

Query:

explain plan for select * from dfs.tmp.`DRILL_5796_test_data.parquet` limit 1

Expected result:
numFiles=1, numRowGroups=1

Actual result:
numFiles=1, numRowGroups=3

00-00    Screen : rowType = RecordType(DYNAMIC_STAR **): rowcount = 1.0, cumulative cost = {274.1 rows, 280.1 cpu, 270.0 io, 0.0 network, 0.0 memory}, id = 13671
00-01      Project(**=[$0]) : rowType = RecordType(DYNAMIC_STAR **): rowcount = 1.0, cumulative cost = {274.0 rows, 280.0 cpu, 270.0 io, 0.0 network, 0.0 memory}, id = 13670
00-02        SelectionVectorRemover : rowType = RecordType(DYNAMIC_STAR **): rowcount = 1.0, cumulative cost = {273.0 rows, 279.0 cpu, 270.0 io, 0.0 network, 0.0 memory}, id = 13669
00-03          Limit(fetch=[1]) : rowType = RecordType(DYNAMIC_STAR **): rowcount = 1.0, cumulative cost = {272.0 rows, 278.0 cpu, 270.0 io, 0.0 network, 0.0 memory}, id = 13668
00-04            Limit(fetch=[1]) : rowType = RecordType(DYNAMIC_STAR **): rowcount = 1.0, cumulative cost = {271.0 rows, 274.0 cpu, 270.0 io, 0.0 network, 0.0 memory}, id = 13667
00-05              Scan(table=[[dfs, tmp, DRILL_5796_test_data.parquet]], groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/DRILL_5796_test_data.parquet]], selectionRoot=maprfs:/tmp/DRILL_5796_test_data.parquet, numFiles=1, numRowGroups=3, usedMetadataFile=false, columns=[`**`]]]) : rowType = RecordType(DYNAMIC_STAR **): rowcount = 270.0, cumulative cost = {270.0 rows, 270.0 cpu, 270.0 io, 0.0 network, 0.0 memory}, id = 13666

Note:
The limit pushdown works with the same data partitioned by files (1 row group for a file )

Issue Links

• relates to: DRILL-5796 Filter pruning for multi rowgroup parquet file
• links to: GitHub Pull Request #1548