[DRILL-4735] Count(dir0) on parquet returns 0 result

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.4.0, 1.6.0, 1.7.0
• Fix Version/s: 1.12.0
• Component/s: Query Planning & Optimization, Storage - Parquet
• Labels: ready-to-commit
• Assignee: Arina Ielchiieva
• Reporter: Krystal
• Reviewer: Jinfeng Ni
• Created: 21/Jun/16 16:03
• Updated: 21/Sep/17 00:09
• Resolved: 15/Aug/17 13:45

Description

Selecting a count of dir0, dir1, etc against a parquet directory returns 0 rows.

select count(dir0) from `min_max_dir`;
---------

EXPR$0

---------

0

---------

select count(dir1) from `min_max_dir`;
---------

EXPR$0

---------

0

---------

If I put both dir0 and dir1 in the same select, it returns expected result:
select count(dir0), count(dir1) from `min_max_dir`;
----------------+

EXPR$0

EXPR$1

----------------+

600

600

----------------+

Here is the physical plan for count(dir0) query:

00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 20.0, cumulative cost = {22.0 rows, 22.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1346
00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1345
00-02        Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1344
00-03          Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@3da85d3b[columns = null, isStarQuery = false, isSkipQuery = false]]) : rowType = RecordType(BIGINT count): rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1343

Here is part of the explain plan for the count(dir0) and count(dir1) in the same select:

00-00    Screen : rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): rowcount = 60.0, cumulative cost = {1206.0 rows, 15606.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1623
00-01      Project(EXPR$0=[$0], EXPR$1=[$1]) : rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): rowcount = 60.0, cumulative cost = {1200.0 rows, 15600.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1622
00-02        StreamAgg(group=[{}], EXPR$0=[COUNT($0)], EXPR$1=[COUNT($1)]) : rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): rowcount = 60.0, cumulative cost = {1200.0 rows, 15600.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1621
00-03          Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/min_max_dir/1999/Apr/voter20.parquet/0_0_0.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/min_max_dir/1999/MAR/voter15.parquet/0_0_0.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/min_max_dir/1985/jan/voter5.parquet/0_0_0.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/min_max_dir/1985/apr/voter60.parquet/0_0_0.parquet],..., ReadEntryWithPath [path=maprfs:/drill/testdata/min_max_dir/2014/jul/voter35.parquet/0_0_0.parquet]], selectionRoot=maprfs:/drill/testdata/min_max_dir, numFiles=16, usedMetadataFile=false, columns=[`dir0`, `dir1`]]]) : rowType = RecordType(ANY dir0, ANY dir1): rowcount = 600.0, cumulative cost = {600.0 rows, 1200.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1620

Notice that in the first case, "org.apache.drill.exec.store.pojo.PojoRecordReader" is used.

Issue Links

• contains: DRILL-5357 Partition pruning information not available in query plan for COUNT aggregate query
• contains: DRILL-3407 CTAS Auto Partition : The plan for count(*) should show the list of files scanned
• contains: DRILL-1691 ConvertCountToDirectScan rule should be applicable for 2 or more COUNT aggregates
• links to: GitHub Pull Request #882
• links to: GitHub Pull Request #900