[DRILL-3577] Counting nested fields on CTAS-created-parquet file/s reports inaccurate results

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Resolved
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Functions - Drill
• Assignee: Vitalii Diravka
• Reporter: Hanifi Gunes
• Created: 29/Jul/15 15:55
• Updated: 20/Apr/16 17:02
• Resolved: 20/Apr/16 17:02

Description

I have not tried this at a smaller scale nor on JSON file directly but the following seems to re-prod the issue

1. Create an input file as follows
20K rows with the following -
{"some":"yes","others":{"other":"true","all":"false","sometimes":"yes"}}
200 rows with the following -
{"some":"yes","others":{"other":"true","all":"false","sometimes":"yes","additional":"last
entries only"}}

2. CTAS as follows

CREATE TABLE dfs.`tmp`.`tp` as select * from dfs.`data.json` t

This should read

Fragment Number of records written
0_0	20200

3. Count on nested fields via

select count(t.others.additional) from dfs.`tmp`.`tp` t
OR
select count(t.others.other) from dfs.`tmp`.`tp` t

reports no rows as follows

EXPR$0
0

While

select count(t.`some`) from dfs.`tmp`.`tp` t where t.others.additional is not null

reports expected 200 rows

EXPR$0
200

Issue Links

• is related to: DRILL-3806 add metadata for untyped null and simple type promotion
• is related to: DRILL-4505 Can't group by or sort across files with different schema
• is related to: DRILL-4614 Drill must appoint one data type per one column for self-describing data while querying directories
• is related to: DRILL-4538 Turn on Union type by default
• relates to: DRILL-3551 CTAS from complex Json source with schema change is not written (and hence not read back ) correctly