[DRILL-3551] CTAS from complex Json source with schema change is not written (and hence not read back ) correctly

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Execution - Data Types
• Assignee: Chun Chang
• Reporter: Parth Chandra
• Created: 24/Jul/15 04:23
• Updated: 15/Sep/15 01:12
• Resolved: 29/Jul/15 14:00

Description

The source data contains -

20K rows with the following -
{"some":"yes","others":{"other":"true","all":"false","sometimes":"yes"}}

200 rows with the following -
{"some":"yes","others":{"other":"true","all":"false","sometimes":"yes","additional":"last
entries only"}}

Creating a table and reading it back returns incorrect data -

CREATE TABLE testparquet as select * from `test.json`;
SELECT * from testparquet;

Yields

yes	{"other":"true","all":"false","sometimes":"yes"}
yes	{"other":"true","all":"false","sometimes":"yes"}
yes	{"other":"true","all":"false","sometimes":"yes"}
yes	{"other":"true","all":"false","sometimes":"yes"}

The "additional" field is missing in all records

Parquet metadata for the created file does not have the 'additional' field

Issue Links

• is related to: DRILL-3577 Counting nested fields on CTAS-created-parquet file/s reports inaccurate results
• relates to: DRILL-3476 Filter on nested element gives wrong results