[DRILL-4479] JsonReader should pick a less restrictive type when creating the default column

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.7.0
• Component/s: Storage - JSON
• Assignee: Aman Sinha
• Reporter: Aman Sinha
• Reviewer: Chun Chang
• Created: 05/Mar/16 01:40
• Updated: 03/Jun/16 21:30
• Resolved: 14/Mar/16 20:48

Description

This JIRA is related to DRILL-3806 but has a narrower scope, so I decided to create separate one.

The JsonReader has the method ensureAtLeastOneField() (see https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java#L91) that ensures that when no columns are found, create an empty one and it chooses to create a nullable int column. One consequence is that queries of the following type fail:

select c1 from dfs.`mostlynulls.json`;
...
...
| null  |
| null  |
Error: DATA_READ ERROR: Error parsing JSON - You tried to write a VarChar type when you are using a ValueWriter of type NullableIntWriterImpl.

File  /Users/asinha/data/mostlynulls.json
Record  4097

In this file the first 4096 rows have NULL values for c1 followed by rows that have a valid string.

It would be useful for the Json reader to choose a less restrictive type such as varchar in order to allow more types of queries to run.

Issue Links

• is related to: DRILL-3806 add metadata for untyped null and simple type promotion