[DRILL-4824] Null maps / lists and non-provided state support for JSON fields. Numeric types promotion.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: Storage - JSON
• Assignee: Unassigned
• Reporter: Roman Kulyk
• Reviewer: Paul Rogers
• Created: 03/Aug/16 10:12
• Updated: 23/Mar/20 09:23

Description

There is incorrect output in case of JSON file with complex nested data.

JSON:

example.json

{
        "Field1" : {
        }
}

{
        "Field1" : {
                "InnerField1": {"key1":"value1"},
                "InnerField2": {"key2":"value2"}
        }
}

{
        "Field1" : {
                "InnerField3" : ["value3", "value4"],
                "InnerField4" : ["value5", "value6"]
        }
}

Query:

select Field1 from dfs.`/tmp/example.json`

Incorrect result:

+---------------------------+
|          Field1           |
+---------------------------+
{"InnerField1":{},"InnerField2":{},"InnerField3":[],"InnerField4":[]}
{"InnerField1":{"key1":"value1"},"InnerField2" {"key2":"value2"},"InnerField3":[],"InnerField4":[]}
{"InnerField1":{},"InnerField2":{},"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
+--------------------------+

Theres is no need to output missing fields. In case of deeply nested structure we will get unreadable result for user.

Correct result:

+--------------------------+
|         Field1           |
+--------------------------+
|{}                                                                     
{"InnerField1":{"key1":"value1"},"InnerField2":{"key2":"value2"}}
{"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
+--------------------------+

Issue Links

• is duplicated by: DRILL-5523 Revert if condition in UnionAllRecordBatch changed in DRILL-5419
• is related to: DRILL-3806 add metadata for untyped null and simple type promotion
• relates to: DRILL-4007 json reader treats empty list inconsistently,
• relates to: DRILL-4555 JsonReader does not support nulls in lists
• relates to: DRILL-4754 Missing values are not missing
• relates to: DRILL-5033 Query on JSON that has null as value for each key
• relates to: DRILL-5396 A flatten query on top of 2 files with one record each causes oversize allocation error randomly
• relates to: DRILL-5521 IndexOutOfBoundsException during union all for map types
• relates to: DRILL-5534 convert_from on a json map with null value produces an NPE
• relates to: DRILL-5756 CONVERT_FROM_JSON on a non-existent column throws NullPointerException
• relates to: DRILL-1913 Drill did not correctly preserve output field's case sensitivity
• relates to: DRILL-5288 Duplicate key names must be caught in JSON data
• relates to: DRILL-4842 SELECT * on JSON data results in NumberFormatException
• relates to: DRILL-1257 Handle schema mutations within a batch
• relates to: DRILL-3444 Implement Is Not Null/Is Null on List of objects - [isnotnull(MAP-REPEATED)] error
• relates to: DRILL-1255 Allow change from integer to double type and vice versa in JSON reader
• relates to: DRILL-1344 Nested json object null is treated as {}
• relates to: DRILL-1644 Promote data types where prudent
• relates to: DRILL-1646 Drill cannot handle empty array/list types in JSON documents
• relates to: DRILL-1767 Add support for nullable maps
• relates to: DRILL-1768 Add multi-type support for numeric types in complex writer. (invisible up-promotion)
• relates to: DRILL-3831 Allow null values in lists
• relates to: DRILL-5283 Support "is not present" as subtype of "is null" for JSON data
• relates to: DRILL-1554 Assign NULL value for empty text fields in JSON data sources
• relates to: DRILL-4710 Document Drill's JSON processing rules
• links to: Design document
• links to: GitHub Pull Request #580
• links to: GitHub Pull Request #580