Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4824

Null maps / lists and non-provided state support for JSON fields. Numeric types promotion.

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.0.0
    • None
    • Storage - JSON
    • None

    Description

      There is incorrect output in case of JSON file with complex nested data.

      JSON:

      example.json
      {
        "Field1" : {
        }
}

{
        "Field1" : {
                "InnerField1": {"key1":"value1"},
                "InnerField2": {"key2":"value2"}
        }
}

{
        "Field1" : {
                "InnerField3" : ["value3", "value4"],
                "InnerField4" : ["value5", "value6"]
        }
}

      Query:

      select Field1 from dfs.`/tmp/example.json`

      Incorrect result:

      +---------------------------+
|          Field1           |
+---------------------------+
{"InnerField1":{},"InnerField2":{},"InnerField3":[],"InnerField4":[]}
{"InnerField1":{"key1":"value1"},"InnerField2" {"key2":"value2"},"InnerField3":[],"InnerField4":[]}
{"InnerField1":{},"InnerField2":{},"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
+--------------------------+

      Theres is no need to output missing fields. In case of deeply nested structure we will get unreadable result for user.

      Correct result:

      +--------------------------+
|         Field1           |
+--------------------------+
|{}                                                                     
{"InnerField1":{"key1":"value1"},"InnerField2":{"key2":"value2"}}
{"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
+--------------------------+

      Attachments

        Issue Links

        is duplicated by

        Bug - A problem which impairs or prevents the functions of the product. DRILL-5523 Revert if condition in UnionAllRecordBatch changed in DRILL-5419

        • Major - Major loss of function.
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. DRILL-3806 add metadata for untyped null and simple type promotion

        • Major - Major loss of function.
        • Open
        relates to

        Bug - A problem which impairs or prevents the functions of the product. DRILL-4007 json reader treats empty list inconsistently,

        • Critical - Crashes, loss of data, severe memory leak.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-4555 JsonReader does not support nulls in lists

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-4754 Missing values are not missing

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-5033 Query on JSON that has null as value for each key

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-5396 A flatten query on top of 2 files with one record each causes oversize allocation error randomly

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-5521 IndexOutOfBoundsException during union all for map types

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-5534 convert_from on a json map with null value produces an NPE

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-5756 CONVERT_FROM_JSON on a non-existent column throws NullPointerException

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-1913 Drill did not correctly preserve output field's case sensitivity

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-5288 Duplicate key names must be caught in JSON data

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. DRILL-4842 SELECT * on JSON data results in NumberFormatException

        • Major - Major loss of function.
        • Reviewable

        New Feature - A new feature of the product, which has yet to be developed. DRILL-1257 Handle schema mutations within a batch

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-3444 Implement Is Not Null/Is Null on List of objects - [isnotnull(MAP-REPEATED)] error

        • Critical - Crashes, loss of data, severe memory leak.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-1255 Allow change from integer to double type and vice versa in JSON reader

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-1344 Nested json object null is treated as {}

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-1644 Promote data types where prudent

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-1646 Drill cannot handle empty array/list types in JSON documents

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-1767 Add support for nullable maps

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-1768 Add multi-type support for numeric types in complex writer. (invisible up-promotion)

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-3831 Allow null values in lists

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-5283 Support "is not present" as subtype of "is null" for JSON data

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-1554 Assign NULL value for empty text fields in JSON data sources

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. DRILL-4710 Document Drill's JSON processing rules

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open
        links to

        Web Link Design document

        Web Link GitHub Pull Request #580

        Web Link GitHub Pull Request #580

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            RomanKulyk Roman Kulyk
            Paul Rogers Paul Rogers
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment