[HIVE-21240] JSON SerDe Re-Write

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1, 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Serializers/Deserializers
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Flags: Patch, Important
• Assignee: David Mollitor
• Reporter: David Mollitor
• Created: 11/Feb/19 16:11
• Updated: 17/Nov/22 08:54
• Resolved: 08/May/19 16:25

Description

The JSON SerDe has a few issues, I will link them to this JIRA.

• Use Jackson Tree parser instead of manually parsing
• Added support for base-64 encoded data (the expected format when using JSON)
• Added support to skip blank lines (returns all columns as null values)
• Current JSON parser accepts, but does not apply, custom timestamp formats in most cases
• Added some unit tests
• Added cache for column-name to column-index searches, currently O(n) for each row processed, for each column in the row

Issue Links

• is depended upon by: HIVE-18785 Make JSON SerDe First-Class SerDe
• relates to: HIVE-21356 Upgrade Jackson to 2.9.8
• supercedes: HIVE-20375 Json SerDe ignoring the timestamp.formats property
• supercedes: HIVE-20900 serde2.JsonSerDe no longer supports timestamp.formats
• supercedes: HIVE-21317 Unit Test kafka_storage_handler Is Failing Regularly
• links to: GitHub Pull Request #530