[HIVE-26784] Provide an option to skip malformed json entries and process the rest of the data in case JSON Serde

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Hive
• Assignee: Unassigned
• Reporter: Taraka Rama Rao Lethavadla
• Created: 28/Nov/22 06:58
• Updated: 28/Nov/22 06:58

Description

We may want to create hive tables on top of audit data generated by ranger (not only HDFS audit but other services as well) to be able to analyze the same.

Based on this case, some audit records may become corrupt. Improving the hive built-in json serde to provide an option (in tblproperties and via session configurations for example) to allow ignoring malformed records (or to provide empty rows) would assist users achieve this goal without using third-party/custom serdes,

for example:
#see ignore.malformed.json

https://aws.amazon.com/premiumsupport/knowledge-center/error-json-athena/#:~:text=you%20can%20ignore%20malformed%20records