[SPARK-29444] Add configuration to support JacksonGenrator to keep fields with null values

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Jackey Lee
• Reporter: Jackey Lee
• Created: 12/Oct/19 02:08
• Updated: 22/Jan/22 00:02
• Resolved: 18/Oct/19 08:07

Description

DataSet.toJSON will lost some column when field data is null. Maybe it is better to keep null data in some scenarios.

Such as sparkmagic, which is widely used in jupyter with livy, we use toJSON to toJSON is used to get sql results. with toJSON sparkmagic may return empty results, which confused users.

Maybe adding a config is the best choice. This configuration retains the current semantics and will remain fields with null until the configuration is modified.

Issue Links

• Blocked: SPARK-23773 JacksonGenerator does not include keys that have null value for StructTypes
• is duplicated by: SPARK-29610 Keys with Null values are discarded when using to_json function
• is duplicated by: SPARK-37981 Deletes columns with all Null as default.
• links to: GitHub Pull Request #26098
• links to: GitHub Pull Request #26227