[FLINK-16043] Support non-BMP Unicode for JsonRowSerializationSchema

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.10.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned pull-request-available
• Assignee: Unassigned
• Reporter: Benchao Li
• Created: 13/Feb/20 13:08
• Updated: 17/Apr/22 22:38

Description

This is a known issue for jackson: https://github.com/FasterXML/jackson-core/issues/223

You can see more details: https://github.com/FasterXML/jackson-core/blob/master/src/main/java/com/fasterxml/jackson/core/json/UTF8JsonGenerator.java#L2105

 

And I also encountered this issue in my production environment. I've figured out a solution to solve this issue. Java's String.getBytes() can deal with UTF-8 encoding well. So we can do it like this:

mapper.writeValueAsString(node).getBytes() instead of 

mapper.writeValueAsBytes(node)

cc jark twalthr

Issue Links

• links to: GitHub Pull Request #11090