[FLINK-21207] 'csv.disable-quote-character' can not take effect during deserialization

• Type: New Feature
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.13.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table SQL / Ecosystem
• Labels: pull-request-available stale-assigned
• Assignee: sharkd tu
• Reporter: sharkd tu
• Created: 29/Jan/21 14:58
• Updated: 21/Oct/21 10:43

Description

source table with 'csv.disable-quote-character' = 'true' can not take effect.

 

Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Missing closing quote for valueCaused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Missing closing quote for value at [Source: UNKNOWN; line: 1, column: 29] at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._reportParsingError(CsvParser.java:1250) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder._nextQuotedString(CsvDecoder.java:785) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder.nextString(CsvDecoder.java:630) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:846) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser.nextFieldName(CsvParser.java:665) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1282) at org.apache.flink.formats.csv.CsvRowDataDeserializationSchema.deserialize(CsvRowDataDeserializationSchema.java:155)

Issue Links

• fixes: FLINK-23503 Failed to deserialize csv format data with double quotes
• links to: GitHub Pull Request #14813
• links to: GitHub Pull Request #16612
• links to: GitHub Pull Request #16614