[FLINK-13589] DelimitedInputFormat index error on multi-byte delimiters with whole file input splits

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.8.1
• Fix Version/s: 1.9.2, 1.10.0
• Component/s: Connectors / FileSystem, Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: pull-request-available
• Assignee: Arvid Heise
• Reporter: Adric Eckstein
• Created: 05/Aug/19 20:58
• Updated: 22/Jun/21 14:04
• Resolved: 17/Dec/19 02:18

Description

The DelimitedInputFormat can drops bytes when using input splits that have a length of -1 (for reading the whole file).  It looks like this is a simple bug in handing the delimiter on buffer boundaries where the logic is inconsistent for different split types.

Attached is a possible patch with fix and test.

 

Issue Links

• links to: GitHub Pull Request #10573
• links to: GitHub Pull Request #10574
• links to: GitHub Pull Request #10599