[FLINK-35097] Table API Filesystem connector with 'raw' format repeats last line

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.17.1
• Fix Version/s: 1.18.2, 1.20.0, 1.19.1
• Component/s: Connectors / FileSystem
• Labels: pull-request-available

Environment

I ran the above test with 1.17.1. I checked for existing bug tickets and release notes, but did not find anything, so assuming this effects 1.18 and 1.19.

• Assignee: Kumar Mallikarjuna
• Reporter: David Perkins
• Created: 12/Apr/24 21:04
• Updated: 26/Apr/24 12:30
• Resolved: 25/Apr/24 05:44

Description

When using the Filesystem connector with 'raw' format to read text data that contains new lines, a row is returned for every line, but always contains the contents of the last line.

For example, with the following file.

line 1
line 2
line 3

And table definition

create TABLE MyRawTable (
`doc` string,
) WITH (
'path' = 'file:///path/to/data',
'format' = 'raw',
'connector' = 'filesystem'
);

Selecting `*` from the table produces three rows all with "line 3" for `doc`.

Issue Links

• links to: GitHub Pull Request #24661