The CSV format plugin allows two ways of reading data:
- As named columns
- As a single array, called columns, that holds all columns for a row
The named columns feature will corrupt the offset vectors if the last row of the file is truncated: leaves off one or more columns.
To illustrate the CSV data corruption, I created a CSV file, test4.csv, of the following form:
Note that the file is truncated: the command and second field is missing on the last line.
Then, I created a simple test using the "cluster fixture" framework:
The results show we've got a problem:
If the last line were:
Then the offset vector should look like this:
Very likely we have an offset vector that looks like this instead:
When we compute the second column of the second row, we should compute:
Instead we get:
The summary is that a premature EOF appears to cause the "missing" columns to be skipped; they are not filled with a blank value to "bump" the offset vectors to fill in the last row. Instead, they are left at 0, causing havoc downstream in the query.