[FLINK-2186] Rework CSV import to support very wide files

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: API / Scala
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned pull-request-available
• Assignee: Unassigned
• Reporter: Theodore Vasiloudis
• Created: 08/Jun/15 14:52
• Updated: 11/May/23 01:10

Description

In the current readVcsFile implementation, importing CSV files with many columns can become from cumbersome to impossible.

For example to import an 11 column file we need to write:

val cancer = env.readCsvFile[(String, String, String, String, String, String, String, String, String, String, String)]("/path/to/breast-cancer-wisconsin.data")

For many use cases in Machine Learning we might have CSV files with thousands or millions of columns that we want to import as vectors.
In that case using the current readCsvFile method becomes impossible.

We therefore need to rework the current function, or create a new one that will allow us to import CSV files with an arbitrary number of columns.

Issue Links

• requires: FLINK-5186 Move Row and RowTypeInfo into Flink core
• links to: GitHub Pull Request #3012
• links to: GitHub Pull Request #3012