Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: API / DataSet
    • Labels:
      None

      Description

      CSV is one of the most commonly used file formats in data wrangling. To load records from CSV files, Flink has provided the basic CsvInputFormat, as well as some variants (e.g., RowCsvInputFormat and PojoCsvInputFormat). However, it seems that the reading process can be improved. For example, we could add a built-in util to automatically infer schemas from CSV headers and samples of data. Also, the current bad record handling method can be improved by somehow keeping the invalid lines (and even the reasons for failed parsing), instead of logging the total number only.

      This is an umbrella issue for all the improvements and bug fixes for the CSV reading process.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              xccui Xingcan Cui
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: