[DRILL-6096] Provide mechanisms to specify field delimiters and quoted text for TextRecordWriter

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.17.0
• Component/s: Storage - Text & CSV
• Labels: doc-impacting ready-to-commit
• Assignee: Arina Ielchiieva
• Reporter: Kunal Khatua
• Reviewer: Vova Vysotskyi
• Created: 18/Jan/18 23:03
• Updated: 17/Feb/20 13:39
• Resolved: 22/Oct/19 10:19

Description

Currently, there is no way for a user to specify the field delimiter for the writing records as a text output. Further more, if the fields contain the delimiter, we have no mechanism of specifying quotes.

By default, quotes should be used to enclose non-numeric fields being written.

Description of the implemented changes:

2 options are added to control text writer output:
store.text.writer.add_header - indicates if header should be added in created text file. Default is true.
store.text.writer.force_quotes - indicates if all value should be quoted. Default is false. It means only values that contain special characters (line / field separators) will be quoted.

Line / field separators, quote / escape characters can be configured using text format configuration using Web UI. User can create special format only for writing data and then use it when creating files. Though such format can be always used to read back written data.

  "formats": {
    "write_text": {
      "type": "text",
      "extensions": [
        "txt"
      ],
      "lineDelimiter": "\n",
      "fieldDelimiter": "!",
      "quote": "^",
      "escape": "^",
    }
   },
...

Next set specified format and create text file:

alter session set `store.format` = 'write_text';
create table dfs.tmp.t as select 1 as id from (values(1));

Notes:
1. To write data univocity-parsers are used, they limit line separator length to not more than 2 characters, though Drill allows setting more 2 chars as line separator since Drill can read data splitting by line separator of any length, during data write exception will be thrown.
2. extractHeader in text format configuration does not affect if header will be written to text file, only store.text.writer.add_header controls this action. extractHeader is used only when reading the data.

Issue Links

• is related to: DRILL-4788 Exporting from Parquet to CSV - commas in strings are not escaped
• is related to: DRILL-6958 CTAS csv with option
• is related to: DRILL-6842 Export to CSV using CREATE TABLE AS (CTAS) wrong parsed
• links to: GitHub Pull Request #1873