[SPARK-24118] Support lineSep format independent from encoding

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Max Gekk
• Created: 29/Apr/18 19:44
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

Currently, the lineSep option of JSON datasource is depend on encoding. It is impossible to define correct lineSep for JSON files with BOM in UTF-16 and UTF-32 encoding, for example. Need to propose a format of lineSep which will represent sequence of octets (bytes) and will be independent from encoding.

Issue Links

• links to: [Github] Pull Request #21192 (MaxGekk)