[SPARK-13108] Encoding not working with non-ascii compatible encodings (UTF-16/32 etc.)

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Later
• Affects Version/s: 2.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Hyukjin Kwon
• Created: 01/Feb/16 05:41
• Updated: 12/Dec/22 18:10
• Resolved: 26/May/16 22:42

Description

This library uses Hadoop's TextInputFormat, which uses LineRecordReader.

According to MAPREDUCE-232, it looks TextInputFormat does not guarantee all encoding types but officially only UTF-8 (as commented in LineRecordReader#L147).

According to MAPREDUCE-232#comment-13183601, it still looks fine with most encodings though but without UTF-16/32.

In more details,

I tested this in Max OS. I converted `cars_iso-8859-1.csv` into `cars_utf-16.csv` as below:

iconv -f iso-8859-1 -t utf-16 < cars_iso-8859-1.csv > cars_utf-16.csv

and run the codes below:

val cars = "cars_utf-16.csv"
sqlContext.read
  .format("csv")
  .option("charset", "utf-16")
  .option("delimiter", 'þ')
  .load(cars)
  .show()

This produces a wrong results below:

+----+-----+-----+--------------------+------+
|year| make|model|             comment|blank�|
+----+-----+-----+--------------------+------+
|2012|Tesla|    S|          No comment|     �|
|   �| null| null|                null|  null|
|1997| Ford| E350|Go get one now th...|     �|
|2015|Chevy|Volt�|                null|  null|
|   �| null| null|                null|  null|
+----+-----+-----+--------------------+------+

Instead of the correct results below:

+----+-----+-----+--------------------+-----+
|year| make|model|             comment|blank|
+----+-----+-----+--------------------+-----+
|2012|Tesla|    S|          No comment|     |
|1997| Ford| E350|Go get one now th...|     |
|2015|Chevy| Volt|                null| null|
+----+-----+-----+--------------------+-----+

Issue Links

• links to: [Github] Pull Request #11016 (HyukjinKwon)