[SPARK-26016] Document that UTF-8 is required in text data source

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Java API
• Assignee: Sean R. Owen
• Reporter: Chris Caspanello
• Created: 12/Nov/18 14:19
• Updated: 12/Dec/22 18:10
• Resolved: 04/Mar/19 23:05

Description

Attached you will find a project with unit tests showing the issue at hand.

If I read in a ISO-8859-1 encoded file and simply write out what was read; the contents in the part file matches what was read.  Which is great.

However, the second I use a map / mapPartitions function it looks like the encoding is not correct.  In addition a simple collectAsList and writing that list of strings to a file does not work either.  I don't think I'm doing anything wrong.  Can someone please investigate?  I think this is a bug.

Issue Links

• links to: GitHub Pull Request #23962