[SPARK-3103] Fix UTF8 encoding in PySpark saveAsTextFile().

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.2, 1.1.0
• Fix Version/s: 1.1.0
• Component/s: PySpark
• Labels: starter
• Target Version/s: 1.1.0
• Assignee: Davies Liu
• Reporter: Josh Rosen
• Created: 18/Aug/14 17:26
• Updated: 18/Aug/14 20:59
• Resolved: 18/Aug/14 20:59

Description

This is a follow-up JIRA for https://github.com/apache/spark/pull/1914, where Ahir and Davies identified a bug in Python JsonRDD when trying to encode non-ASCII strings into unicode.

The same underlying issue affects saveAsTextFile, so we should apply the same fix there, too, and search for any other code that needs to be updated (and maybe refactor this out into a utility function).

Issue Links

• links to: [Github] Pull Request #2018 (davies)