[SPARK-11772] DataFrame.show() fails with non-ASCII strings

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.5.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Greg Baker
• Created: 17/Nov/15 05:43
• Updated: 18/Nov/15 18:49
• Resolved: 18/Nov/15 18:49

Description

When given a non-ASCII string (in pyspark at least), the DataFrame.show() method fails.

df = sqlContext.createDataFrame([[u'ab\u0255']])
df.show()

Results in:

15/11/16 21:36:54 INFO DAGScheduler: ResultStage 1 (showString at NativeMethodAccessorImpl.java:-2) finished in 0.148 s
15/11/16 21:36:54 INFO DAGScheduler: Job 1 finished: showString at NativeMethodAccessorImpl.java:-2, took 0.192634 s
Traceback (most recent call last):
  File ".../show_bug.py", line 8, in <module>
    df.show()
  File ".../spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 256, in show
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0255' in position 21: ordinal not in range(128)
15/11/16 21:36:54 INFO SparkContext: Invoking stop() from shutdown hook