[SPARK-39130] How do I read parquet with python object

• Type: Question
• Status: Resolved
• Priority: Trivial
• Resolution: Won't Do
• Affects Version/s: 2.4.5
• Component/s: PySpark

Environment

pyspark2.4.5

Language

python

• Assignee: Unassigned
• Reporter: Ben Wan
• Created: 09/May/22 13:17
• Updated: 09/May/22 13:24
• Resolved: 09/May/22 13:24

Description

python:

 

import pandas as pd

a=pd.DataFrame([[1,[2.3,1.2]]],columns=['a','b'])

a.to_parquet('a.parquet')

 

pyspark:

 

d2 = spark.read.parquet('a.parquet')

 

will return error:

An error was encountered: An error occurred while calling o277.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 14 in stage 9.0 failed 4 times, most recent failure: Lost task 14.2 in stage 9.0 (TID 63, 10.169.0.196, executor 15): java.lang.IllegalArgumentException: Illegal Capacity: -221

how can I fix it?

Thanks.