[SPARK-3681] Failed to serialized ArrayType or MapType after accessing them in Python

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: PySpark
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 24/Sep/14 21:12
• Updated: 27/Sep/14 19:21
• Resolved: 27/Sep/14 19:21

Description

files_schema_rdd.map(lambda x: x.files).take(1)

Also it will lose the schema after iterate an ArrayType.

files_schema_rdd.map(lambda x: [f.batch for f in x.files]).take(1)

Issue Links

• links to: [Github] Pull Request #2526 (davies)