[SPARK-17808] BinaryType fails in Python 3 due to outdated Pyrolite

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.1
• Fix Version/s: 2.0.2, 2.1.0
• Component/s: PySpark

Environment

spark-2.0.1-bin-hadoop2.7 with Python 3.4.3 on Ubuntu 14.04.4 LTS

• Assignee: Bryan Cutler
• Reporter: Pete Fein
• Created: 06/Oct/16 17:03
• Updated: 14/Jul/19 05:22
• Resolved: 11/Oct/16 06:30

Description

Attempting to create a DataFrame using a BinaryType field fails under Python 3 because the underlying Pyrolite library is out of date. Spark appears to be using Pyrolite 4.9; this issue was fixed in Pyrolite 4.12. See original bug report and patch

Test case & output attached. I'm just a Python guy, not really sure how to build Spark / do classpath magic to test if this works correctly with updated Pyrolite.

Issue Links

• links to: [Github] Pull Request #15386 (BryanCutler)
• links to: GitHub Pull Request #25143