[SPARK-17804] Pandas dtypes are not correctly inferred by pyspark

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: PySpark
• Labels: bulk-closed
• External issue URL: http://stackoverflow.com/questions/39888188/
• Assignee: Unassigned
• Reporter: Georg Heiler
• Created: 06/Oct/16 08:06
• Updated: 12/Dec/22 18:10
• Resolved: 21/May/19 04:14

Description

As mentioned on SF http://stackoverflow.com/questions/39888188/pyspark-type-error-on-reading-a-pandas-dataframe/39889639?noredirect=1#comment67065964_39889639 pyspark has a problem correctly identifying the dtypes.

Issue Links

• relates to: SPARK-20791 Use Apache Arrow to Improve Spark createDataFrame from Pandas.DataFrame