[SPARK-33445] Can't parse decimal type from csv file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.1.3, 2.2.3, 2.3.4, 2.4.7
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Punit Shah
• Created: 13/Nov/20 14:47
• Updated: 12/Dec/22 18:11
• Resolved: 19/Nov/20 00:45

Description

The attached file is a one column csv file containing decimals.

Execute: mydf2 = spark_session.read.csv("tsd.csv", header=True, inferSchema=True)

Then invoking mydf2.schema will result in error:

ValueError: Could not parse datatype: decimal(6,-7)

Issue Links

• is related to: SPARK-26645 CSV infer schema bug infers decimal(9,-1)