[SPARK-44142] Utility to convert python types to spark types compares Python "type" object rather than user's "tpe" for categorical data types

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.3.3, 3.4.1, 3.5.0
• Component/s: PySpark
• Assignee: Ted Chester Jenks
• Reporter: Ted Chester Jenks
• Created: 22/Jun/23 12:05
• Updated: 23/Jun/23 00:11
• Resolved: 23/Jun/23 00:09

Description

In the typehints utility that converts python types to spark types, the line:

    # categorical types
    elif isinstance(tpe, CategoricalDtype) or (isinstance(tpe, str) and type == "category"):
        return types.LongType() 

uses Python's 'type' keyword in the comparison. Hence, it will always be false. Here, the user's type is actually stored in the variable 'tpe'.

 

 

See line here.