[SPARK-48248] Fix nested array to respect legacy conf of inferArrayTypeFromFirstElement

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0, 3.5.2, 3.4.4
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 13/May/24 02:39
• Updated: 22/Nov/24 05:01
• Resolved: 13/May/24 08:18

Description

>>> spark.conf.set("spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled", True)
>>> spark.createDataFrame([[[[1, "a"]]]])
DataFrame[_1: array<array<string>>]

should infer it as an integer of array

Issue Links

• links to: GitHub Pull Request #46548
• links to: GitHub Pull Request #48931