[SPARK-43240] df.describe() method may- return wrong result if the last RDD is RDD[UnsafeRow]

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.3.3
• Component/s: Spark Core
• Labels: correctness
• Assignee: Ke Jia
• Reporter: Ke Jia
• Created: 23/Apr/23 02:50
• Updated: 24/Nov/23 22:20
• Resolved: 26/Apr/23 09:25

Description

When calling the df.describe() method, the result  maybe wrong when the last RDD is RDD[UnsafeRow]. It is because the UnsafeRow will be released after the row is used.