[SPARK-32559] Fix the trim logic in UTF8String.toInt/toLong did't handle Chinese characters correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.1
• Component/s: SQL
• Labels: correctness
• Assignee: EdisonWang
• Reporter: EdisonWang
• Created: 06/Aug/20 09:02
• Updated: 12/Jul/23 09:47
• Resolved: 09/Aug/20 19:13

Description

The trim logic in Cast expression introduced in https://github.com/apache/spark/pull/26622 will trim chinese characters unexpectly.

For example,  sql  select cast("1中文" as float) gives 1 instead of null

 

Issue Links

• is caused by: SPARK-28023 Trim the string when cast string type to other types
• is related to: SPARK-44383 Fix the trim logic did't handle ASCII control characters correctly
• links to: [Github] Pull Request #29375 (WangGuangxin)
• links to: [Github] Pull Request #29393 (WangGuangxin)