[SPARK-47977] DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, SQL
• Labels: pull-request-available
• Assignee: Vitalii Li
• Reporter: Vitalii Li
• Created: 24/Apr/24 21:24
• Updated: 03/Jun/24 18:49
• Resolved: 03/Jun/24 18:49

Description

https://github.com/apache/spark/pull/44263 converted `IllegalStateException` to `InternalError` when a unit for DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd is not from a sanctioned list.

Originally incorrect unit should have been caught in parser before an expression is constructed. However, PySpark introduced PythonSQLUtils that creates expressions without validation. Since then the `INTERNAL_ERROR` is incorrect error class and should be converted to execution error with correct error class instead.

Issue Links

• links to: GitHub Pull Request #46210