[SPARK-42399] CONV() silently overflows returning wrong results

• Type: Bug
• Status: In Progress
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: SQL
• Labels: correctness pull-request-available
• Assignee: Unassigned
• Reporter: Serge Rielau
• Created: 10/Feb/23 19:03
• Updated: 28/Feb/24 00:17

Description

spark-sql> SELECT CONV(SUBSTRING('0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff', 3), 16, 10);

18446744073709551615

Time taken: 2.114 seconds, Fetched 1 row(s)

spark-sql> set spark.sql.ansi.enabled = true;

spark.sql.ansi.enabled true

Time taken: 0.068 seconds, Fetched 1 row(s)

spark-sql> SELECT CONV(SUBSTRING('0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff', 3), 16, 10);

18446744073709551615

Time taken: 0.05 seconds, Fetched 1 row(s)

In ANSI mode we should raise an error for sure.
In non ANSI either an error or a NULL maybe be acceptable.

Alternatively, of course, we could consider if we can support arbitrary domains since the result is a STRING again. 

Issue Links

• is related to: SPARK-42427 Conv should return an error if the internal conversion overflows
• links to: [Github] Pull Request #40040 (NarekDW)
• links to: GitHub Pull Request #40040