[SPARK-34909] conv() does not convert negative inputs to unsigned correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 2.4.8, 3.0.3, 3.1.2, 3.2.0
• Component/s: SQL
• Labels: correctness
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 30/Mar/21 17:38
• Updated: 01/Jun/21 23:00
• Resolved: 31/Mar/21 05:02

Description

scala> spark.sql("select conv('-10', 11, 7)").show(20, 150)
+-----------------------+
|       conv(-10, 11, 7)|
+-----------------------+
|4501202152252313413456|
+-----------------------+
scala> spark.sql("select hex(conv('-10', 11, 7))").show(20, 150)
+----------------------------------------------+
|                         hex(conv(-10, 11, 7))|
+----------------------------------------------+
|3435303132303231353232353233313334313334353600|
+----------------------------------------------+

The correct result is 45012021522523134134555. The above output has an incorrect second-to-last digit (6 instead of 5) and the last digit is a non-printing character the null byte.

I tracked the bug down to NumberConverter.unsignedLongDiv returning incorrect results. I tried replacing with java.lang.Long.divideUnsigned and that fixed it.

Issue Links

• links to: [Github] Pull Request #32006 (timarmstrong)
• links to: [Github] Pull Request #32006 (timarmstrong)