Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34909

conv() does not convert negative inputs to unsigned correctly

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      scala> spark.sql("select conv('-10', 11, 7)").show(20, 150)
      +-----------------------+
      |       conv(-10, 11, 7)|
      +-----------------------+
      |4501202152252313413456|
      +-----------------------+
      scala> spark.sql("select hex(conv('-10', 11, 7))").show(20, 150)
      +----------------------------------------------+
      |                         hex(conv(-10, 11, 7))|
      +----------------------------------------------+
      |3435303132303231353232353233313334313334353600|
      +----------------------------------------------+
      

      The correct result is 45012021522523134134555. The above output has an incorrect second-to-last digit (6 instead of 5) and the last digit is a non-printing character the null byte.

      I tracked the bug down to NumberConverter.unsignedLongDiv returning incorrect results. I tried replacing with java.lang.Long.divideUnsigned and that fixed it.

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment