Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39839

Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0, 3.2.0, 3.3.0
    • 3.3.1, 3.2.3, 3.4.0
    • SQL
    • None

    Description

      The UnsafeRow structural integrity check in UnsafeRowUtils.validateStructuralIntegrity is added in Spark 3.1.0. It’s supposed to validate that a given UnsafeRow conforms to the format that the UnsafeRowWriter would have produced.

      Currently the check expects all fields that are marked as null should also have its field (i.e. the fixed-length part) set to all zeros. It needs to be updated to handle a special case for variable-length Decimals, where the UnsafeRowWriter may mark a field as null but also leave the fixed-length part of the field as OffsetAndSize(offset=current_offset, size=0). This may happen when the Decimal being written is either a real null or has overflowed the specified precision.

      Logic in UnsafeRowWriter:

      in general:

        public void setNullAt(int ordinal) {
          BitSetMethods.set(getBuffer(), startingOffset, ordinal); // set null bit
          write(ordinal, 0L);                                      // also zero out the fixed-length field
        } 

      special case for DecimalType:

            // Make sure Decimal object has the same scale as DecimalType.
            // Note that we may pass in null Decimal object to set null for it.
            if (input == null || !input.changePrecision(precision, scale)) {
              BitSetMethods.set(getBuffer(), startingOffset, ordinal); // set null bit
              // keep the offset for future update
              setOffsetAndSize(ordinal, 0);                            // doesn't zero out the fixed-length field
            } 

      The special case is introduced to allow all {{DecimalType}}s (including both fixed-length and variable-length ones) to be mutable – thus need to leave space for the variable-length field even if it’s currently null.

      Note that this special case in UnsafeRowWriter has been there since Spark 1.6.0, where as the integrity check was added in Spark 3.1.0. The check was originally added for Structured Streaming’s checkpoint evolution validation, so that a newer version of Spark can check whether or not an older checkpoint file for Structured Streaming queries can be supported, and/or if the contents of the checkpoint file is corrupted.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rednaxelafx Kris Mok
            rednaxelafx Kris Mok
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment