Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41118

to_number/try_to_number throws NullPointerException when format is null

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.1, 3.4.0
    • 3.3.2, 3.4.0
    • SQL
    • None

    Description

      Example:

      spark-sql> SELECT to_number('454', null);
      [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
      org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
      	at org.apache.spark.SparkException$.internalError(SparkException.scala:88)
      	at org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:498)
      	at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
      	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
      ...
      Caused by: java.lang.NullPointerException
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormat$lzycompute(numberFormatExpressions.scala:72)
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormat(numberFormatExpressions.scala:72)
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormatter$lzycompute(numberFormatExpressions.scala:73)
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormatter(numberFormatExpressions.scala:73)
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.checkInputDataTypes(numberFormatExpressions.scala:81)
      

      Also:

      spark-sql> SELECT try_to_number('454', null);
      [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
      org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
      	at org.apache.spark.SparkException$.internalError(SparkException.scala:88)
      	at org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:498)
      	at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
      	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
      ...
      Caused by: java.lang.NullPointerException
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormat$lzycompute(numberFormatExpressions.scala:72)
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormat(numberFormatExpressions.scala:72)
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormatter$lzycompute(numberFormatExpressions.scala:73)
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormatter(numberFormatExpressions.scala:73)
      	at org.apache.spark.sql.catalyst.expressions.ToNumber.checkInputDataTypes(numberFormatExpressions.scala:81)
      	at org.apache.spark.sql.catalyst.expressions.TryToNumber.checkInputDataTypes(numberFormatExpressions.scala:146)
      

      Compare to to_binary and try_to_binary:

      spark-sql> SELECT to_binary('abc', null);
      NULL
      Time taken: 3.111 seconds, Fetched 1 row(s)
      spark-sql> SELECT try_to_binary('abc', null);
      NULL
      Time taken: 0.06 seconds, Fetched 1 row(s)
      spark-sql>
      

      Attachments

        Activity

          People

            bersprockets Bruce Robbins
            bersprockets Bruce Robbins
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: