Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25519

ArrayRemove function may return incorrect result when right expression is implicitly downcasted.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      In ArrayRemove, we currently cast the right hand side expression to match the element type of the left hand side Array. This may result in down casting and may return wrong result or questionable result.

      spark-sql> select array_remove(array(1,2,3), 1.23D)
               > ;
      [2,3]
      Time taken: 2.57 seconds, Fetched 1 row(s)
      spark-sql> select array_remove(array(1,2,3), 'foo')
               > ;
      NULL
      

      We should safely coerce both left and right hand side expressions.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dkbiswal Dilip Biswal
                Reporter:
                dkbiswal Dilip Biswal
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: