Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24934

Complex type and binary type in in-memory partition pruning does not work due to missing upper/lower bounds cases

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.3.1, 2.4.0
    • 2.3.2, 2.4.0
    • SQL

    Description

      For example, if array is used (where the lower and upper bounds for its column batch are null)), it looks wrongly filtering all data out:

      scala> import org.apache.spark.sql.functions
      import org.apache.spark.sql.functions
      
      scala> val df = Seq(Array("a", "b"), Array("c", "d")).toDF("arrayCol")
      df: org.apache.spark.sql.DataFrame = [arrayCol: array<string>]
      
      scala> df.filter(df.col("arrayCol").eqNullSafe(functions.array(functions.lit("a"), functions.lit("b")))).show()
      +--------+
      |arrayCol|
      +--------+
      |  [a, b]|
      +--------+
      
      
      scala> df.cache().filter(df.col("arrayCol").eqNullSafe(functions.array(functions.lit("a"), functions.lit("b")))).show()
      +--------+
      |arrayCol|
      +--------+
      +--------+
      

      Attachments

        Activity

          People

            hyukjin.kwon Hyukjin Kwon
            hyukjin.kwon Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: