Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33098

Explicit or implicit casts involving partition columns can sometimes result in a MetaException.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      The following cases throw
      MetaException(message:Filtering is supported only on partition keys of type string)

      sql("create table test (a int) partitioned by (b int) stored as parquet")
      sql("insert into test values (1, 1), (1, 2), (2, 2)")
      
      // These throw MetaExceptions
      sql("select * from test where b in ('2')").show(false)
      sql("select * from test where cast(b as string) = '2'").show(false)
      sql("select * from test where cast(b as string) in ('2')").show(false)
      sql("select * from test where cast(b as string) in (2)").show(false)
      sql("select cast(b as string) as b from test where b in ('2')").show(false)
      sql("select cast(b as string) as b from test").filter("b = '2'").show(false) // [1]
      sql("select cast(b as string) as b from test").filter("b in (2)").show(false) // [2]
      sql("select cast(b as string) as b from test").filter("b in ('2')").show(false)
      sql("select * from test where cast(b as string) > '1'").show(false)
      sql("select cast(b as string) b from test").filter("b > '1'").show(false) // [3]
      
      // [1] but not sql("select cast(b as string) as b from test where b = '2'").show(false)
      // [2] but not sql("select cast(b as string) as b from test where b in (2)").show(false)
      // [3] but not sql("select cast(b as string) b from test where b > '1'").show(false)
      

      The message ("Filtering is supported only on partition keys of type string") is misleading. Filter is supported on integer columns, for example.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                bersprockets Bruce Robbins
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: