[SPARK-19912] String literals are not escaped while performing Hive metastore level partition pruning

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1, 2.2.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: SQL
• Labels: correctness
• Target Version/s: 2.2.0
• Assignee: Dongjoon Hyun
• Reporter: Cheng Lian
• Created: 10/Mar/17 23:17
• Updated: 21/Mar/17 04:18
• Resolved: 21/Mar/17 04:18

Description

Shim_v0_13.convertFilters() doesn't escape string literals while generating Hive style partition predicates.

The following SQL-injection-like test case illustrates this issue:

  test("SPARK-19912") {
    withTable("spark_19912") {
      Seq(
        (1, "p1", "q1"),
        (2, "p1\" and q=\"q1", "q2")
      ).toDF("a", "p", "q").write.partitionBy("p", "q").saveAsTable("spark_19912")

      checkAnswer(
        spark.table("foo").filter($"p" === "p1\" and q = \"q1").select($"a"),
        Row(2)
      )
    }
  }

The above test case fails like this:

[info] - spark_19912 *** FAILED *** (13 seconds, 74 milliseconds)
[info]   Results do not match for query:
[info]   Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
[info]   Timezone Env:
[info]
[info]   == Parsed Logical Plan ==
[info]   'Project [unresolvedalias('a, None)]
[info]   +- Filter (p#27 = p1" and q = "q1)
[info]      +- SubqueryAlias spark_19912
[info]         +- Relation[a#26,p#27,q#28] parquet
[info]
[info]   == Analyzed Logical Plan ==
[info]   a: int
[info]   Project [a#26]
[info]   +- Filter (p#27 = p1" and q = "q1)
[info]      +- SubqueryAlias spark_19912
[info]         +- Relation[a#26,p#27,q#28] parquet
[info]
[info]   == Optimized Logical Plan ==
[info]   Project [a#26]
[info]   +- Filter (isnotnull(p#27) && (p#27 = p1" and q = "q1))
[info]      +- Relation[a#26,p#27,q#28] parquet
[info]
[info]   == Physical Plan ==
[info]   *Project [a#26]
[info]   +- *FileScan parquet default.spark_19912[a#26,p#27,q#28] Batched: true, Format: Parquet, Location: PrunedInMemoryFileIndex[], PartitionCount: 0, PartitionFilters: [isnotnull(p#27), (p#27 = p1" and q = "q1)], PushedFilters: [], ReadSchema: struct<a:int>
[info]   == Results ==
[info]
[info]   == Results ==
[info]   !== Correct Answer - 1 ==   == Spark Answer - 0 ==
[info]    struct<>                   struct<>
[info]   ![2]

Issue Links

• relates to: HIVE-2943 Escaping quote in partition value encodes both backslash and quote in the partition name
• links to: [Github] Pull Request #17266 (dongjoon-hyun)
• links to: [Github] Pull Request #17271 (dongjoon-hyun)