[SPARK-23523] Incorrect result caused by the rule OptimizeMetadataOnlyQuery

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.1.2, 2.2.1, 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: SQL
• Labels: correctness
• Assignee: Xiao Li
• Reporter: Xiao Li
• Created: 27/Feb/18 05:09
• Updated: 02/Mar/20 19:53
• Resolved: 27/Feb/18 16:45

Description

 val tablePath = new File(s"${path.getCanonicalPath}/cOl3=c/cOl1=a/cOl5=e")
 Seq(("a", "b", "c", "d", "e")).toDF("cOl1", "cOl2", "cOl3", "cOl4", "cOl5")
 .write.json(tablePath.getCanonicalPath)
 val df = spark.read.json(path.getCanonicalPath).select("CoL1", "CoL5", "CoL3").distinct()
 df.show()

This returns a wrong result
[c,e,a]

Issue Links

• is caused by: SPARK-15752 Optimize metadata only query that has an aggregate whose children are deterministic project or filter operators
• links to: [Github] Pull Request #20684 (gatorsmile)
• links to: [Github] Pull Request #20693 (jiangxb1987)
• links to: [Github] Pull Request #20763 (gatorsmile)