[SPARK-24175] improve the Spark 2.4 migration guide

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Component/s: Documentation, SQL
• Assignee: Unassigned
• Reporter: Wenchen Fan
• Created: 04/May/18 02:21
• Updated: 18/Jun/19 02:00
• Resolved: 18/Jun/19 02:00

Description

The current Spark 2.4 migration guide is not well phrased. We should
1. State the before behavior
2. State the after behavior
3. Add a concrete example with code to illustrate.

For example:

Since Spark 2.4, Spark compares a DATE type with a TIMESTAMP type after promotes both sides to TIMESTAMP. To set `false` to `spark.sql.hive.compareDateTimestampInTimestamp` restores the previous behavior. This option will be removed in Spark 3.0.

-->

In version 2.3 and earlier, Spark implicitly casts a timestamp column to date type when comparing with a date column. In version 2.4 and later, Spark casts the date column to timestamp type instead. As an example, "xxx" would result in ".." in Spark 2.3, and in Spark 2.4, the result would be "..."