Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24175

improve the Spark 2.4 migration guide

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: Documentation, SQL
    • Labels:
      None

      Description

      The current Spark 2.4 migration guide is not well phrased. We should
      1. State the before behavior
      2. State the after behavior
      3. Add a concrete example with code to illustrate.

      For example:

      Since Spark 2.4, Spark compares a DATE type with a TIMESTAMP type after promotes both sides to TIMESTAMP. To set `false` to `spark.sql.hive.compareDateTimestampInTimestamp` restores the previous behavior. This option will be removed in Spark 3.0.

      -->

      In version 2.3 and earlier, Spark implicitly casts a timestamp column to date type when comparing with a date column. In version 2.4 and later, Spark casts the date column to timestamp type instead. As an example, "xxx" would result in ".." in Spark 2.3, and in Spark 2.4, the result would be "..."

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cloud_fan Wenchen Fan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: