[SPARK-40791] The semantics of `F` in `DateTimeFormatter` have changed

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yang Jie
• Created: 13/Oct/22 12:04
• Updated: 18/Jun/23 08:36

Description

val createSql =
  """
    |create temporary view v as select col from values
    | (timestamp '1582-06-01 11:33:33.123UTC+080000'),
    | (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
    | (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
    | (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
    | (timestamp '2018-11-17 13:33:33.123Z'),
    | (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
    | (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') t(col)
    | """.stripMargin
sql(createSql)
withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> false.toString) {
  val rows = sql("select col, date_format(col, 'F') from v").collect()
  // scalastyle:off
  rows.foreach(println)
} 

 

Before Java 19, the result is 

 

[1582-05-31 19:40:35.123,3]
[1969-12-31 15:00:00.0,3]
[1970-12-31 04:59:59.999,3]
[1996-03-31 07:03:33.123,3]
[2018-11-17 05:33:33.123,3]
[2019-12-31 09:33:33.123,3]
[2100-01-01 01:33:33.123,1] 

Java 19

 

[1582-05-31 19:40:35.123,5]
[1969-12-31 15:00:00.0,5]
[1970-12-31 04:59:59.999,5]
[1996-03-31 07:03:33.123,5]
[2018-11-17 05:33:33.123,3]
[2019-12-31 09:33:33.123,5]
[2100-01-01 01:33:33.123,1]