[SPARK-36970] Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 11/Oct/21 03:37
• Updated: 12/Oct/21 17:14
• Resolved: 12/Oct/21 17:14

Description

The `date_format` function has some behavioral differences when using JDK 8 and JDK 17 as following:

the result of select date_format('2018-11-17 13:33:33.333', 'B') in datetime-formatting-invalid.sql with Java 8 is:

-- !query
select date_format('2018-11-17 13:33:33.333', 'B')
-- !query schema
struct<>
-- !query output
java.lang.IllegalArgumentException
Unknown pattern letter: B

and with Java 17 the result is:

- datetime-formatting-invalid.sql *** FAILED ***
  datetime-formatting-invalid.sql
  Expected "struct<[]>", but got "struct<[date_format(2018-11-17 13:33:33.333, B):string]>" Schema did not match for query #34
  select date_format('2018-11-17 13:33:33.333', 'B'): -- !query
  select date_format('2018-11-17 13:33:33.333', 'B')
  -- !query schema
  struct<date_format(2018-11-17 13:33:33.333, B):string>
  -- !query output
  in the afternoon (SQLQueryTestSuite.scala:469)

 
From the javadoc we can find that 'B' is used to represent `Pattern letters to output a day period` in Java 17.

Issue Links

• links to: [Github] Pull Request #34237 (LuciferYang)