It's an un-fixable behavior change to fill the gap between SimpleDateFormat and DateTimeFormater and backward-compatibility for different JDKs.
A lot of effort has been made to prove this at https://github.com/apache/spark/pull/28674
The existing behavior itself in 2.4 is confusing, e.g.
the 'u' here seems not to go to the Monday of the first week in week-based form or the first day of the year in non-week-based form but go to the Monday of the second week in week-based form.
I think we don't need to introduce all the weird behavior from Java.