[SPARK-23673] PySpark dayofweek does not conform with ISO 8601

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: PySpark, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Ivan SPM
• Created: 13/Mar/18 19:11
• Updated: 08/Oct/19 05:43
• Resolved: 08/Oct/19 05:43

Description

The new function dayofweek, that returns 1 = Sunday and 7 for Saturday, does not conform with ISO 8601, that states that the first day of the week is Monday, so 1 = Monday and 7 = Sunday. This behavior is also different from Pandas, that uses 0 = Monday and 6 = Sunday, but pandas is at least consistent with the ordering of ISO 8601.

https://en.wikipedia.org/wiki/ISO_8601#Week_dates

(Also reported by Antonio Pedro Vieira ([e-mail]))