Details
-
Documentation
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.3.0
-
None
Description
Added details about the supported column input types, the column return type, behaviour on invalid input, supporting examples and clarifications to the datetime functions in `org.apache.spark.sql.functions` for Java/Scala.
These changes stemmed from confusion over behaviour of the `date_add` method. On first use I thought it would add the specified days to the input timestamp, but it also truncated (cast) the input timestamp to a date, loosing the time part.
Some examples:
- Noted that the week definition for `dayofweek` method starts on a Sunday
- Corrected documentation for methods such as `last_day` that only listed one type of input i.e. "date column" changed to "date, timestamp or string"
- Renamed the parameters of the `months_between` method to match those of the `datediff` method and to indicate which parameter is expected to be before then other chronologically
- `from_unixtime` documentation referenced the "given format" when there was no format parameter
- Documentation for `to_timestamp` methods detailed that a unix timestamp in seconds would be returned (implying 1521926327) when they would actually return the input cast to a timestamp type
Some observations:
- The first day of the week by the `dayofweek` method is a Sunday, but by the `weekofyear` method it is a Monday
- The `datediff` method returns a integer value, even with timestamp input, whereas the `months_between` method returns a double, which seems inconsistent
