Note: wording is borrowed from
SPARK-22395. Symptom is similar and I think that JIRA is well descriptive.
When converting R's DataFrame from/to Spark DataFrame using createDataFrame or collect, timestamp values behave to respect R system timezone instead of session timezone.
For example, let's say we use "America/Los_Angeles" as session timezone and have a timestamp value "1970-01-01 00:00:01" in the timezone. Btw, I'm in South Korea so R timezone would be "KST".
The timestamp value from current collect() will be the following:
As you can see, the value becomes "1970-01-01 17:00:01" because it respects R system timezone.