[SPARK-24950] scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.3, 2.2.2, 2.3.1, 2.4.0
• Fix Version/s: 2.1.4, 2.2.3, 2.3.2, 2.4.0
• Component/s: Build, Tests
• Assignee: Chris Martin
• Reporter: Shane Knapp
• Created: 27/Jul/18 18:20
• Updated: 23/Oct/19 22:28
• Resolved: 09/Aug/18 22:42

Description

during my travails to port the spark builds to run on ubuntu 16.04LTS, i have encountered a strange and apparently java version-specific failure on one specific unit test.

the failure is here:

https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6-ubuntu-test/868/testReport/junit/org.apache.spark.sql.catalyst.util/DateTimeUtilsSuite/daysToMillis_and_millisToDays/

the java version on this worker is:

sknapp@ubuntu-testing:~$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

however, when i run this exact build on the other ubuntu workers, it passes.  they systems are set up (for the most part) identically except for the java version:

sknapp@amp-jenkins-staging-worker-02:~$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

there are some minor kernel and other package differences on these ubuntu workers, but nothing that (in my opinion) would affect this test.  i am willing to help investigate this, however.

the test also passes on the centos 6.9 workers, which have the following java version installed:

[sknapp@amp-jenkins-worker-05 ~]$ java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)my guess is that either:

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

or

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala

is doing something wrong.  i am not a scala expert by any means, so i'd really like some help in trying to un-block the project to port the builds to ubuntu.

Issue Links

• is related to: SPARK-29578 JDK 1.8.0_232 timezone updates cause "Kwajalein" test failures again
• links to: [Github] Pull Request #21901 (d80tb7)
• links to: [Github] Pull Request #21903 (srowen)