[SPARK-19571] tests are failing to run on Windows with another instance Derby error with Hadoop 2.6.5

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: SparkR, SQL
• Assignee: Hyukjin Kwon
• Reporter: Felix Cheung
• Created: 13/Feb/17 02:55
• Updated: 12/Dec/22 18:10
• Resolved: 14/Feb/17 19:02

Description

Between https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/build/751-master
https://github.com/apache/spark/commit/7a7ce272fe9a703f58b0180a9d2001ecb5c4b8db

And

https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/build/758-master
https://github.com/apache/spark/commit/c618ccdbe9ac103dfa3182346e2a14a1e7fca91a

Something is changed (not likely caused by R) such that tests running on Windows are consistently failing with

Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database C:\Users\appveyor\AppData\Local\Temp\1\spark-75266bb9-bd54-4ee2-ae54-2122d2c011e8\metastore.
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockOnDB(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)

Since we run appveyor only when there is R changes, it is a bit harder to track down which change specifically caused this.
We also can't run appveyor on branch-2.1, so it could also be broken there.

This could be a blocker, since it could fail tests for the R release.

Issue Links

• links to: [Github] Pull Request #16927 (HyukjinKwon)