[SPARK-17591] Fix/investigate the failure of tests in Scala On Windows

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: Build, DStreams, Spark Core, SQL
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 19/Sep/16 04:21
• Updated: 12/Dec/22 18:10
• Resolved: 08/Dec/16 15:21

Description

Tests run: 90, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.53 sec <<< FAILURE! - in org.apache.spark.JavaAPISuite
wholeTextFiles(org.apache.spark.JavaAPISuite)  Time elapsed: 0.313 sec  <<< FAILURE!
java.lang.AssertionError: 
expected:<spark is easy to use.
> but was:<null>
	at org.apache.spark.JavaAPISuite.wholeTextFiles(JavaAPISuite.java:1089)

Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.062 sec <<< FAILURE! - in org.apache.spark.launcher.SparkLauncherSuite
testChildProcLauncher(org.apache.spark.launcher.SparkLauncherSuite)  Time elapsed: 0.047 sec  <<< FAILURE!
java.lang.AssertionError: expected:<0> but was:<1>
	at org.apache.spark.launcher.SparkLauncherSuite.testChildProcLauncher(SparkLauncherSuite.java:177)

Tests run: 53, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.325 sec <<< FAILURE! - in org.apache.spark.streaming.JavaAPISuite
testCheckpointMasterRecovery(org.apache.spark.streaming.JavaAPISuite)  Time elapsed: 3.418 sec  <<< ERROR!
java.io.IOException: Failed to delete: C:\projects\spark\streaming\target\tmp\1474255953021-0
	at org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery(JavaAPISuite.java:1808)
Running org.apache.spark.streaming.JavaDurationSuite

Running org.apache.spark.streaming.JavaAPISuite
Tests run: 53, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.325 sec <<< FAILURE! - in org.apache.spark.streaming.JavaAPISuite
testCheckpointMasterRecovery(org.apache.spark.streaming.JavaAPISuite)  Time elapsed: 3.418 sec  <<< ERROR!
java.io.IOException: Failed to delete: C:\projects\spark\streaming\target\tmp\1474255953021-0
	at org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery(JavaAPISuite.java:1808)

Results :
Tests in error: 
  JavaAPISuite.testCheckpointMasterRecovery:1808 � IO Failed to delete: C:\proje...
Tests run: 74, Failures: 0, Errors: 1, Skipped: 0

The tests were aborted for unknown reason during SQL tests - BroadcastJoinSuite emitting the exceptions below continuously:

20:48:09.876 ERROR org.apache.spark.deploy.worker.ExecutorRunner: Error running executor
java.io.IOException: Cannot run program "C:\Progra~1\Java\jdk1.8.0\bin\java" (in directory "C:\projects\spark\work\app-20160918204809-0000\0"): CreateProcess error=206, The filename or extension is too long
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:167)
	at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
Caused by: java.io.IOException: CreateProcess error=206, The filename or extension is too long
	at java.lang.ProcessImpl.create(Native Method)
	at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)
	at java.lang.ProcessImpl.start(ProcessImpl.java:137)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 2 more

Here is the full log for the test - https://ci.appveyor.com/project/spark-test/spark/build/15-scala-tests

We may have to create sub-tasks if these are actual issues on Windows rather than just mistakes in tests.

I am willing to test this again after fixing some issues here in particular the last one.

I trigger the build by the comments below:

mvn -DskipTests -Phadoop-2.6 -Phive -Phive-thriftserver package
mvn -Phadoop-2.6 -Phive -Phive-thriftserver --fail-never test

Issue Links

• is related to: SPARK-18785 Fix/Investigate the test failures in Java/Scala on Windows

Sub-Tasks

• Fix wholeTextFiles test to pass on Windows in JavaAPISuite: Resolved
• ReceiverTracker should close checkpoint dir when stopped even if it was not started: Resolved
• Close recordwriter in SparkHadoopMapReduceWriter before committing: Resolved
• Fix all tests in ExecutorClassLoaderSuite to pass on Windows: Resolved
• Skip some test failures due to path length limitation and fix tests to pass on Windows: Resolved