Tibor Digana added a comment - 20/Nov/16 19:06 Piotr Turski Can you check it out with Version 2.19.2-SNAPSHOT? Method #rethrowAnyTestMechanismFailures() changed and I think it should not throw NPE.

Piotr Turski added a comment - 20/Nov/16 22:10 - edited still a bug but a bit different output. i took the project that reproduces the issue, added: <pluginRepositories> <pluginRepository> <id> apache.snapshots </id> <url> http://repository.apache.org/snapshots/ </url> </pluginRepository> </pluginRepositories> and changed version to <version>2.19.2-SNAPSHOT</version> and after running tests: [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building playground 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ playground --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /xxxxxxxxx/zohhak_parallel_bug/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ playground --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ playground --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /xxxxxxxx/zohhak_parallel_bug/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ playground --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.19.2-SNAPSHOT:test (default-test) @ playground --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running testMethod [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in testMethod [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.240s [INFO] Finished at: Sun Nov 20 23:01:58 CET 2016 [INFO] Final Memory: 11M/303M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.2-SNAPSHOT:test (default-test) on project playground: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.19.2-SNAPSHOT:test failed: There was an error in the forked process [ERROR] org.apache.maven.surefire.testset.TestSetFailedException: Test mechanism :: null [ERROR] at org.apache.maven.surefire.common.junit4.JUnit4RunListener.rethrowAnyTestMechanismFailures(JUnit4RunListener.java:210) [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:138) [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157) [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:300) [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:251) [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:130) [ERROR] Caused by: java.lang.NullPointerException [ERROR] at org.apache.maven.surefire.junitcore.ConcurrentRunListener.testStarting(ConcurrentRunListener.java:134) [ERROR] at org.apache.maven.surefire.common.junit4.JUnit4RunListener.testStarted(JUnit4RunListener.java:93) [ERROR] at org.junit.runner.notification.SynchronizedRunListener.testStarted(SynchronizedRunListener.java:49) [ERROR] at org.junit.runner.notification.RunNotifier$3.notifyListener(RunNotifier.java:121) [ERROR] at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) [ERROR] at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:118) [ERROR] at org.apache.maven.surefire.common.junit4.Notifier.fireTestStarted(Notifier.java:100) [ERROR] at org.junit.internal.runners.model.EachTestNotifier.fireTestStarted(EachTestNotifier.java:42) [ERROR] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:323) [ERROR] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [ERROR] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [ERROR] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [ERROR] at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393) [ERROR] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [ERROR] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [ERROR] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [ERROR] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [ERROR] at java.lang.Thread.run(Thread.java:745) [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException Process finished with exit code 1

Tibor Digana added a comment - 21/Nov/16 03:59 Piotr Turski I guess the custom runner does not satisfy the interface of Description . We have to extract method and class from it which is important to run method and/or class in parallel. Try to attach Surefire source, run it in debug mode and debug classes ConcurrentRunListener and JUnitCoreRunListener . There is no TestSet for Description . Is the runner mutating the description? You should debug the method fillTestCountMap in class JUnitCoreRunListener . Either the class name does not come to the map classMethodCounts or the description of class was mutated before ConcurrentRunListener#testStarting() . The JUnit description must be parsed successfully in JUnit4RunListener#testStarted( Description description ) .

Piotr Turski added a comment - 24/Nov/16 00:29 any links to documentation on what's the interface of `Description`? what exactly should be satisfied? if it runs correctly in non-parallel model, shouldn't it also work in parallel mode? Is the runner mutating the description? mutating? runner creates it. it creates parameterized tests. is there a simple way to run surefire programatically inside IDE? or how does surefire-team run tests for the surefire itself?

Tibor Digana added a comment - 25/Nov/16 05:13 - edited is there a simple way to run surefire programatically inside IDE? or how does surefire-team run tests for the surefire itself? Here is a hint in documentation: http://maven.apache.org/surefire/maven-surefire-plugin/examples/debugging.html It would be more simple for you to debug tests within Maven process and not in forked JVM. To do so add <forkCount>0</forkCount> in Surefire plugin configuration and debug afterwards.

Tibor Digana added a comment - 02/Dec/16 01:53 https://git-wip-us.apache.org/repos/asf?p=maven-surefire.git;a=commit;h=c2a7d4b7f28b0fb5456b9871fd87cbac5c70cbd3

Hudson added a comment - 02/Dec/16 02:49 SUCCESS: Integrated in Jenkins build maven-surefire #1649 (See https://builds.apache.org/job/maven-surefire/1649/ ) SUREFIRE-1305 surefire fails on parallel tests when newline character (tibor17: rev c2a7d4b7f28b0fb5456b9871fd87cbac5c70cbd3) (edit) surefire-providers/common-junit4/src/main/java/org/apache/maven/surefire/common/junit4/JUnit4RunListener.java

Tibor Digana added a comment - 03/Dec/16 21:30 https://git-wip-us.apache.org/repos/asf?p=maven-surefire.git;a=commit;h=d7bf6ce49043a3301494fdd26c538afad4e1938d

Hudson added a comment - 03/Dec/16 23:27 SUCCESS: Integrated in Jenkins build maven-surefire #1652 (See https://builds.apache.org/job/maven-surefire/1652/ ) SUREFIRE-1305 surefire fails on parallel tests when newline character (tibor17: rev d7bf6ce49043a3301494fdd26c538afad4e1938d) (edit) surefire-providers/common-junit4/src/main/java/org/apache/maven/surefire/common/junit4/JUnit4RunListener.java (edit) surefire-providers/common-junit4/src/main/java/org/apache/maven/surefire/common/junit4/JUnit4StackTraceWriter.java (add) surefire-api/src/main/java/org/apache/maven/surefire/util/internal/TestClassMethodNameUtils.java