Major Description
rg.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey.
junit.framework.AssertionFailedError at junit.framework.Assert.fail(Assert.java:48) at junit.framework.Assert.assertTrue(Assert.java:20) at junit.framework.Assert.assertFalse(Assert.java:34) at junit.framework.Assert.assertFalse(Assert.java:41) at org.apache.oozie.action.hadoop.PigTestCase.testPig_withNullExternalID(PigTestCase.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:24) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)
We've seen this for months but has been really tricky to track down. The output of the test has this:
Run pig script using PigRunner.run() for Pig version 0.8+ Apache Pig version 0.12.0-cdh5.5.0-SNAPSHOT (rexported) compiled Sep 30 2015, 22:11:56 Hadoop Job IDs executed by Pig: job_20151001151153510_0001 Run pig script using PigRunner.run() for Pig version 0.8+
Compared with a successful run of the test, the difference is that it shows a Hadoop Job ID. This test is supposed to make Pig fail and not run any jobs, which can be confirmed in the test output, yet there's a Hadoop Job ID here!
It turns out that PigRunner eventually gets to PigStats, which is a singleton. And because we're running all of the Pig tests in the same JVM, PigStats is actually leaking through the tests. It doesn't seem to be a problem most of the time because it must be getting overwritten at each run, however, because testPig_withNullExternalID is purposefully failing the Pig job, it's not, and we end up with the PigStats from the previous test or a non-initialized PigStats depending on the order of the tests. If it's a previous PigStats, then PigMain will write the Hadoop Job ID to a new file for the testPig_withNullExternalID test, which will then fail because it's not expecting this file.
Unfortunately, there isn't a clean way to reset PigStats. Pig v 0.9 has a set method which we can pass null to in order to reset the PigStats. However, this was changed in Pig v 0.13 to the start method. In either case, they're both package private, so we have to usereflection to find the existing method for whichever version of Pig we're using and make it public. We should do this after every Pig test, even if they don't all need it just to make sure that we're starting with a fresh Pig each time as unit tests are supposed to start fresh anyway.