After debugging further, I learned that Join_1 was failing because of TezSession reuse. By tearing down TezSession after each test, All the e2e tests pass now.
The new patch includes the following changes-
- Makes the e2e tests more deterministic.
- Makes reuse of TezSession configurable via a property pig.tez.session.reuse.
- Sets this property to false in all the e2e tests except LoaderStorer_1. In this test, TezSession must be reused since a jar is registered as local resource by a DAG and it's reused by another DAG.
I will commit this patch unless anyone objects.