[BEAM-3261] Apex runner does not detect pipeline failure

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: Missing
• Component/s: runner-apex
• Assignee: Unassigned
• Reporter: Eugene Kirpichov
• Created: 27/Nov/17 22:33
• Updated: 13/Jan/22 21:52
• Resolved: 15/May/21 17:57

Description

I was looking at https://github.com/apache/beam/pull/4074/files and asked myself "Don't we already have a ValidatesRunner test for this?"

Turns out we do: https://github.com/apache/beam/blob/3b79b6298e84711528b5ad1302200cb8acbac07e/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L793

I ran this test with TestApexRunner, and I observed the same exception as in https://stackoverflow.com/questions/46982827/error-when-using-side-input-withsideinputs-method-not-accepting-kv-type-as-inp?noredirect=1#comment81040223_46982827 , however the test passed.

It seems that it is passing because ApexRunnerResult.waitUntilFinish() detects only PAssert assertion errors, but not any other errors: https://github.com/apache/beam/blob/3b79b6298e84711528b5ad1302200cb8acbac07e/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunnerResult.java#L62

This seems very problematic, as it means that 1) the status of Apex ValidatesRunner tests cannot be trusted: if they fail in any other way other than a PAssert error, the failure will be undetected, and 2) in general, a user can not trust a successful pipeline.run() from Apex runner.

For tests in particular, some other TestXRunner's guard against such a failure mode by verifying not only that there were no assertion failures, but also that all assertions succeeded - using metrics: e.g. https://github.com/apache/beam/blob/3b79b6298e84711528b5ad1302200cb8acbac07e/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TestDataflowRunner.java#L248 . Fixing this would be optimal, but meanwhile, the runner should at least fail the pipeline in case of error.

Issue Links

• is related to: BEAM-2004 Verify PAssert execution in TestApexRunner
• links to: GitHub Pull Request #5780