[BEAM-1050] PipelineResult.State is not set to FAILED when a streaming job fails

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Component/s: runner-spark
• Assignee: Stas Levin
• Reporter: Stas Levin
• Created: 27/Nov/16 12:06
• Updated: 16/May/20 13:22
• Resolved: 05/Dec/16 11:28

Description

In case of failure, SteamingContext#awaitTerminationOrTimeout and SteamingContext#awaitTermination will both throw an exception, and so state = State.DONE will not be executed in the code block below.

In addition, it would probably make sense to set state = State.FAILED in cases where an exception takes place.

if (isStreamingPipeline()) {
      // stop streaming context
      if (timeout > 0) {
        jssc.awaitTerminationOrTimeout(timeout);
      } else {
        jssc.awaitTermination();
      }
      // stop streaming context gracefully, so checkpointing (and other computations) get to
      // finish before shutdown.
      jssc.stop(false, gracefully);
    }
    state = State.DONE;

Issue Links

• links to: GitHub Pull Request #1438