[FLINK-2850] Limit the types of jobs which can run in detached mode

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Assignee: Sachin Goel
• Reporter: Sachin Goel
• Created: 13/Oct/15 13:37
• Updated: 13/Nov/15 15:54
• Resolved: 13/Nov/15 15:54

Description

Currently, there are two types of programs which can be submitted.
1. Program interface
2. Interactive programs, whose main method is run.

For interactive mode,
In blocking mode, the programs can be executed without any issues since every execution is blocked, and its results can be made available at a further point in the program, as well as the Client is kept alive and is able to receive job outputs.
However, in detached mode, interactive programs which are any of the following three types of programs create issues:
1.

DataSet<T> dataset = ...
List<T> result = dataset.collect(); // accesses accumulators but the results of the inner call to execute aren't available

2.

env.execute().getAccumulatorResult(); // fails again because the results aren't availabe.

3.

env.execute();
env.execute(); // two executions. Will return two different job ids

These problems make the current yarn detached job mode fail, and will lead to issues when fixing FLINK-2797.