[FLINK-24757] Yarn application is not terminated after the job finishes when submitting a yarn-per-job insert job with SQL client

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Fix Version/s: 1.20.0
• Component/s: Table SQL / Client
• Assignee: Unassigned
• Reporter: Caizhi Weng
• Created: 04/Nov/21 04:01
• Updated: 11/Mar/24 12:44

Description

I've seen this problem for about three times in the user mailing thread (previously I suspect that the users are specifying the wrong execution.target) until I myself also bumped into this problem. I've submitted a yarn-per-job batch insert SQL with Flink SQL client and after the job finishes Yarn application is not terminated.

This is because yarn job cluster is using MiniDispatcher and it will directly terminate only in detached execution mode. This execution mode is (through some function calls) related to DeploymentOptions#ATTACHED which is true by default if jobs are submitted from SQL client.

When submitting an insert job, SQL client will not wait for the job to finish. Instead it only reports the job id. So I think it is reasonable to set detached mode for every insert job.