[FLINK-24300] MultipleInputOperator is running much more slowly in TPCDS

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.14.0, 1.15.0
• Fix Version/s: 1.14.0, 1.15.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Dawid Wysakowicz
• Reporter: Zhilong Hong
• Created: 15/Sep/21 15:22
• Updated: 26/Jan/22 14:00
• Resolved: 17/Sep/21 19:21

Description

When we are running TPCDS with release 1.14 we find that the job with MultipleInputOperator is running much more slowly than before. With a binary search among the commits, we find that the issue may be introduced by FLINK-23408. 

At the commit 64570e4c56955713ca599fd1d7ae7be891a314c6, the job in TPCDS runs normally, as the image below illustrates:

At the commit e3010c16947ed8da2ecb7d89a3aa08dacecc524a, the job q2.sql gets stuck for a pretty long time (longer than half an hour), as the image below illustrates:

The detail of the job is illustrated below:

The job uses a MultipleInputOperator with one normal input and two chained FileSource. It has finished reading the normal input and start to read the chained source. Each chained source has one source data fetcher.

We capture the jstack of the stuck tasks and attach the file below. From the jstack.txt we can see the main thread is blocked on waiting for the lock, and the lock is held by a source data fetcher. The source data fetcher is still running but the stack keeps on CompletableFuture.cleanStack.

This issue happens in a batch job. However, from where it get blocked, it seems also affects the streaming jobs.

For the reference, the code of TPCDS we are running is located at https://github.com/ververica/flink-sql-benchmark/tree/dev.

Issue Links

• is caused by: FLINK-23408 Wait for a checkpoint completed after finishing a task
• relates to: FLINK-25728 Potential memory leaks in StreamMultipleInputProcessor
• relates to: FLINK-25827 Potential memory leaks in SourceOperator
• links to: GitHub Pull Request #31
• links to: GitHub Pull Request #17303
• links to: GitHub Pull Request #17315