[FLINK-13245] Network stack is leaking files

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Zhijiang
• Reporter: Chesnay Schepler
• Created: 12/Jul/19 10:52
• Updated: 02/Oct/19 17:50
• Resolved: 29/Jul/19 10:27

Description

There's file leak in the network stack / shuffle service.

When running the SlotCountExceedingParallelismTest on Windows a large number of .channel files continue to reside in a flink-netty-shuffle-XXX directory.

From what I've gathered so far these files are still being used by a BoundedBlockingSubpartition. The cleanup logic in this class uses ref-counting to ensure we don't release data while a reader is still present. However, at the end of the job this count has not reached 0, and thus nothing is being released.

The same issue is also present on the ResultPartition level; the ReleaseOnConsumptionResultPartition also are being released while the ref-count is greater than 0.

Overall it appears like there's some issue with the notifications for partitions being consumed.

It is feasible that this issue has recently caused issues on Travis where the build were failing due to a lack of disk space.

Issue Links

• causes: FLINK-12655 SlotCountExceedingParallelismTest failed on Travis
• causes: FLINK-12645 high parallelism e2e test fails due to missing disk space
• relates to: FLINK-13371 Release partitions in JM if producer restarts
• links to: GitHub Pull Request #9132