[CELEBORN-1580] Job hang due to ReadBufferDispatcher not notify exception to listener

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0, 0.5.2
• Labels: pull-request-available
• Assignee: xuhuang
• Reporter: xuhuang
• Created: 27/Aug/24 07:21
• Updated: 28/Aug/24 18:16
• Resolved: 28/Aug/24 18:15

Description

We discovered a flink job, which utilizing Celeborn as remote shuffle service, that had been suspended for over 20 days.

Upon conducting a thorough investigation involving heap dumps of both Flink TaskManagers and Celeborn workers, as well as examing log files, we found that a Celeborn Worker encountered an OOM exception in ReadBufferDispatcher#run, and the exception has been catch, not notify exception to listener. Consequently, the MapPartitionDataReader not further processing as it was awaiting buffer resouces, this lead to the Flink job's suspension.

Issue Links

• links to: GitHub Pull Request #2707