[SPARK-32663] TransportClient getting closed when there are outstanding requests to the server

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: Shuffle
• Assignee: Attila Zsolt Piros
• Reporter: Chandni Singh
• Created: 20/Aug/20 00:47
• Updated: 21/Aug/20 06:05
• Resolved: 21/Aug/20 06:05

Description

The implementation of removeBlocks and getHostLocalDirs in ExternalBlockStoreClient closes the client after processing a response in the callback.

This is a cached client which will be re-used for other responses. There could be other outstanding request to the shuffle service, so it should not be closed after processing a response.
Seems like this is a bug introduced with SPARK-27651 and SPARK-27677.

The older methods registerWithShuffleServer and fetchBlocks didn't close the client.

cc attilapiros vanzin mridulm80

Issue Links

• links to: [Github] Pull Request #29492 (attilapiros)
• links to: [Github] Pull Request #29492 (attilapiros)