[SPARK-37060] Report driver status does not handle response from backup masters

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.1, 3.1.2, 3.2.0
• Fix Version/s: 3.1.3, 3.2.1, 3.3.0
• Component/s: Spark Core
• Assignee: Mohamadreza Rostami
• Reporter: Mohamadreza Rostami
• Created: 19/Oct/21 14:29
• Updated: 16/Dec/21 07:25
• Resolved: 15/Dec/21 13:01

Description

After an improvement in SPARK-31486, contributor uses 'asyncSendToMasterAndForwardReply' method instead of 'activeMasterEndpoint.askSync' to get the status of driver. Since the driver's status is only available in active master and the 'asyncSendToMasterAndForwardReply' method iterate over all of the masters, we have to handle the response from the backup masters in the client, which the developer did not consider in the SPARK-31486 change. So drivers running in cluster mode and on a cluster with multi-master affected by this bug. I created the patch for this bug and will soon be sent the pull request.

Issue Links

• links to: [Github] Pull Request #34331 (mohamadrezarostami)
• links to: [Github] Pull Request #34911 (mohamadrezarostami)
• links to: [Github] Pull Request #34912 (mohamadrezarostami)
• links to: [Github] Pull Request #34912 (mohamadrezarostami)