[SPARK-39324] Log ExecutorDecommission as INFO level in TaskSchedulerImpl

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 28/May/22 01:31
• Updated: 16/Dec/22 17:34
• Resolved: 28/May/22 17:38

Description

Like the other module, `TaskSchedulerImpl` should log the decommission as `INFO` level.

22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission executors: 8
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 to decommissioning.
22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: Executor decommission.
22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason statistics: ...
22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(8, 100.103.40.13, 43353, None)
22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in removeExecutor
22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 (epoch 7)
22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
22/05/28 01:25:34 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 8

Issue Links

• links to: [Github] Pull Request #36707 (dongjoon-hyun)