[SPARK-49804] Fix to use the exit code of executor container always

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.1, 4.0.0, 3.4.3, 3.5.3
• Fix Version/s: 4.0.0, 3.4.4, 3.5.4
• Component/s: Kubernetes
• Labels: pull-request-available
• Assignee: Oleksiy Dyagilev
• Reporter: Oleksiy Dyagilev
• Created: 27/Sep/24 02:12
• Updated: 27/Sep/24 05:03
• Resolved: 27/Sep/24 05:03

Description

When deploying Spark pods on Kubernetes with sidecars, the reported executor's exit code may be incorrect.

For example, the reported executor's exit code is 0, but the actual is 52 (OOM).

2024-09-25 02:35:29,383 ERROR TaskSchedulerImpl: org.apache.spark.scheduler.TaskSchedulerImpl.logExecutorLoss(TaskSchedulerImpl.scala:972) - Lost executor 1 on XXXXX: The executor with id 1 exited with exit code 0(success).
  
The API gave the following container statuses:
 
     container name: fluentd
     container image: docker-images-release.XXXXX.com/XXXXX/fluentd:XXXXX
     container state: terminated
     container started at: 2024-09-25T02:32:17Z
     container finished at: 2024-09-25T02:34:52Z
     exit code: 0
     termination reason: Completed
 
     container name: istio-proxy
     container image: docker-images-release.XXXXX.com/XXXXX-istio/proxyv2:XXXXX
     container state: running
     container started at: 2024-09-25T02:32:16Z
 
     container name: spark-kubernetes-executor
     container image: docker-dev-artifactory.XXXXX.com/XXXXX/spark-XXXXX:XXXXX
     container state: terminated
     container started at: 2024-09-25T02:32:17Z
     container finished at: 2024-09-25T02:35:28Z
     exit code: 52
     termination reason: Error 

Issue Links

• links to: GitHub Pull Request #48275