[SPARK-34426] Add driver and executors POD logs to integration tests log when the test fails

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Kubernetes, Tests
• Assignee: Attila Zsolt Piros
• Reporter: Attila Zsolt Piros
• Created: 12/Feb/21 20:57
• Updated: 16/Feb/21 20:49
• Resolved: 16/Feb/21 20:49

Description

Now both the driver and executors logs are lost.

In https://spark.apache.org/developer-tools.html there is a hint:
"Getting logs from the pods and containers directly is an exercise left to the reader."

But when the test is executed by Jenkins and there is failure we really need the POD logs to analyze problem.

Issue Links

• links to: [Github] Pull Request #31561 (attilapiros)
• links to: [Github] Pull Request #31561 (attilapiros)