[SPARK-32185] User Guide - Monitoring

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Later
• Affects Version/s: 3.1.0
• Component/s: Documentation, PySpark
• Epic Link: Project Zen
• Assignee: Abhijeet Prasad
• Reporter: Hyukjin Kwon
• Created: 06/Jul/20 08:35
• Updated: 12/Dec/22 18:11
• Resolved: 23/Jun/21 13:50

Description

Monitoring. We should focus on how to monitor PySpark jobs.

• Custom Worker, see also https://github.com/apache/spark/tree/master/python/test_coverage to enable test coverage that include worker sides too.
• Sentry Support (?) https://blog.sentry.io/2019/11/12/sentry-for-data-error-monitoring-with-pyspark
• Link back https://spark.apache.org/docs/latest/monitoring.html .
• ...

Issue Links

• is blocked by: SPARK-32179 Replace and redesign the documentation base
• Is contained by: SPARK-31851 Redesign PySpark documentation