Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.0.0
-
None
Description
The Apache Spark metrics system provides many useful insights on the Spark workload. In particular, the executor source metrics (https://github.com/apache/spark/blob/master/docs/monitoring.md#component-instance--executor) provide detailed info, including the number of active tasks, some I/O metrics, and task metrics details. Executor source metrics, contrary to other sources (for example ExecutorMetrics source), are not yet available when running in local mode.
This JIRA proposes to register the executor source with the Spark metrics system when running in local mode, as this can be very useful when testing and troubleshooting Spark workloads.