[SPARK-18115] Custom metrics Sink/Source prevent Executor from starting

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Kostya Golikov
• Created: 26/Oct/16 12:45
• Updated: 21/May/19 04:15
• Resolved: 21/May/19 04:15

Description

Even though there is a semi-official support for custom metrics, in practice specifying either custom sink or custom source will lead to NoClassDefFound exceptions on executor side (but will be fine on driver side).

The initialization goes as:

1. CoarseGrainedExecutorBackend prepares SparkEnv for executor
2. SparkEnv initializes MetricSystem. In case of executor it also starts it
3. On `.start()` MetricsSystem parses configuration files and creates instances of sinks and sources. This is where the issue actually happens – it tries to instantiate classes which are not there yet – jars and files are downloaded downstream, in Executor

One of the possible solutions is to NOT start MetricSystem this fast, just like driver does, but postpone it until jar with user defined code is fetched and available on classpath.