[ZEPPELIN-4799] Use spark resource configuration

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Component/s: Interpreters, Kubernetes
• Assignee: Philipp Dallig
• Reporter: Philipp Dallig
• Created: 05/May/20 08:43
• Updated: 24/Dec/20 03:19
• Resolved: 11/May/20 18:33

Description

Spark has the possibility to define the resource usage if you run your driver within YARN or Kubernetes in cluster mode.
The following configuration values are used to request or limit resources.

• spark.driver.memory
• spark.driver.memoryOverhead
• spark.driver.cores

We should use this configuration, when setting up a Zeppelin interpreter within YARN or Kubernetes.

A good resource definition is very important in Kubernetes.

My Goal behind:

• Create multiple Spark interpreter with different resource usage.