[KYLIN-4450] Add the feature that adjusting spark driver memory adaptively

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v4.0.0-alpha
• Component/s: Storage - Parquet
• Sprint: Sprint 51
• Assignee: Wang Rupeng
• Reporter: xuekaiqi
• Created: 17/Apr/20 09:44
• Updated: 13/Sep/20 12:17
• Resolved: 01/Jun/20 02:42

Description

For now the cubing job can adaptively adjust the following spark properties to use of resources retionally, but the driver memory of the spark job uploaded to cluster haven't been done.

 

spark.executor.memory
spark.executor.cores
spark.executor.memoryOverhead
spark.executor.instances
spark.sql.shuffle.partitions

Issue Links

• links to: #188