[SPARK-23285] Allow spark.executor.cores to be fractional

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Kubernetes, Scheduler, Spark Core, Spark Submit
• Assignee: Yinan Li
• Reporter: Anirudh Ramanathan
• Created: 31/Jan/18 14:23
• Updated: 17/May/20 17:48
• Resolved: 02/Apr/18 19:26

Description

There is a strong check for an integral number of cores per executor in SparkSubmitArguments.scala#L270-L272. Given we're reusing that property in K8s, does it make sense to relax it?

 

K8s treats CPU as a "compressible resource" and can actually assign millicpus to individual containers. Also to be noted - spark.driver.cores has no such check in place.

Issue Links

• links to: [Github] Pull Request #20460 (liyinan926)
• links to: [Github] Pull Request #20553 (liyinan926)
• links to: [Github] Pull Request #22585 (dongjoon-hyun)