[SPARK-27760] Spark resources - user configs change .count to be .amount

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Epic Link: GPU-aware Scheduling
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 17/May/19 15:40
• Updated: 06/Jun/19 19:20
• Resolved: 06/Jun/19 19:17

Description

For the Spark resources, we created the config

spark.{driver/executor}.resource.{resourceName}.count

I think we should change .count to be .amount. That more easily allows users to specify things with suffix like memory in a single config and they can combine the value and unit. Without this they would have to specify 2 separate configs (like .count and .unit) which seems more of a hassle for the user.

Note the yarn configs for resources use amount:  spark.yarn.{executor/driver/am}.resource=<some amount>, where the <some amont> is value and unit together. I think that makes a lot of sense. Filed a separate Jira to add .amount to the yarn configs as well.

Issue Links

• is blocked by: SPARK-27361 YARN support for GPU-aware scheduling
• links to: GitHub Pull Request #24810