[SPARK-26993] _minRegisteredRatio default value is zero not 0.8 for Yarn

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Yifan Guo
• Created: 26/Feb/19 07:24
• Updated: 17/May/20 18:13
• Resolved: 27/Feb/19 09:29

Description

private[spark]

class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: RpcEnv)
extends ExecutorAllocationClient with SchedulerBackend with Logging {

// Use an atomic variable to track total number of cores in the cluster for simplicity and speed
protected val totalCoreCount = new AtomicInteger(0)
// Total number of executors that are currently registered
protected val totalRegisteredExecutors = new AtomicInteger(0)
protected val conf = scheduler.sc.conf
private val maxRpcMessageSize = RpcUtils.maxMessageSizeBytes(conf)
private val defaultAskTimeout = RpcUtils.askRpcTimeout(conf)
// Submit tasks only after (registered resources / total expected resources)
// is equal to at least this value, that is double between 0 and 1.
private val _minRegisteredRatio =
math.min(1, conf.getDouble("spark.scheduler.minRegister*edResourcesRatio", 0))*

 

override val minRegisteredRatio =
if (conf.getOption("spark.scheduler.minRegisteredResourcesRatio").isEmpty)

{ 0.8 }

else

{ super.minRegisteredRatio }

 

Apparently, if "spark.scheduler.minRegisteredResourcesRatio" is not configured, default value is zero not 0.8

 

is that on purpose ?