[SPARK-27198] Heartbeat interval mismatch in driver and executor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.3, 2.4.0
• Fix Version/s: 2.4.1
• Component/s: Spark Core
• Labels: release-notes
• Assignee: Ajith S
• Reporter: Ajith S
• Created: 19/Mar/19 04:33
• Updated: 21/Apr/19 22:37
• Resolved: 25/Mar/19 20:38

Description

When heartbeat interval is configured via spark.executor.heartbeatInterval without specifying units, we have time mismatched between driver(considers in seconds) and executor(considers as milliseconds)

 https://github.com/apache/spark/blob/v2.4.1-rc8/core/src/main/scala/org/apache/spark/SparkConf.scala#L613

vs

https://github.com/apache/spark/blob/v2.4.1-rc8/core/src/main/scala/org/apache/spark/executor/Executor.scala#L858

 

 

Issue Links

• relates to: SPARK-27419 When setting spark.executor.heartbeatInterval to a value less than 1 seconds, it will always fail
• links to: GitHub Pull Request #24140