[SPARK-25546] RDDInfo uses SparkEnv before it may have been initialized

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core, Tests
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 26/Sep/18 18:10
• Updated: 27/Sep/18 16:27
• Resolved: 27/Sep/18 16:27

Description

This code:

private[spark] object RDDInfo {
  private val callsiteLongForm = SparkEnv.get.conf.get(EVENT_LOG_CALLSITE_LONG_FORM)

Has two problems:

• it keeps that value across different SparkEnv instances. So e.g. if you have two tests that rely on different values for that config, one of them will break.
• it assumes tests always initialize a SparkEnv. e.g. if you run "core/testOnly *.AppStatusListenerSuite", it will fail because SparkEnv.get returns null.

Issue Links

• is caused by: SPARK-23820 Allow the long form of call sites to be recorded in the log
• links to: [Github] Pull Request #22558 (vanzin)