[SPARK-19466] Improve Fair Scheduler Logging

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Scheduler, Spark Core
• Assignee: Eren Avsarogullari
• Reporter: Eren Avsarogullari
• Created: 05/Feb/17 20:43
• Updated: 17/May/20 17:47
• Resolved: 10/Feb/17 16:33

Description

Fair Scheduler Logging for the following cases can be useful for the user.

1- If valid spark.scheduler.allocation.file property is set, user can be informed so user can aware which scheduler file is processed when SparkContext initializes.
2- If invalid spark.scheduler.allocation.file property is set, currently, the following stacktrace is shown to user. In addition to this, more meaningful message can be shown to user by emphasizing the problem at building level of fair scheduler and covering other potential issues at this level.

Exception in thread "main" java.io.FileNotFoundException: INVALID_FILE (No such file or directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at java.io.FileInputStream.<init>(FileInputStream.java:93)
	at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:76)
	at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:75)

3- If spark.scheduler.allocation.file property is not set and default fair scheduler file(fairscheduler.xml) is found in classpath, it will be loaded but currently, user is not informed so logging can be useful.
4- If spark.scheduler.allocation.file property is not set and default fair scheduler file does not exist, currently, user is not informed so logging can be useful.

Issue Links

• links to: [Github] Pull Request #16813 (erenavsarogullari)