[SPARK-31921] Wrong warning of "WARN Master: App app-xxx requires more resource than any of Workers could have."

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: Spark Core
• Assignee: wuyi
• Reporter: wuyi
• Created: 06/Jun/20 14:24
• Updated: 09/Jun/20 16:23
• Resolved: 09/Jun/20 16:21

Description

When starting spark-shell using local cluster mode, e.g. ./bin/spark-shell --master "local-cluster[2, 1, 1024]", there will be a warning:

 

20/06/06 22:09:09 WARN Master: App app-20200606220908-0000 requires more resource than any of Workers could have.

which means the application can not get enough resources to launch at least one executor.

But that's not true since we can successfully complete a job.

Issue Links

• is caused by: SPARK-27371 Standalone master receives resource info from worker and allocate driver/executor properly
• links to: [Github] Pull Request #28742 (Ngone51)
• links to: [Github] Pull Request #28742 (Ngone51)