Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31921

Wrong warning of "WARN Master: App app-xxx requires more resource than any of Workers could have."

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.1, 3.1.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When starting spark-shell using local cluster mode, e.g. ./bin/spark-shell --master "local-cluster[2, 1, 1024]", there will be a warning:

       

      20/06/06 22:09:09 WARN Master: App app-20200606220908-0000 requires more resource than any of Workers could have.
      

      which means the application can not get enough resources to launch at least one executor.

      But that's not true since we can successfully complete a job.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Ngone51 wuyi
                Reporter:
                Ngone51 wuyi
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: