Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39541

[Yarn] Diagnostics of yarn UI did not display the exception of driver when driver exit before regiserAM

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • YARN
    • None

    Description

      If commit a job in yarn cluster mode and driver exited before registerAM,Diagnostics of yarn UI did not show the exception that was throwed by driver .Yarn UI only show :

      Application application_xxx failed 1 times (global limit =10; local limit is =1) due to AM Container for appattempt_xxx_000001 exited with exitCode: 13

       

      User must view spark log to find the real reason.for example,spark log shows 

      2022-06-21,17:58:28,273 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User class threw exception: java.lang.ArithmeticException: / by zero
      java.lang.ArithmeticException: / by zero
      	at org.examples.appErrorDemo3$.main(appErrorDemo3.scala:10)
      	at org.examples.appErrorDemo3.main(appErrorDemo3.scala)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:736) 

       

      The reason of this issue is that if driver would not call unregisterAM exited before registerAM ,then yarn UI could not show the real diagnostic information.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            liangyongyuan liangyongyuan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: