Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7668

Improve error logging for spark test crashes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Spark, Testing Infrastructure
    • None

    Description

      While working on HIVE-7665, I noticed that spark local cluster will crash entire JVM on any dependency version mismatch, either due to native code linkage errors if you are unlucky, or gracefully (see spark's ExecutorUncaughtExceptionHandler.scala) most of the time.

      To debug those issues, it would be nice to log a thread-dump upon normal JVM exits by spark by ExecutorUncaughtExceptionHandler, as its unexpected by Hive's test framework. One way is to add a shutdownHook that does this.

      Attachments

        Issue Links

          is part of

          Sub-task - The sub-task of the issue HIVE-7665 Create TestSparkCliDriver to run test in spark local mode [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              szehon Szehon Ho
              szehon Szehon Ho
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: