Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33539

Standardize exception messages in Spark

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Spark Core, SQL
    • None

    Description

      In the SPIP: Standardize Exception Messages in Spark, there are three major improvements proposed:

      1. Group error messages in dedicated files.
      2. Establish an error message guideline for developers.
      3. Improve error message quality.

      The first step is to centralize error messages for each component into its own dedicated file(s). This can help with auditing error messages and subsequent tasks to establish a guideline and improve message quality in the future. 

      A general rule of thumb for grouping exceptions: 

      • AnalysisException => QueryCompilationErrors
      • SparkException, RuntimeException(UnsupportedOperationException, IllegalArgumentException...) => QueryExecutionErrors
      • ParseException => QueryParsingErrors

      (A special case for Command: since commands are executed eagerly, users can immediately see the errors even if the exceptions are thrown in SparkPlan.execute. In this case, we should group the exceptions in commands as QueryCompilationErrors.)

      Here is an example RP to group all `AnalysisExcpetion` in Analyzer into QueryCompilationErrors: SPARK-32670

      Please see the SPIP for more details.

      Exceptions per component:

      component exception
      sql 714
      core 334
      mllib 161
      streaming 43
      resource-managers 42
      external 35
      examples 20
      mllib-local 10
      graphx 6
      repl 5
      hadoop-cloud 1

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              allisonwang-db Allison Wang
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: