Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33539

Standardize exception messages in Spark

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Spark Core, SQL
    • None

    Description

      In the SPIP: Standardize Exception Messages in Spark, there are three major improvements proposed:

      1. Group error messages in dedicated files.
      2. Establish an error message guideline for developers.
      3. Improve error message quality.

      The first step is to centralize error messages for each component into its own dedicated file(s). This can help with auditing error messages and subsequent tasks to establish a guideline and improve message quality in the future. 

      A general rule of thumb for grouping exceptions: 

      • AnalysisException => QueryCompilationErrors
      • SparkException, RuntimeException(UnsupportedOperationException, IllegalArgumentException...) => QueryExecutionErrors
      • ParseException => QueryParsingErrors

      (A special case for Command: since commands are executed eagerly, users can immediately see the errors even if the exceptions are thrown in SparkPlan.execute. In this case, we should group the exceptions in commands as QueryCompilationErrors.)

      Here is an example RP to group all `AnalysisExcpetion` in Analyzer into QueryCompilationErrors: SPARK-32670

      Please see the SPIP for more details.

      Exceptions per component:

      component exception
      sql 714
      core 334
      mllib 161
      streaming 43
      resource-managers 42
      external 35
      examples 20
      mllib-local 10
      graphx 6
      repl 5
      hadoop-cloud 1

      Attachments

        Issue Links

          blocks

          Umbrella - An overarching type made of sub-tasks SPARK-37935 Migrate onto error classes

          • Major - Major loss of function.
          • In Progress
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-35866 Improve error message quality

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link SPIP: Standardize Exception Messages in Spark

          1.
          		 Group exception messages in Catalyst Analyzer in one file Sub-task Resolved Xinyi Yu
          2.
          		 Group exception messages in catalyst/analysis Sub-task Resolved Jiaan Geng
          3.
          		 Group exception messages in catalyst/catalog Sub-task Resolved Jiaan Geng
          4.
          		 Group exception messages in catalyst/expressions Sub-task Resolved Jiaan Geng
          5.
          		 Group exception messages in catalyst/parser Sub-task Resolved Apache Spark
          6.
          		 Group exception messages in execution/command Sub-task Resolved dgd_contributor
          7.
          		 Group exception messages in execution/datasources Sub-task Resolved Jiaan Geng
          8.
          		 Group exception messages in execution/datasources/v2 Sub-task Resolved Karen Feng
          9.
          		 Group exception messages in sql/execution Sub-task Resolved Jiaan Geng
          10.
          		 Group exception about data format schema Sub-task Resolved angerszhu
          11.
          		 Unify overflow exception error message of integral types Sub-task Resolved Gengliang Wang
          12.
          		 Improve error message for casting cause overflow error Sub-task Resolved angerszhu
          13.
          		 Group exception messages in sql/types Sub-task Resolved Jiaan Geng
          14.
          		 Group exception messages in catalyst/util Sub-task Resolved Jiaan Geng
          15.
          		 Group exception messages in connector/catalog Sub-task Resolved Jiaan Geng
          16.
          		 Group exception messages in execution/streaming Sub-task Resolved Jiaan Geng
          17.
          		 Group exception messages in sql/streaming Sub-task Resolved Jiaan Geng
          18.
          		 Group exception messages in sql/catalyst Sub-task Resolved Jiaan Geng
          19.
          		 Group exception messages in spark/sql (catalyst) Sub-task Resolved dgd_contributor
          20.
          		 Group exception messages in spark/sql (core) Sub-task Resolved Jiaan Geng
          21.
          		 Group exception messages in hive/thriftserver Sub-task Resolved Jiaan Geng
          22.
          		 Group exception messages in hive/client Sub-task Resolved Jiaan Geng
          23.
          		 Group exception messages in hive/execution Sub-task Resolved Jiaan Geng
          24.
          		 Group exception messages in core/rdd Sub-task Resolved dgd_contributor
          25.
          		 Group exception messages in core/resource Sub-task In Progress Unassigned
          26.
          		 Group exception messages in core/scheduler Sub-task Resolved dgd_contributor
          27.
          		 Group exception messages in core/storage Sub-task Resolved dgd_contributor
          28.
          		 Group exception messages in core/util Sub-task In Progress Unassigned
          29.
          		 Group exception messages in core/status Sub-task In Progress Unassigned
          30.
          		 Group exception messages in core/api Sub-task Resolved dgd_contributor
          31.
          		 Group exception messages in core/deploy Sub-task In Progress Unassigned

          Activity

            People

              Unassigned Unassigned
              allisonwang-db Allison Wang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: