Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37935

Migrate onto error classes

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • Spark Core, SQL
    • None

    Description

      The PR https://github.com/apache/spark/pull/32850 introduced error classes as a part of the error messages framework (https://issues.apache.org/jira/browse/SPARK-33539). Need to migrate all exceptions from QueryExecutionErrors, QueryCompilationErrors and QueryParsingErrors on the error classes using instances of SparkThrowable, and carefully test every error class by writing tests in dedicated test suites:

      • QueryExecutionErrorsSuite for the errors that are occurred during query execution
      • QueryCompilationErrorsSuite ... query compilation or eagerly executing commands
      • QueryParsingErrorsSuite ... parsing errors

      Here is an example https://github.com/apache/spark/pull/35157 of how an existing Java exception can be replaced, and testing of related error classes.At the end, we should migrate all exceptions from the files Query.*Errors.scala and cover all error classes from the error-classes.json file by tests.

      Attachments

        Issue Links

          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. SPARK-34920 Introduce SQLSTATE and ERRORCODE to SQL Exception

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-33539 Standardize exception messages in Spark

          • Major - Major loss of function.
          • Open
          is related to

          Umbrella - An overarching type made of sub-tasks SPARK-41288 Server-specific improvement, error handling and API

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Umbrella - An overarching type made of sub-tasks SPARK-44111 Prepare Apache Spark 4.0.0

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          links to

          Pull request #35291 [Github] Pull Request #35291 (khalidmammadov)

          Pull request #35291 [Github] Pull Request #35291 (khalidmammadov)

          Pull request #35293 [Github] Pull Request #35293 (khalidmammadov)

          Pull request #36160 [Github] Pull Request #36160 (panbingkun)

          Pull request #36160 [Github] Pull Request #36160 (panbingkun)

          Pull request #37357 [Github] Pull Request #37357 (MaxGekk)

          Pull request #37357 [Github] Pull Request #37357 (MaxGekk)

          Pull request #38051 [Github] Pull Request #38051 (MaxGekk)

          1.
          		 Use error classes in the compilation errors of partitions Sub-task Open Unassigned
          2.
          		 Use error classes in the execution errors of casting Sub-task In Progress Unassigned
          3.
          		 Use error classes in the execution errors related to partitions Sub-task In Progress Unassigned
          4.
          		 Use error classes in the compilation errors of windows Sub-task Open Unassigned
          5.
          		 Replace the error classes related to invalid parameters by `INVALID_PARAMETER_VALUE` Sub-task Open Max Gekk
          6.
          		 Use error classes in org.apache.spark.input Sub-task In Progress Unassigned
          7.
          		 Use error classes in org.apache.spark.launcher Sub-task Open Unassigned
          8.
          		 Use error classes in org.apache.spark.mapred Sub-task In Progress Unassigned
          9.
          		 Use error classes in org.apache.spark.partial Sub-task In Progress Unassigned
          10.
          		 Use error classes in org.apache.spark.rdd Sub-task Open Unassigned
          11.
          		 Use error classes in org.apache.spark.rpc Sub-task Open Unassigned
          12.
          		 Use error classes in org.apache.spark.scheduler Sub-task Open Unassigned
          13.
          		 Use error classes in org.apache.spark.security Sub-task In Progress Unassigned
          14.
          		 Use error classes in org.apache.spark.ui Sub-task Open Unassigned
          15.
          		 Use error classes in the compilation errors of SHOW CREATE TABLE Sub-task Open Unassigned
          16.
          		 Use error classes in the compilation errors of column/attr resolving Sub-task Open Unassigned
          17.
          		 Use error classes in the compilation errors of function args Sub-task In Progress Unassigned
          18.
          		 Use error classes in the execution errors of dictionary encoding Sub-task In Progress Unassigned
          19.
          		 Test the error class: CANNOT_CAST_DATATYPE Sub-task Open Unassigned
          20.
          		 Test the error class: MISSING_STATIC_PARTITION_COLUMN Sub-task In Progress Apache Spark
          21.
          		 Introduce error sub-classes of the operation not allowed parse error Sub-task Open Unassigned
          22.
          		 Refine Sequence#checkInputDataTypes related DataTypeMismatch Sub-task Open Unassigned
          23.
          		 Remove the class TypeCheckFailure Sub-task In Progress Max Gekk
          24.
          		 Add utils for testing `checkError` in various test mode Sub-task Open Unassigned
          25.
          		 Assign a name to the error class _LEGACY_ERROR_TEMP_1203 Sub-task In Progress Unassigned
          26.
          		 Add missing dots for error messages in error classes. Sub-task In Progress Unassigned
          27.
          		 Rename `FIELD_NOT_FOUND` Sub-task Open Unassigned
          28.
          		 Integrate COLUMN_NOT_FOUND and FIELD_NOT_FOUND for DDL. Sub-task Open Unassigned
          29.
          		 Integrate all errors that causes re-create of a view failing into one error class Sub-task Open Unassigned
          30.
          		 Assign name to _LEGACY_ERROR_TEMP_2232 Sub-task Open Unassigned
          31.
          		 Assign name to _LEGACY_ERROR_TEMP_2432 Sub-task Open Unassigned
          32.
          		 Assign name to_LEGACY_ERROR_TEMP_2235 Sub-task Open Unassigned
          33.
          		 Assign name to _LEGACY_ERROR_TEMP_0035 Sub-task In Progress Unassigned
          34.
          		 Assign name to _LEGACY_ERROR_TEMP_2256 Sub-task Open Unassigned
          35.
          		 Make suggestion in error message smarter Sub-task Open Unassigned
          36.
          		 Assign a name to the error class _LEGACY_ERROR_TEMP_2000 Sub-task In Progress Unassigned
          37.
          		 Assign a name to the error class _LEGACY_ERROR_TEMP_2003 Sub-task In Progress Unassigned
          38.
          		 Assign a name to the error class _LEGACY_ERROR_TEMP_2005 Sub-task Open Unassigned
          39.
          		 Assign a name to the error class _LEGACY_ERROR_TEMP_2011 Sub-task Open Unassigned
          40.
          		 Assign a name to the error class _LEGACY_ERROR_TEMP_2013 Sub-task Open Unassigned
          41.
          		 Assign a name to the error class _LEGACY_ERROR_TEMP_2017 Sub-task Open Unassigned
          42.
          		 Write test for CANNOT_FIND_BATCH (Prev _LEGACY_ERROR_TEMP_2132) Sub-task Open Unassigned
          43.
          		 Convert HiveException to SparkException Sub-task Open Unassigned
          44.
          		 Assign names to the error class _LEGACY_ERROR_TEMP_[1017,1073,1074,1076,1125,1126] Sub-task Open Unassigned
          45.
          		 Migrate SchemaColumnConvertNotSupportedException onto DATATYPE_MISMATCH error classes Sub-task Open Unassigned
          46.
          		 Replace all Scala `require` with `SparkException.require` Sub-task Open Unassigned
          47.
          		 Assign error classes to ALTER COLUMN errors Sub-task Open Unassigned
          48.
          		 Assign error classes to SHOW CREATE TABLE errors Sub-task Open Unassigned
          49.
          		 Assign classes to interval errors Sub-task Open Unassigned
          50.
          		 Assign classes to Row to JSON errors Sub-task Open Unassigned
          51.
          		 Assign classes to Parquet type errors Sub-task Open Unassigned
          52.
          		 Assign classes to Parquet converter errors Sub-task Open Unassigned
          53.
          		 Assign classes to DEFAULT value errors Sub-task Open Unassigned

          Activity

            People

              maxgekk Max Gekk
              maxgekk Max Gekk
              Max Gekk Max Gekk
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated: