Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37935

Migrate onto error classes

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • Spark Core, SQL
    • None

    Description

      The PR https://github.com/apache/spark/pull/32850 introduced error classes as a part of the error messages framework (https://issues.apache.org/jira/browse/SPARK-33539). Need to migrate all exceptions from QueryExecutionErrors, QueryCompilationErrors and QueryParsingErrors on the error classes using instances of SparkThrowable, and carefully test every error class by writing tests in dedicated test suites:

      • QueryExecutionErrorsSuite for the errors that are occurred during query execution
      • QueryCompilationErrorsSuite ... query compilation or eagerly executing commands
      • QueryParsingErrorsSuite ... parsing errors

      Here is an example https://github.com/apache/spark/pull/35157 of how an existing Java exception can be replaced, and testing of related error classes.At the end, we should migrate all exceptions from the files Query.*Errors.scala and cover all error classes from the error-classes.json file by tests.

      Attachments

        Issue Links

          1.
          Throw Spark exceptions from AES functions Sub-task Resolved Max Gekk
          2.
          Use error classes in the parsing errors of intervals Sub-task In Progress Unassigned
          3.
          Use error classes in the parsing errors of lateral join Sub-task Resolved Terry Kim
          4.
          Use error classes in the parsing errors of partitions Sub-task Resolved panbingkun
          5.
          Use error classes in the parsing errors of properties Sub-task Resolved panbingkun
          6.
          Use error classes in the compilation errors of partitions Sub-task Open Unassigned
          7.
          Use error classes in the compilation errors of casting Sub-task Resolved huangtengfei
          8.
          Use error classes in the compilation errors of properties Sub-task In Progress Unassigned
          9.
          Use error classes in the compilation errors of grouping Sub-task Resolved Apache Spark
          10.
          Use error classes in the execution errors of casting Sub-task Open Unassigned
          11.
          Use error classes in the execution errors of arithmetic ops Sub-task Open Unassigned
          12.
          Use error classes in the execution errors related to partitions Sub-task In Progress Apache Spark
          13.
          Switch to more generic error classes in AES functions Sub-task Resolved Max Gekk
          14.
          Use error classes in the execution errors related to unsupported input type Sub-task Resolved leesf
          15.
          Replace the unsupported error classes by `UNSUPPORTED_FEATURE` Sub-task Resolved Max Gekk
          16.
          Improve the error for pivoting of unsupported value types Sub-task Resolved Yuto Akutsu
          17.
          Use error classes in the parsing errors of transform Sub-task Resolved Yuto Akutsu
          18.
          Use error classes in the parsing errors of windows Sub-task Resolved Yuto Akutsu
          19.
          Use error classes in the parsing errors of joins Sub-task Resolved huangtengfei
          20.
          Use error classes in the parsing errors of functions Sub-task Resolved huangtengfei
          21.
          Use error classes in the compilation errors of python/pandas UDFs Sub-task Resolved Haejoon Lee
          22.
          Use error classes in the compilation errors of UDF/UDAF Sub-task Resolved huangtengfei
          23.
          Use error classes in the compilation errors of windows Sub-task Open Unassigned
          24.
          Use error classes in the execution errors of date/timestamp handling Sub-task Resolved huangtengfei
          25.
          Use error classes in the execution errors of pivoting Sub-task Resolved huangtengfei
          26.
          Replace the error classes related to invalid parameters by `INVALID_PARAMETER_VALUE` Sub-task Open Max Gekk
          27.
          Check the whole message of error classes Sub-task Resolved Max Gekk
          28.
          Keep only user-facing error classes Sub-task Resolved Max Gekk
          29.
          Preserve the error class of `AnalysisException` while constructing of function builder Sub-task Resolved Max Gekk
          30.
          Use error classes in org.apache.spark.metrics Sub-task Resolved Bo Zhang
          31.
          Use error classes in org.apache.spark.broadcast Sub-task In Progress Unassigned
          32.
          Use error classes in org.apache.spark.executor Sub-task Open Unassigned
          33.
          Use error classes in org.apache.spark.input Sub-task In Progress Unassigned
          34.
          Use error classes in org.apache.spark.io Sub-task Open Unassigned
          35.
          Use error classes in org.apache.spark.launcher Sub-task Open Unassigned
          36.
          Use error classes in org.apache.spark.mapred Sub-task In Progress Unassigned
          37.
          Use error classes in org.apache.spark.memory Sub-task Open Unassigned
          38.
          Use error classes in org.apache.spark.metrics Sub-task Resolved Unassigned
          39.
          Use error classes in org.apache.spark.network Sub-task Open Unassigned
          40.
          Use error classes in org.apache.spark.partial Sub-task In Progress Unassigned
          41.
          Use error classes in org.apache.spark.rdd Sub-task Open Unassigned
          42.
          Use error classes in org.apache.spark.rpc Sub-task Open Unassigned
          43.
          Use error classes in org.apache.spark.scheduler Sub-task Open Unassigned
          44.
          Use error classes in org.apache.spark.security Sub-task Open Unassigned
          45.
          Use error classes in org.apache.spark.serializer Sub-task Open Unassigned
          46.
          Use error classes in org.apache.spark.shuffle Sub-task Open Unassigned
          47.
          Use error classes in org.apache.spark.storage Sub-task Open Unassigned
          48.
          Use error classes in org.apache.spark.ui Sub-task Open Unassigned
          49.
          Substitute Java overflow exception from TIMESTAMPADD by Spark exception Sub-task Resolved Max Gekk
          50.
          Use error classes in the compilation errors of generators Sub-task In Progress Unassigned
          51.
          Use error classes in the compilation errors of deserializer Sub-task Resolved panbingkun
          52.
          Use error classes in the compilation errors of not allowed DESC PARTITION Sub-task Resolved huangtengfei
          53.
          Use error classes in the compilation errors of SHOW CREATE TABLE Sub-task Open Unassigned
          54.
          Use error classes in the compilation errors of column/attr resolving Sub-task Open Unassigned
          55.
          Use error classes in the compilation errors of function args Sub-task In Progress Unassigned
          56.
          Use error classes in the execution errors of dictionary encoding Sub-task Open Unassigned
          57.
          Use error classes in the execution errors of save mode Sub-task Resolved panbingkun
          58.
          Inline IllegalStateException out from QueryExecutionErrors Sub-task Resolved Max Gekk
          59.
          Test the error class: AMBIGUOUS_FIELD_NAME Sub-task Resolved panbingkun
          60.
          Test the error class: CANNOT_CAST_DATATYPE Sub-task Open Unassigned
          61.
          Test the error class: CANNOT_CHANGE_DECIMAL_PRECISION Sub-task Resolved panbingkun
          62.
          Test the error class: CANNOT_PARSE_DECIMAL Sub-task Resolved panbingkun
          63.
          Test the error class: CAST_CAUSES_OVERFLOW Sub-task Resolved panbingkun
          64.
          Test the error class: CONCURRENT_QUERY Sub-task Open Unassigned
          65.
          Test the error class: DIVIDE_BY_ZERO Sub-task Resolved panbingkun
          66.
          Test the error class: DUPLICATE_KEY Sub-task Resolved panbingkun
          67.
          Test the error class: FAILED_EXECUTE_UDF Sub-task Resolved panbingkun
          68.
          Test the error class: FAILED_RENAME_PATH Sub-task Open Unassigned
          69.
          Test the error class: FAILED_SET_ORIGINAL_PERMISSION_BACK Sub-task Resolved panbingkun
          70.
          Move tests for the grouping error classes to QueryCompilationErrorsSuite Sub-task Resolved Max Gekk
          71.
          Move the tests `GROUPING_SIZE_LIMIT_EXCEEDED` to QueryCompilationErrorsSuite Sub-task Resolved Max Gekk
          72.
          Test the error class: INCOMPARABLE_PIVOT_COLUMN Sub-task Resolved lvshaokang
          73.
          Test the error class: INCOMPATIBLE_DATASOURCE_REGISTER Sub-task Resolved panbingkun
          74.
          Test the error class: INDEX_OUT_OF_BOUNDS Sub-task In Progress Unassigned
          75.
          Test the error class: INTERNAL_ERROR Sub-task Open Unassigned
          76.
          Test the error classes: INVALID_ARRAY_INDEX* Sub-task Resolved panbingkun
          77.
          Test the error classes: INVALID_FIELD_NAME Sub-task Resolved Apache Spark
          78.
          Test the error class: INVALID_FRACTION_OF_SECOND Sub-task Resolved panbingkun
          79.
          Test the error class: INVALID_INPUT_SYNTAX_FOR_NUMERIC_TYPE Sub-task Resolved panbingkun
          80.
          Test the error class: INVALID_JSON_SCHEMA_MAPTYPE Sub-task Resolved panbingkun
          81.
          Test the error class: MAP_KEY_DOES_NOT_EXIST* Sub-task Resolved panbingkun
          82.
          Move the tests `MISSING_COLUMN` to QueryCompilationErrorsSuite Sub-task Resolved panbingkun
          83.
          Test the error class: MISSING_STATIC_PARTITION_COLUMN Sub-task Open Unassigned
          84.
          Test the pivot error classes Sub-task Resolved panbingkun
          85.
          Move the tests for `NON_PARTITION_COLUMN` to QueryCompilationErrorsSuite Sub-task Resolved Max Gekk
          86.
          Move the tests for `PARSE_EMPTY_STATEMENT` to QueryParsingErrorsSuite Sub-task Resolved panbingkun
          87.
          Test the error class: PIVOT_VALUE_DATA_TYPE_MISMATCH Sub-task Resolved panbingkun
          88.
          Move the tests for `PARSE_SYNTAX_ERROR` to QueryParsingErrorsSuite Sub-task Resolved panbingkun
          89.
          Test the error class: RENAME_SRC_PATH_NOT_FOUND Sub-task Open Unassigned
          90.
          Test the error class: SECOND_FUNCTION_ARGUMENT_NOT_INTEGER Sub-task Resolved panbingkun
          91.
          Test the error class: UNRECOGNIZED_SQL_TYPE Sub-task Resolved panbingkun
          92.
          Test the error class: UNSUPPORTED_DATATYPE Sub-task Open Unassigned
          93.
          Move the tests for `WRITING_JOB_ABORTED` to QueryExecutionErrorsSuite Sub-task Resolved Max Gekk
          94.
          Output parameter values of error classes in SQL style Sub-task Resolved Max Gekk
          95.
          Output identifiers in error messages in SQL style Sub-task Resolved Max Gekk
          96.
          Output types in error messages in SQL style Sub-task Resolved Max Gekk
          97.
          Wrap SQL statements by double quotes in error messages Sub-task Resolved Max Gekk
          98.
          Move error class tests requiring ANSI SQL mode to QueryExecutionAnsiErrorsSuite Sub-task Resolved Gengliang Wang
          99.
          Prepend error class tag to error messages Sub-task Resolved Max Gekk
          100.
          Use double quotes for types in error messages Sub-task Resolved Max Gekk
          101.
          Use double quotes for SQL configs in error messages Sub-task Resolved Max Gekk
          102.
          Output SQL statements in upper case in error messages Sub-task Resolved Max Gekk
          103.
          Replace the error class ILLEGAL_SUBSTRING by INVALID_PARAMETER_VALUE Sub-task Resolved Max Gekk
          104.
          Move error message of INCONSISTENT_BEHAVIOR_CROSS_VERSION to the json file Sub-task Resolved Max Gekk
          105.
          Improve messages of error classes Sub-task Resolved Max Gekk
          106.
          Test the error class: PIVOT_VALUE_DATA_TYPE_MISMATCH Sub-task Open Unassigned
          107.
          Throw an exception w/ error class for an invalid bucket file Sub-task Resolved panbingkun
          108.
          Wrap asserts/illegal state exceptions by the INTERNAL_ERROR exception in actions Sub-task Resolved Max Gekk
          109.
          Replace sys.error by IllegalStateException in Spark SQL Sub-task Resolved Max Gekk
          110.
          Throw an exception w/ an error class for multiple rows from a subquery used as an expression Sub-task Resolved panbingkun
          111.
          Remove SparkIllegalStateException Sub-task Resolved Max Gekk
          112.
          Use double quotes for values of SQL configs/DS options in error messages Sub-task Resolved Max Gekk
          113.
          Improve errors related to CAST Sub-task Resolved Max Gekk
          114.
          Code clean up in SparkThrowableHelper.getMessage Sub-task Resolved Gengliang Wang
          115.
          Describe the rules of quoting elements in error messages Sub-task Resolved Max Gekk

          Activity

            People

              maxgekk Max Gekk
              maxgekk Max Gekk
              Max Gekk Max Gekk
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: