[SPARK-47361] Improve JDBC data sources

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available releasenotes
• Assignee: Kent Yao
• Reporter: Dongjoon Hyun
• Created: 12/Mar/24 14:36
• Updated: 30/May/24 09:33
• Resolved: 12/Mar/24 14:46

Issue Links

• is related to: SPARK-44111 Prepare Apache Spark 4.0.0

Sub-Tasks

• Add guidelines for timestamp mapping in JdbcDialect#getCatalystType: Resolved
• SPARK-45561 causes mysql unsigned tinyint overflow: Resolved
• Add a general mapping for TIME WITHOUT TIME ZONE to TimestampNTZType: Resolved
• Support Oracle TIMESTAMP WITH LOCAL TIME ZONE type: Resolved
• Remove timezone limitation for ORACLE TIMESTAMP WITH TIMEZONE: Resolved
• Spark: Getting SQLException: Unsupported type -102 reading from Oracle: Resolved
• Upgrade oracle jdbc driver to 23.3.0.23.09: Resolved
• Use gvenzl/oracle-free:23.3-slim to reduce disk usage for docker it: Resolved
• Revert SPARK-43049: Resolved
• Fix TimestampNTZ in Postgres Array: Resolved
• Error in postgresql when pushing down filter by timestamp_ntz field: Resolved
• PostgresDialect distinguishes TIMESTAMP from TIMESTAMP_TZ: Resolved
• Upgrade `PostgreSQL` JDBC driver to 42.7.2 and docker image to 16.2: Resolved
• Convert TINYINT catalyst properly in MySQL Dialect: Resolved
• Handle TIMESTAMP and DATETIME in MYSQLDialect: Resolved
• Align the mapping ways of other unsigned numeric types with TINYINT UNSGINED in MySQLDialect: Resolved
• Upgrade MySQL docker image version to 8.3.0: Resolved
• Improve test coverage for mysql bool synonyms: Resolved
• Support TimestampNTZ for DB2 TIMESTAMP WITH TIME ZONE: Closed
• Bump DB2 Docker version to 11.5.8.0: Resolved
• Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1: Resolved
• Mapped TINYINT to ShortType for MsSqlServerDialect: Resolved
• Support TIMESTAMP WITH TIME ZONE for H2Dialect: Resolved
• Add DatabricksDialect to handle SQL type conversion: Resolved
• Spark wrongly map the BOOLEAN Type to BIT(1) in Snowflake: Resolved
• Redact JDBC url in errors and logs: Resolved
• Port classifyException() in JDBC dialects on error classes: Resolved
• Restore backward compatibility of JdbcDialect.classifyException: Resolved
• Assign error classes/subclasses to JdbcUtils.classifyException: Resolved
• JdbcUtils hardcodes some SQL statements: Resolved
• Add some comment for param of JdbcDialect createTable: Resolved
• Call convertJavaTimestampToTimeStamp in Array getter: Resolved
• Improve the debuggability for docker integration test: Resolved
• Clickhouse's JDBC throws `java.lang.IllegalArgumentException: Unknown data type: string` when write array string with Apache Spark scala: Resolved
• Upgrade docker-java to 3.3.5: Resolved
• upgrade guava from 18.0 to 19.0 for docker-integration-tests: Resolved
• Support java.sql.Types.NULL: Resolved
• Correctness issue of converting postgres INFINITIES timestamps: Resolved
• Java SPI Support for JDBC dialect registering: Resolved
• Add convertDateToDate like the existing convertTimestampToTimestamp for JdbcDialects: Resolved
• Save TimestampNTZType as DATETIME in MySQL: Resolved
• Read MySQL FLOAT as FloatType to keep consistent with the write side: Resolved
• Use MySQL Connector/J for MySQL DB instead of MariaDB Connector/J: Resolved
• Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME ZONE: Resolved
• Respect spark.sql.legacy.charVarcharAsString when casting jdbc type to catalyst type in jdbc: Resolved
• Audit MySQL ENUM/SET Types: Resolved
• Show a warning message about SQLException if `JDBCTableCatalog.loadTable` fails: Resolved
• Cleanup dead code in MySQLDialect.getCatalystType: Resolved
• Document Mapping Spark SQL Data Types from MySQL: Resolved
• Document Mapping Spark SQL Data Types to MySQL: Resolved
• Fix Postgres bit array issue 'Cannot cast to boolean': Resolved
• Make MySQL data source be able to read bit(n>1) as BinaryType like Postgres: Resolved
• Using SMALLINT to Write ShortType to MYSQL: Resolved
• Fix NPE when reading mysql bit array as LongType: Resolved
• Postgres: Map length unspecified bpchar to StringType: Resolved
• Postgres: Support multi dimensional array on the write side: Resolved
• Postgres: Add test for complex types: Resolved
• Postgres: Document Mapping Spark SQL Data Types from PostgreSQL: Resolved
• Special handling of JSON type for MySQL Connector/J 5.x: Resolved
• Parameterize JDBC Drivers for docker integration tests: Resolved
• Postgres: Document Mapping Spark SQL Data Types to PostgreSQL: Resolved
• Postgres: Support reading multidimensional arrays: Resolved
• Postgres: Add test and doc for Postgres special numeric values: Resolved
• Oracle: Support reading AnsiIntervalTypes: Resolved
• Oracle: Use VARCHAR2 instead of VARCHAR for VarcharType mapping: Resolved
• Oracle: Document Mapping Spark SQL Data Types from Oracle and add tests: Resolved
• Oracle: Document Mapping Spark SQL Data Types to Oracle: Resolved
• Handle negative scale for JDBC data sources: Resolved
• Truncate exceed decimals with scale first instead of precision from JDBC datasource: Resolved
• Replace getArrayDimension with updateExtraColumnMeta: Resolved
• Utilize java.sql.RowId.getBytes API directly for UTF8String: Resolved
• createTableColumnTypes need to be mapped to database types instead of using directly: Resolved
• MsSQLServer: Cannot find data type BYTE error: Resolved
• MsSQLServer: Document Mapping Spark SQL Data Types from Microsoft SQL Server and add tests: Resolved
• MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04: Resolved
• MsSQLServer: Document Mapping Spark SQL Data Types to Microsoft SQL Server: Resolved
• Make JdbcUtils.makeGetter handle reading time type as NTZ correctly: Resolved
• MsSQLServer: Map datatimeoffset to TimestampType: Resolved
• MsSQLServer: upgrade mssql.jdbc.version to 12.6.1.jre11: Resolved
• MsSQLServer: Fix the scope of spark.sql.legacy.mssqlserver.numericMapping.enabled: Resolved
• SQLServer does not support LIKE operator in binary comparison: Resolved
• DB2: Read SMALLINT as ShortType: Resolved
• Remove overrides DockerJDBCIntegrationSuite.connectionTimeout to make related tests configurable: Resolved
• DB2: Document Mapping Spark SQL Data Types from DB2 and add tests: Resolved
• DB2: Map BooleanType to Boolean instead of Char(1): Resolved
• Precision loss of reading jdbc time values: Resolved
• DB2: Document Mapping Spark SQL Data Types to DB2: Resolved
• Postgres: Map TimestampType to TIMESTAMP WITH TIME ZONE: Resolved
• Teradata: ByteType wrongly mapping to teradata byte(binary) type: Resolved
• Teradata: Document Type Conversion rules between Spark SQL and teradata: Resolved
• Upgrade MySQL & Postgres & mariadb docker image version: Resolved
• Derby: Retain as many significant digits as possible when decimal precision greater than 31: Resolved
• Upgrade docker-java to 3.3.6: Resolved
• Fix docker-integration-tests on Apple Chips: Resolved
• Update postgres docker image to 17.0: Resolved