Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47361

Improve JDBC data sources

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0
    • SQL

    Attachments

      Issue Links

        1.
        Add guidelines for timestamp mapping in JdbcDialect#getCatalystType Sub-task Resolved Kent Yao  
        2.
        SPARK-45561 causes mysql unsigned tinyint overflow Sub-task Resolved Kent Yao  
        3.
        Add a general mapping for TIME WITHOUT TIME ZONE to TimestampNTZType Sub-task Resolved Kent Yao  
        4.
        Support Oracle TIMESTAMP WITH LOCAL TIME ZONE type Sub-task Resolved Kent Yao  
        5.
        Remove timezone limitation for ORACLE TIMESTAMP WITH TIMEZONE Sub-task Resolved Kent Yao  
        6.
        Spark: Getting SQLException: Unsupported type -102 reading from Oracle Sub-task Resolved Kent Yao  
        7.
        Upgrade oracle jdbc driver to 23.3.0.23.09 Sub-task Resolved BingKun Pan  
        8.
        Use gvenzl/oracle-free:23.3-slim to reduce disk usage for docker it Sub-task Resolved Kent Yao  
        9.
        Revert SPARK-43049 Sub-task Resolved Kent Yao  
        10.
        Fix TimestampNTZ in Postgres Array Sub-task Resolved Kent Yao  
        11.
        Error in postgresql when pushing down filter by timestamp_ntz field Sub-task Resolved Pablo Langa Blanco  
        12.
        PostgresDialect distinguishes TIMESTAMP from TIMESTAMP_TZ Sub-task Resolved Kent Yao  
        13.
        Upgrade `PostgreSQL` JDBC driver to 42.7.2 and docker image to 16.2 Sub-task Resolved Dongjoon Hyun  
        14.
        Convert TINYINT catalyst properly in MySQL Dialect Sub-task Resolved Michael Zhang  
        15.
        Handle TIMESTAMP and DATETIME in MYSQLDialect Sub-task Resolved Kent Yao  
        16.
        Align the mapping ways of other unsigned numeric types with TINYINT UNSGINED in MySQLDialect Sub-task Resolved Kent Yao  
        17.
        Upgrade MySQL docker image version to 8.3.0 Sub-task Resolved Kent Yao  
        18.
        Improve test coverage for mysql bool synonyms Sub-task Resolved Kent Yao  
        19.
        Support TimestampNTZ for DB2 TIMESTAMP WITH TIME ZONE Sub-task Closed Unassigned  
        20.
        Bump DB2 Docker version to 11.5.8.0 Sub-task Resolved Kent Yao  
        21.
        Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1 Sub-task Resolved Kent Yao  
        22.
        Mapped TINYINT to ShortType for MsSqlServerDialect Sub-task Resolved Zerui Bao  
        23.
        Support TIMESTAMP WITH TIME ZONE for H2Dialect Sub-task Resolved Kent Yao  
        24.
        Add DatabricksDialect to handle SQL type conversion Sub-task Resolved Ivan Sadikov  
        25.
        Spark wrongly map the BOOLEAN Type to BIT(1) in Snowflake Sub-task Resolved Hayssam Saleh

        0%

        Original Estimate - 2h
        Remaining Estimate - 2h
        26.
        Redact JDBC url in errors and logs Sub-task Resolved Max Gekk  
        27.
        Port classifyException() in JDBC dialects on error classes Sub-task Resolved Max Gekk  
        28.
        Restore backward compatibility of JdbcDialect.classifyException Sub-task Resolved Max Gekk  
        29.
        Assign error classes/subclasses to JdbcUtils.classifyException Sub-task Resolved Max Gekk  
        30.
        JdbcUtils hardcodes some SQL statements Sub-task Resolved Jia Fan  
        31.
        Add some comment for param of JdbcDialect createTable Sub-task Resolved Jia Fan  
        32.
        Call convertJavaTimestampToTimeStamp in Array getter Sub-task Resolved Kent Yao  
        33.
        Improve the debuggability for docker integration test Sub-task Resolved Kent Yao  
        34.
        Clickhouse's JDBC throws `java.lang.IllegalArgumentException: Unknown data type: string` when write array string with Apache Spark scala Sub-task Resolved Nguyen Phan Huy  
        35.
        Upgrade docker-java to 3.3.5 Sub-task Resolved Kent Yao  
        36.
        upgrade guava from 18.0 to 19.0 for docker-integration-tests Sub-task Resolved Kent Yao  
        37.
        Support java.sql.Types.NULL Sub-task Resolved Kent Yao  
        38.
        Correctness issue of converting postgres INFINITIES timestamps Sub-task Resolved Kent Yao  
        39.
        Java SPI Support for JDBC dialect registering Sub-task Resolved Kent Yao  
        40.
        Add convertDateToDate like the existing convertTimestampToTimestamp for JdbcDialects Sub-task Resolved Kent Yao  
        41.
        Save TimestampNTZType as DATETIME in MySQL Sub-task Resolved Kent Yao  
        42.
        Read MySQL FLOAT as FloatType to keep consistent with the write side Sub-task Resolved Kent Yao  
        43.
        Use MySQL Connector/J for MySQL DB instead of MariaDB Connector/J Sub-task Resolved Kent Yao  
        44.
        Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME ZONE Sub-task Resolved Kent Yao  
        45.
        Respect spark.sql.legacy.charVarcharAsString when casting jdbc type to catalyst type in jdbc Sub-task Resolved Kent Yao  
        46.
        Audit MySQL ENUM/SET Types Sub-task Resolved Kent Yao  
        47.
        Show a warning message about SQLException if `JDBCTableCatalog.loadTable` fails Sub-task Resolved xleoken  
        48.
        Cleanup dead code in MySQLDialect.getCatalystType Sub-task Resolved Kent Yao  
        49.
        Document Mapping Spark SQL Data Types from MySQL Sub-task Resolved Kent Yao  
        50.
        Document Mapping Spark SQL Data Types to MySQL Sub-task Resolved Kent Yao  
        51.
        Fix Postgres bit array issue 'Cannot cast to boolean' Sub-task Resolved Kent Yao  
        52.
        Make MySQL data source be able to read bit(n>1) as BinaryType like Postgres Sub-task Resolved Kent Yao  
        53.
        Using SMALLINT to Write ShortType to MYSQL Sub-task Resolved Kent Yao  
        54.
        Fix NPE when reading mysql bit array as LongType Sub-task Resolved Kent Yao  
        55.
        Postgres: Map length unspecified bpchar to StringType Sub-task Resolved Kent Yao  
        56.
        Postgres: Support multi dimensional array on the write side Sub-task Resolved Kent Yao  
        57.
        Postgres: Add test for complex types Sub-task Resolved Kent Yao  
        58.
        Postgres: Document Mapping Spark SQL Data Types from PostgreSQL Sub-task Resolved Kent Yao  
        59.
        Special handling of JSON type for MySQL Connector/J 5.x Sub-task Resolved wangjunbo  
        60.
        Parameterize JDBC Drivers for docker integration tests Sub-task Resolved Kent Yao  
        61.
        Postgres: Document Mapping Spark SQL Data Types to PostgreSQL Sub-task Resolved Kent Yao  
        62.
        Postgres: Support reading multidimensional arrays Sub-task Resolved Kent Yao  
        63.
        Postgres: Add test and doc for Postgres special numeric values Sub-task Resolved Kent Yao  
        64.
        Oracle: Support reading AnsiIntervalTypes Sub-task Resolved Kent Yao  
        65.
        Oracle: Use VARCHAR2 instead of VARCHAR for VarcharType mapping Sub-task Resolved Kent Yao  
        66.
        Oracle: Document Mapping Spark SQL Data Types from Oracle and add tests Sub-task Resolved Kent Yao  
        67.
        Oracle: Document Mapping Spark SQL Data Types to Oracle Sub-task Resolved Kent Yao  
        68.
        Handle negative scale for JDBC data sources Sub-task Resolved Kent Yao  
        69.
        Truncate exceed decimals with scale first instead of precision from JDBC datasource Sub-task Resolved Kent Yao  
        70.
        Replace getArrayDimension with updateExtraColumnMeta Sub-task Resolved Kent Yao  
        71.
        Utilize java.sql.RowId.getBytes API directly for UTF8String Sub-task Open Unassigned  
        72.
        createTableColumnTypes need to be mapped to database types instead of using directly Sub-task Resolved Kent Yao  
        73.
        MsSQLServer: Cannot find data type BYTE error Sub-task Resolved Kent Yao  
        74.
        MsSQLServer: Document Mapping Spark SQL Data Types from Microsoft SQL Server and add tests Sub-task Resolved Kent Yao  
        75.
        MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04 Sub-task Resolved Kent Yao  
        76.
        MsSQLServer: Document Mapping Spark SQL Data Types to Microsoft SQL Server Sub-task Resolved Kent Yao  
        77.
        Make JdbcUtils.makeGetter handle reading time type as NTZ correctly Sub-task Resolved Kent Yao  
        78.
        MsSQLServer: Map datatimeoffset to TimestampType Sub-task Resolved Kent Yao  
        79.
        MsSQLServer: upgrade mssql.jdbc.version to 12.6.1.jre11 Sub-task Resolved Kent Yao  
        80.
        MsSQLServer: Fix the scope of spark.sql.legacy.mssqlserver.numericMapping.enabled Sub-task Resolved Kent Yao  
        81.
        SQLServer does not support LIKE operator in binary comparison Sub-task Resolved Stefan Bukorovic  
        82.
        DB2: Read SMALLINT as ShortType Sub-task Resolved Kent Yao  
        83.
        Remove overrides DockerJDBCIntegrationSuite.connectionTimeout to make related tests configurable Sub-task Resolved Kent Yao  
        84.
        DB2: Document Mapping Spark SQL Data Types from DB2 and add tests Sub-task Resolved Kent Yao  
        85.
        DB2: Map BooleanType to Boolean instead of Char(1) Sub-task Resolved Kent Yao  
        86.
        Precision loss of reading jdbc time values Sub-task Resolved Kent Yao  
        87.
        DB2: Document Mapping Spark SQL Data Types to DB2 Sub-task Resolved Kent Yao  
        88.
        Postgres: Map TimestampType to TIMESTAMP WITH TIME ZONE Sub-task Resolved Kent Yao  
        89.
        Teradata: ByteType wrongly mapping to teradata byte(binary) type Sub-task Resolved Kent Yao  
        90.
        Teradata: Document Type Conversion rules between Spark SQL and teradata Sub-task Resolved Kent Yao  

        Activity

          People

            yao Kent Yao
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 2h
                2h
                Remaining:
                Remaining Estimate - 2h
                2h
                Logged:
                Time Spent - Not Specified
                Not Specified