[IMPALA-5741] External JDBC Read Support

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Distributed Exec
• Epic Color: ghx-label-1
• Assignee: Wenzhe Zhou
• Reporter: Istvan Vajnorak
• Created: 28/Jul/17 17:20
• Updated: 25/Jan/24 04:31

Description

In Hive there is a generic JDBC Storage handler that would be beneficial to be replicated into Impala. There are several workloads out that that could make good use of it.

The hive version of the handler is tracked under:
https://issues.apache.org/jira/browse/HIVE-1555

Please evaluate the possibility of including this into the roadmap at some point.

Issue Links

• Blocked: IMPALA-7131 Support external data sources in local catalog mode
• is duplicated by: IMPALA-8210 Support reading/writing tiny RDBMS tables
• is related to: HIVE-1555 JDBC Storage Handler

Sub-Tasks

• DataSource objects are not persistent: Resolved
• DataSourceScanNode drop some returned rows if FLAGS_data_source_batch_size is greater than default value: Resolved
• Improve count star performance for external data source: Resolved
• Auto Ship Data Source library in Impala: Resolved
• Detect available jdbc drivers without restarting Impala: Resolved
• Securing dbcp.password for JDBC external data source: Resolved
• Add jdbc related properties to JDBC data source object: Closed
• Support different schemes for jdbc driver url when creating external jdbc table: Resolved
• Scripts to run unit-tests of external jdbc table for MySQL: Resolved
• Caching of JDBC drivers in local storage for external JDBC tables: Resolved
• Support Impala to Impala federation: Resolved
• Support date data type for predicates for external data source table: Resolved
• Allow Postgres server to accept remote connections from hosts in same subnet: Resolved
• Support reading hive "information_schema" views in Impala: Open
• Support setting query options for Impala in the JdbcStorageConfig: Resolved
• LdapHS2Test.testImpalaExtJdbcTables failed for jdk17 builds: Resolved
• Add test cases for MySQL and Postgres to set additional properties with jdbc.properties: Closed
• Update Impala document to cover external jdbc table: Open
• Fix unit-test code JdbcDataSourceTest.java: Open
• Change JDBC table creation syntax: Resolved
• Support ALTER TABLE for JDBC tables: Resolved
• Support timestamp data type for predicates for external data source table: Resolved
• TestExtDataSources::test_catalogd_ha_failover seems flaky: Resolved
• Support password protected keystore: Open
• Avoid JDBC table to be set as transactional table: Resolved
• Generate distributed plan for query accessing multiple JDBC tables: Open
• Run TPCH/TPCDS queries for external JDBC tables: Resolved
• Fix decimal data type for external JDBC table: Resolved
• Mask dbcp.password table property of JDBC table for 'desc formatted' and 'show create table' commands: Resolved
• Performance improvement for impala-impala federation: Open
• Support for Hive JDBC Storage handler tables: Open
• Encrypt password in JDBC table properties when saving into HMS DB: Open
• Fix test_tpcds_queries.py/TestTpcdsQueryForJdbcTables.test_tpcds-decimal_v2-q80a failure: Resolved
• Add query option to keep DBCP DataSource objects in cache for longer time: Resolved
• Add TExprs of the columns to TBinaryPredicate for JDBC table: Open