[HIVE-26350] IndexOutOfBoundsException when generating splits for external JDBC table with partition columns

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-2
• Component/s: CBO, JDBC storage handler
• Labels: pull-request-available
• Target Version/s: 4.0.0-alpha-2
• Assignee: Stamatis Zampetakis
• Reporter: Stamatis Zampetakis
• Created: 22/Jun/22 15:09
• Updated: 16/Nov/22 13:50
• Resolved: 02/Aug/22 10:57

Description

Create the following table in some JDBC database (e.g., Postgres).

CREATE TABLE country
(
    id   int,
    name varchar(20)
);

Create the following tables in Hive ensuring that the external JDBC table has the hive.sql.partitionColumn table property set.

CREATE TABLE city (id int);

CREATE EXTERNAL TABLE country
(
    id int,
    name varchar(20)
)
STORED BY                                          
'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (                                    
    "hive.sql.database.type" = "POSTGRES",
    "hive.sql.jdbc.driver" = "org.postgresql.Driver",
    "hive.sql.jdbc.url" = "jdbc:postgresql://localhost:5432/qtestDB",
    "hive.sql.dbcp.username" = "qtestuser",
    "hive.sql.dbcp.password" = "qtestpassword",
    "hive.sql.table" = "country",
    "hive.sql.partitionColumn" = "name",
    "hive.sql.numPartitions" = "2"
);

The query below fails with IndexOutOfBoundsException when the mapper scanning the JDBC table tries to generate the splits by exploiting the partitioning column.

select country.id from country cross join city;

The full stack trace is given below.

java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
        at java.util.ArrayList.rangeCheck(ArrayList.java:659) ~[?:1.8.0_261]
        at java.util.ArrayList.get(ArrayList.java:435) ~[?:1.8.0_261]
        at org.apache.hive.storage.jdbc.JdbcInputFormat.getSplits(JdbcInputFormat.java:102) [hive-jdbc-handler-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:564) [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:858) [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:263) [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:281) [tez-dag-0.10.1.jar:0.10.1]
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:272) [tez-dag-0.10.1.jar:0.10.1]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_261]
        at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_261]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682) [hadoop-common-3.1.0.jar:?]
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:272) [tez-dag-0.10.1.jar:0.10.1]
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:256) [tez-dag-0.10.1.jar:0.10.1]
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [guava-19.0.jar:?]
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [guava-19.0.jar:?]
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [guava-19.0.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_261]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_261]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]

Issue Links

• depends upon: HIVE-25945 Upgrade H2 database version to 2.1.210
• links to: GitHub Pull Request #3405
• links to: GitHub Pull Request #3470