[HIVE-20720] Add partition column option to JDBC handler

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: StorageHandler
• Target Version/s: 4.0.0
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 10/Oct/18 04:12
• Updated: 27/Feb/24 22:23
• Resolved: 19/Oct/18 00:26

Description

Currently JdbcStorageHandler does not split input in Tez. The reason is numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. SQLStdAuth) is in use. User ends up always use 1 split.

We need to rely on this new feature if we want to support multi-splits. Here is my proposal:
1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in tblproperties if user want to split jdbc data source. In case lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min query to get this in planner. We can currently limit partitionColumn to only numeric/date/timestamp column for simplicity
2. If partitionColumn/numPartitions are not specified, don't split input
3. Splits are equal intervals without respect to data distribution
4. There is also a "hive.sql.query.split" flag vetos the split (can be set manually or automatically by calcite)
5. If partitionColumn is not defined, but numPartitions is defined, use original limit/offset logic (however, don't rely on numSplit).

Issue Links

• links to: RB