[SPARK-22814] JDBC support date/timestamp type as partitionColumn

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.2, 2.2.1
• Fix Version/s: 2.4.0
• Component/s: SQL
• Flags: Patch
• Assignee: Takeshi Yamamuro
• Reporter: Yuechen Chen
• Created: 16/Dec/17 05:38
• Updated: 19/May/19 13:43
• Resolved: 30/Jul/18 14:42

Description

In spark, you can partition MySQL queries by partitionColumn.
val df = (spark.read.jdbc(url=jdbcUrl,
table="employees",
columnName="emp_no",
lowerBound=1L,
upperBound=100000L,
numPartitions=100,
connectionProperties=connectionProperties))
display(df)

But, partitionColumn must be a numeric column from the table.
However, there are lots of table, which has no primary key, and has some date/timestamp indexes.

Issue Links

• relates to: SPARK-25453 OracleIntegrationSuite IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]
• links to: [Github] Pull Request #21834 (maropu)
• links to: GitHub Pull Request #21834