[FLINK-27244] Support subdirectories with Hive tables

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: luoyuxia
• Reporter: luoyuxia
• Created: 14/Apr/22 05:29
• Updated: 07/Jul/22 13:04
• Resolved: 07/Jul/22 13:04

Description

Hive support to read recursive directory by setting the property 'set mapred.input.dir.recursive=true', and Spark also support [such behavior|https://stackoverflow.com/questions/42026043/how-to-recursively-read-hadoop-files-from-directory-using-spark].

For normal case, it won't happed for reading recursive directory. But it may happen in the following case:

I have a paritioned table `fact_tz` with partition day/hour

CREATE TABLE fact_tz(x int) PARTITIONED BY (ds STRING, hr STRING) 

Then I want to create an external table `fact_daily` refering to  `fact_tz`, but with a coarse-grained partition day. 

create external table fact_daily(x int) PARTITIONED BY (ds STRING) location 'fact_tz_localtion' ;

ALTER TABLE fact_daily ADD PARTITION (ds='1') location 'fact_tz_localtion/ds=1'

But it wll throw exception "Not a file: fact_tz_localtion/ds=1" when try to query this table `fact_daily` for it's the first level of the origin partition and is actually a directory .

 

 

Issue Links

• links to: GitHub Pull Request #19482