[FLINK-23010] HivePartitionFetcherContextBase::getComparablePartitionValueList can return partitions that don't exist

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0, 1.12.5, 1.13.2
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: Rui Li
• Reporter: Rui Li
• Created: 16/Jun/21 08:59
• Updated: 25/Jun/21 09:53
• Resolved: 25/Jun/21 09:53

Description

When consume order is CREATE_TIME, HivePartitionFetcherContextBase lists folders under table location to find partitions. This is wrong because HMS is the single source of truth to decide whether a partition exists for a hive table. Returning a non-existing partition will lead to unnecessary job FO when HiveContinuousPartitionFetcher fails to get the partition from HMS.

Issue Links

• links to: GitHub Pull Request #16182