[FLINK-35118] StreamingHiveSource cannot track tables that have more than 32,767 partitions

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.19.0
• Component/s: Connectors / Hive
• Assignee: Unassigned
• Reporter: roland
• Created: 16/Apr/24 02:43
• Updated: 16/Apr/24 02:43

Description

Description:

The Streaming Hive Source cannot track tables that have more than 32,767 partitions.

 Root Cause:

The Streaming Hive Source uses the following lines to get all partitions of a table:

(git hub link)

HivePartitionFetcherContextBase.java:

    @Override
    public List<ComparablePartitionValue> getComparablePartitionValueList() throws Exception {
        List<ComparablePartitionValue> partitionValueList = new ArrayList<>();
        switch (partitionOrder) {
            case PARTITION_NAME:
                List<String> partitionNames =
                        metaStoreClient.listPartitionNames(
                                tablePath.getDatabaseName(),
                                tablePath.getObjectName(),
                                Short.MAX_VALUE);
                for (String partitionName : partitionNames) {
                    partitionValueList.add(getComparablePartitionByName(partitionName));
                }
                break;
            case CREATE_TIME:
                Map<List<String>, Long> partValuesToCreateTime = new HashMap<>();
                partitionNames =
                        metaStoreClient.listPartitionNames(
                                tablePath.getDatabaseName(),
                                tablePath.getObjectName(),
                                Short.MAX_VALUE); 

Where the `metaStoreClient` is a wrapper of the `IMetaStoreClient`, and the function `listPartitionNames` can only list no more than `Short.MAX_VALUE` partitions, which is 32,767.

 

For tables that have more partitions, the source fails to track new partitions and read from it.