[FLINK-30873] FileSystem Table API connector doesn't discover new files when partitions are specified

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.16.1
• Component/s: Connectors / FileSystem
• Assignee: Unassigned
• Reporter: Yaroslav Tkachenko
• Created: 01/Feb/23 21:20
• Updated: 09/Feb/23 10:35

Description

When source.monitor-interval is configured I expect the source connector to discover new files. However, if the source table was created with partitions (using PARTITIONED BY), it only discovers new files in the partition folders that existed during the startup; it doesn't discover new partitions. 

I believe the problem is here, only these file paths are monitored internally, no partition discovery is currently implemented.