[SPARK-46339] Directory with number name should not be treated as metadata log

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.3, 3.4.2, 3.5.0
• Fix Version/s: 4.0.0, 3.5.1, 3.4.3
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 09/Dec/23 05:54
• Updated: 09/Dec/23 23:29
• Resolved: 09/Dec/23 23:22

Description

HDFSMetadataLog takes a metadata path as parameter. When it goes to retrieves all batches metadata, it calls `CheckpointFileManager.list` to get all files under the metadata path. However, currently all implementations of `CheckpointFileManager.list` returns all files/directories under the given path. So if there is a dictionary with name of batch number (a long value), the directory will be returned too and cause trouble when HDFSMetadataLog goes to read it.

Actually, `CheckpointFileManager.list` method clearly defines that it lists the "files" in a path. That's being said, current implementations don't follow the doc. We should fix it.

Issue Links

• links to: GitHub Pull Request #44272