[HUDI-1723] DFSPathSelector skips files with the same modify date when read up to source limit

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: deltastreamer
• Labels: pull-request-available sev:critical user-support-issues
• Assignee: Shiyan Xu
• Reporter: Shiyan Xu
• Created: 26/Mar/21 08:43
• Updated: 25/May/21 15:14
• Resolved: 25/May/21 15:14

Description

org.apache.hudi.utilities.sources.helpers.DFSPathSelector#listEligibleFiles filters the input files based on last saved checkpoint, which was the modification date from last read file. However, the last read file's modification date could be duplicated for multiple files and resulted in skipping a few of them when reading up to source limit. An illustration is shown in the attached picture.

Issue Links

• relates to: HUDI-1896 HudiStreamer Source for cloud object stores
• links to: GitHub Pull Request #2845