[BEAM-14315] Update fileio.MatchContinuously to allow reading already read files with a new timestamp

• Type: New Feature
• Status: In Progress
• Priority: P2
• Resolution: Unresolved
• Component/s: io-py-common
• Assignee: Yi Hu
• Reporter: Yi Hu
• Created: 15/Apr/22 20:08
• Updated: 05/Jun/22 01:14

Description

This will be the Python counterpart of BEAM-14267.

For fileio.MatchContinuously, we want to add an option to choose to consider a file new if it has a different timestamp from an existing file, even if the file itself has the same name.

See the following design doc for more detail:

https://docs.google.com/document/d/1xnacyLGNh6rbPGgTAh5D1gZVR8rHUBsMMRV3YkvlL08/edit?usp=sharing&resourcekey=0-be0uF-DdmwAz6Vg4Li9FNw

Issue Links

• is blocked by: BEAM-14497 Python Reshuffle holds elements
• is blocked by: BEAM-14498 Python sdk's PeriodicImpulse generates a bounded PCollection
• is blocked by: BEAM-14314 Add last_updated field in filesystem.FileMetaData
• links to: GitHub Pull Request #17604