[HUDI-6803] Create marker file for every log file

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0-beta1
• Epic Link: Non-blocking concurrency control
• Assignee: Unassigned
• Reporter: Jing Zhang
• Created: 30/Aug/23 02:04
• Updated: 11/Oct/23 01:12
• Resolved: 11/Oct/23 01:12

Description

The issue aims to introduce marker file for log files , which is similar with HUDI-1517.
However, the pr of HUDI-1517 (#4913) is reverted. Refer to #9090 and HUDI-6484 for more info about why this needs to be reverted.
And after solve HUDI-5672, HUDI-6725, HUDI-6743, HUDI-6742, we could avoid the problem referred in #9090 and HUDI-6484. Then we could introduce marker file for log files in a more simpler way.