[HUDI-2271] Follow-up items for timeline-server-based marker files

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Ethan Guo
• Reporter: Ethan Guo
• Created: 03/Aug/21 08:40
• Updated: 10/Mar/23 01:50

Description

Follow-up items after HUDI-1138:

• Implement upgrade/downgrade, rollback regarding markers
  • Rollback should be based on the markers in the file system, with marker type indicated by MARKERS.type, or assumed to be "direct", if no MARKERS.type
  • New write follows the `hoodie.write.markers.type` config to create markers
• Verify multiple concurrent writers with timeline-server-based markers
• Let's ensure the parallelism passed for local engine context is reasonable
• Fix `Task not serializable` due to new APIs in FSUtils for recursive, level by level listing (`java.io.NotSerializableException: org.apache.hudi.common.fs.FSUtils$$Lambda$4224/1845791682`)
• Optimize fs.exists(dirPath) call in marker creation in DirectMarkerFiles
• Optimize future timeouts due to I/O failure in the marker handler
• Test new marker strategy for all engines (spark, flink and java) w/o any additional changes

Sub-Tasks

• Fix `Task not serializable` due to new APIs in FSUtils for marker mechanism: Resolved
• Enable timeline-server-based markers for Spark structured streaming: Open