[SPARK-43152] User-defined output metadata path (_spark_metadata)

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Structured Streaming
• Labels: pull-request-available

Language

scala

• Assignee: Unassigned
• Reporter: Wojciech Indyk
• Created: 15/Apr/23 17:29
• Updated: 07/Jan/24 00:21

Description

Currently path of metadata of output checkpoint is hardcoded. The metadata is saved in output path in _spark_metadata folder. It's a constraint on structure of paths, that might be easily relaxed by parametrisable path of output metadata. It would help with issues like changing output directory of spark streaming job, two jobs writing to the same output path or partition discovery. It would also help with separation of metadata from data in path structure.

The main target of change is getMetadataLogPath method in FileStreamSink. It has got access to sqlConf, so this method can override the default _spark_metadata path if defined it config. Introduction of parametrised metadata path needs reconsidering of meaning of  hasMetadata method in FileStreamSink.

Issue Links

• fixes: SPARK-30542 Two Spark structured streaming jobs cannot write to same base path
• links to: [Github] Pull Request #40821 (woj-i)
• links to: GitHub Pull Request #40821
• links to: GitHub Pull Request #43147