[SPARK-30542] Two Spark structured streaming jobs cannot write to same base path

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.3.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Sivakumar
• Created: 17/Jan/20 07:44
• Updated: 18/Apr/23 15:12
• Resolved: 23/Jan/20 02:18

Description

Hi All,

Spark Structured Streaming doesn't allow two structured streaming jobs to write data to the same base directory which is possible with using dstreams.

As _spark_metadata directory will be created by default for one job, second job cannot use the same directory as base path as already _spark_metadata directory is created by other job, It is throwing exception.

Is there any workaround for this, other than creating separate base path's for both the jobs.

Is it possible to create the _spark_metadata directory else where or disable without any data loss.

If I had to change the base path for both the jobs, then my whole framework will get impacted, So i don't want to do that.

 

Issue Links

• is fixed by: SPARK-43152 User-defined output metadata path (_spark_metadata)