[BEAM-12932] File based sink doesn't respect temp_location pipeline option

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.32.0
• Component/s: io-py-files
• Labels: documentation file-component python

Language

Python

• Assignee: Unassigned
• Reporter: Marek Suscak
• Created: 22/Sep/21 13:02
• Updated: 04/Jun/22 21:53

Description

Unlike FileIO module, the FileBasedSink module doesn't respect the temp_location pipeline option. This isn't expected and documented properly in the respective module's documentation.

I do understand that the temp_location is technically a pipeline option that's only required by the DataflowRunner but I think it might be worth it to pluck it out of the DataflowRunner and make it be a global pipeline option.

In addition, I couldn't info on what is the difference between WriteToFiles and WriteToText transforms. It seems that the former uses the FileBasedSink and thus correctly writes temporary files to the temp_location when specified but the latter doesn't do that.