My workflow : KAFKA -> Dataflow streaming -> BigQuery
Given that having low-latency isn't important in my case, I use FILE_LOADS to reduce the costs. I'm using BigQueryIO.Write with a DynamicDestination, which is a table with the current hour as a suffix.
This BigQueryIO.Write is configured like this :
The first table is successfully created and is written to. But then the following tables are never created and I get these exceptions:
The CreateDisposition used is CREATE_NEVER, contrary as CREATE_IF_NEEDED as specified.