Details
-
Bug
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When streaming inserts into BigQuery using BigQueryIO.Write, if there is an error other than a row insertion error (e.g. an IOException), BigQueryIO assumes this must be a rate limit error, and so enters an infinite loop of retries. This is logical for rate limit errors, but other types of error may also appear.
One example is a "Not found" error for the BigQuery table. This can happen if the table was originally created by BigQueryIO.Write (where CreateDisposition is CREATE_IF_NEEDED), but has since been deleted (since created tables are cached). This may be more likely to happen in a long-term streaming job. The infinite loop of retries does not help, as table creation is not retried, only row insertion. The table either needs to be created with an external process, or the pipeline needs to be restarted (thereby clearing the cache).
This can be the case regardless of setting InsertRetryPolicy, as these errors are not insertion errors. As a result, we see logs such as "INFO: BigQuery insertAll error, retrying: Not found: Table <project>:<dataset>.<table>" even if we have set InsertRetryPolicy to "neverRetry()", which is confusing behaviour. I expect similar issues to occur for other types of error (e.g. no response from BigQuery API).
To recreate this issue, you can create a pipeline which inserts into BigQuery using BigQueryIO, where the table does not exist beforehand but should be created by BigQueryIO (i.e. CreateDisposition = CREATE_IF_NEEDED). Then mock BigQueryServicesImpl's call to create the BigQuery table, causing no table to be created (I did this in a brute force method by creating my own BigQueryServicesImpl and feeding in using ".withTestServices()"). The pipeline will enter an infinite loop, logging "INFO: BigQuery insertAll error, retrying: Not found: Table <project>:<dataset>.<table>".
One suggestion to avoid this is to add another retry policy, one to control retries for non-insertion errors. This could be optional for users and effective here. An alternative/additional option could be to check for "table not found" errors in this clause and if encountered, retry table creation before next retrying insertion.