[BEAM-13162] BigQueryIO Storage Write API method - suppress logging API errors for known use cases.

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.33.0
• Component/s: io-java-gcp
• Assignee: Unassigned
• Reporter: Sergei Lilichenko
• Created: 01/Nov/21 19:51
• Updated: 04/Jun/22 21:53

Description

Many BigQuery Storage API errors get logged at ERROR level even though they are expected and handled by the Beam's transform. They should be suppressed, or output at DEBUG/INFO level to differentiate from abnormal behavior.

Example:

{
 "jsonPayload":

{ "message": "Got error io.grpc.StatusRuntimeException: ALREADY_EXISTS: The offset is within stream, expected offset 52125, received 51264 Entity: projects/event-processing-demo/datasets/bigquery_io/tables/events/streams/CiQ2MmZlOTFjNS0wMDAwLTIzNTItOWMxYS01ODI0MjlhOWRiOGM closing projects/event-processing-demo/datasets/bigquery_io/tables/events/streams/CiQ2MmZlOTFjNS0wMDAwLTIzNTItOWMxYS01ODI0MjlhOWRiOGM", "step": "Save Rows to BigQuery/StorageApiLoads/StorageApiWriteSharded/Write Records", "worker": "data-processing-streaming-10250846-kps9-harness-hnj3", "stage": "P6", "job": "2021-10-25_08_46_20-4863073108555028756", "thread": "258", "work": "3800026c09938fac-5cf2f68daacdd", "logger": "org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords" }

,
 "resource": {
 "type": "dataflow_step",
 "labels":

{ "project_id": "event-processing-demo", "step_id": "Save Rows to BigQuery/StorageApiLoads/StorageApiWriteSharded/Write Records", "job_id": "2021-10-25_08_46_20-4863073108555028756", "region": "us-central1", "job_name": "data-processing-streaming-storage-write-api-200-2" }

},
 "timestamp": "2021-10-25T15:57:43.617Z",
 "severity": "ERROR",
 "labels":

{ "compute.googleapis.com/resource_type": "instance", "dataflow.googleapis.com/job_name": "data-processing-streaming-storage-write-api-200-2", "dataflow.googleapis.com/log_type": "supportability", "compute.googleapis.com/resource_id": "6365565973148954518", "dataflow.googleapis.com/job_id": "2021-10-25_08_46_20-4863073108555028756", "dataflow.googleapis.com/region": "us-central1", "compute.googleapis.com/resource_name": "data-processing-streaming-10250846-kps9-harness-hnj3" }

,
 "logName": "projects/event-processing-demo/logs/dataflow.googleapis.com%2Fworker",
 "receiveTimestamp": "2021-10-25T15:57:47.557344849Z"
 }

Additional exceptions that fit this category:

• Got error io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: server_shutting_down closing
• Got error com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Closing the stream because server is restarted. This is expected and client is advised to reconnect.
• Got error io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is closed due to com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Closing the stream because server is restarted. This is expected and client is advised to reconnect.