Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
1.18.0
-
None
-
None
-
We are using Flink 1.18.0 with the gs-plugin.
It is a rare bug but something we have noticed multiple times.
Description
We had a Flink pipeline that started to, all of a sudden, fail on a single subtask [Image 1]. It does not block the rest of the DAG checkpointing so the checkpoint barriers are continuing on.
We investigated the issue and found that the checkpoint was trying to write over and over and over. It retried writing the file thousands of times. And the issue persisted across checkpoints and savepoints but only failed for one specific file.
An example log:
Dec 10, 2024 6:06:05 PM com.google.cloud.hadoop.util.RetryHttpInitializer$LoggingResponseHandler handleResponse
INFO: Encountered status code 503 when sending PUT request to URL 'https://storage.googleapis.com/upload/storage/v1/b/<bucket>/o?ifGenerationMatch=0&name=flink-checkpoints/2394318276860454f7b6d1689f770796/shared/7d6bb60b-e0cf-4873-afc1-f2d785a4418e&uploadType=resumable&upload_id=<upload_id>'. Delegating to response handler for possible retry.
...
It is important to note that the file was in fact there. I am not sure if it was complete however it was not an .inprogress.file so I believe it was complete.
I even tried deleting the file in GCS and waiting for a new checkpoint to occur and the same issue persisted.
There is no issue when we restarted the job from a savepoint. There seems to be only an issue with a very specific file.
I also tried it locally. It had a 503 from this endpoint with the same upload_id
https://storage.googleapis.com/upload/storage/v1/<bucket>
However worked fine with this API (with a new upload_id)
https://storage.googleapis.com/<path>
I could not find the merged file on the Task Manager to try from the pod when it was failing.
