[BEAM-9082] "Socket closed" Spurious GRPC errors in Flink/Spark runner log output

• Type: Sub-task
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-flink, runner-spark
• Labels: portability-flink portability-spark
• Assignee: Unassigned
• Reporter: Kyle Weaver
• Created: 09/Jan/20 23:11
• Updated: 04/Jun/22 15:33

Description

We often see "Socket closed" errors on job shutdown, even though the pipeline has finished successfully. They are misleading and especially annoying at scale.

ERROR:root:Failed to read inputs in the data plane.
...
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "

{"created":"@1578597616.309419460","description":"Error received from peer ipv6:[::1]:37211","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}

"

Issue Links

• is related to: BEAM-3811 Add a CancellableQueue to make close methods safer
• relates to: BEAM-9793 Method not found: org.apache.beam.model.job_management.v1.ArtifactStagingService/ReverseArtifactRetrievalService