[BEAM-7476] Datastore write failures with "[Errno 32] Broken pipe"

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Affects Version/s: 2.11.0
• Fix Version/s: 2.16.0
• Component/s: io-py-gcp
• Labels: pull-request-available
• Environment: dataflow python 2.7
• External issue URL: https://github.com/apache/beam/pull/8346
• Assignee: Dmytro Sadovnychyi
• Reporter: Dmytro Sadovnychyi
• Created: 01/Jun/19 11:13
• Updated: 16/May/20 13:26
• Resolved: 09/Sep/19 20:46

Description

We are getting lots of Broken pipe errors and it's only a matter of luck for write to succeed. It's been happening for months.

Partial stack trace:

File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/helper.py", line 225, in commit
response = datastore.commit(request)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 140, in commit
datastore_pb2.CommitResponse)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 199, in _call_method
method='POST', body=payload, headers=headers)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/transport.py", line 169, in new_request
redirections, connection_type)
File "/usr/local/lib/python2.7/dist-packages/httplib2/_init_.py", line 1609, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/local/lib/python2.7/dist-packages/httplib2/_init_.py", line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/local/lib/python2.7/dist-packages/httplib2/_init_.py", line 1273, in _conn_request
conn.request(method, request_uri, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1042, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1082, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 882, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 858, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/ssl.py", line 753, in sendall
v = self.send(data[count:])
File "/usr/lib/python2.7/ssl.py", line 719, in send
v = self._sslobj.write(data)
RuntimeError: error: [Errno 32] Broken pipe [while running 'Groups to datastore/Write Mutation to Datastore']

Workaround: https://github.com/apache/beam/pull/8346

Issue Links

• links to: GitHub Pull Request #8346