Minor Description
In the Python3 Avro IPC library, if the responder returns a response that doesn't match the schema, this error is swallowed and the response data is truncated. This results in a malformed response and a parsing error at the client end.
An example showing the problem is on GitHub.
The problem appears to come down to these lines in ipc.py:
except schema.AvroException as exn: error = AvroRemoteException(str(exn)) buffer_encoder = avro_io.BinaryEncoder(io.StringIO()) META_WRITER.write(response_metadata, buffer_encoder) buffer_encoder.write_boolean(True) self._WriteError(SYSTEM_ERROR_SCHEMA, error, buffer_encoder)
I can think of two ways to fix this:
- Error on the server side; or
- Pass the error through to the client.
Choice (1) seems to make logical sense since sending a non-conforming response is a bug in the server, not the client, but (2) might be easier to detect in real applications.
The way to do (2) would be something like the following, which I am happy to make a pull request for if it would be helpful:
# write response using local protocol META_WRITER.write(response_metadata, buffer_encoder) saved_pos = buffer_writer.tell() buffer_encoder.write_boolean(error is not None) if error is None: writers_schema = local_message.response try: self.write_response(writers_schema, response, buffer_encoder) except schema.AvroException as e: buffer_writer.seek(saved_pos) buffer_writer.truncate(saved_pos) buffer_encoder.write_boolean(True) writers_schema = local_message.errors self.write_error(writers_schema, e, buffer_encoder) else: writers_schema = local_message.errors self.write_error(writers_schema, error, buffer_encoder)