Major Description
The Qpid Interop Test large content test repeatedly fails when run against a single-node dispatch router (dispatch config file attached).
The test that reproduces this most readily but without too much length is the following:
python -m qpid_interop_test.amqp_large_content_test --include-shim ProtonCpp --include-type list
WARNING: Rhea Javascript shims not found
Test Broker: qpid-dispatch-router v.1.0.0 on <platform not found>
test_list_ProtonCpp->ProtonCpp (__main__.ListTestCase) ... ERROR
======================================================================
ERROR: test_list_ProtonCpp->ProtonCpp (__main__.ListTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/kvdr/RedHat/install/lib/python2.7/site-packages/qpid_interop_test/amqp_large_content_test.py", line 196, in inner_test_method
timeout)
File "/home/kvdr/RedHat/install/lib/python2.7/site-packages/qpid_interop_test/amqp_large_content_test.py", line 121, in run_test
raise InteropTestError('Receive shim \'%s\':\n%s' % (receive_shim.NAME, receive_obj))
InteropTestError: Receive shim 'ProtonCpp':
amqp_large_content_test receiver error: receiver read failure
----------------------------------------------------------------------
Ran 1 test in 0.801s
FAILED (errors=1)
The router left the following message:
SERVER (info) Connection from ::1:57020 (to ::1:amqp) failed: amqp:connection:framing-error connection aborted
The attached capture file shows a typical observable sequence of events on the wire that lead up to the failure. The test that created this error consists of 4 messages:
- A 1MB message consisting of a list containing a single 1MB string (delivery-id 0);
- A 1MB message consisting of a list containing 16 64kB strings (delivery-id 1);
- A 10MB message consisting of a list containing a single 10MB string (delivery-id 2);
- A 10MB message consisting of a list containing 16 655MB strings (delivery-id 3).
The following is a summary of what transpires:
- Frame 1527: The sender completes sending the last message (delivery-id 3) to the router, and closes the connection (without waiting for dispositions). At this point, the receiver is in the process of being sent message-id 2.
- Frame 1539: Last transfer for message-id 2 from dispatch to receiver.
- Frame 1545 - 1598: Message-id 3 starts being sent to receiver.
- Frame 1600: Dispatch router returns close to sender (initiated in frame 1527). No errors or arguments.
- Frame 1605: Receiver sends flow, disposition for delivery-id 2 (completed in frame 1539).
- Frame 1607 - 1618: Continue to send message with delivery-id 3 to receiver.
- Frame 1619: Transfer aborted. A transfer performative with more=False, Settled=True, Aborted=True.
- Frame 1622: Close sent from Dispatch to Receiver with Condition: amqp:connection:framing-error, Description: connection aborted.
All instances of this error have occurred between dispatch router and the receiver once the sender has closed its connection.