Uploaded image for project: 'Qpid Proton'
  1. Qpid Proton
  2. PROTON-2472

assert epoll.c:2519: poller_do_epoll: Assertion `!p->sched_ready_first' failed.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • proton-c-0.37.0
    • proton-c-0.37.0
    • proton-c
    • None

    Description

      Hit while running qpid-dispatch CI tests:

      https://master-jenkins-csb-entmqcl.apps.ocp4.prod.psi.redhat.com/job/dispatch/job/dispatch-rawhide-main/133/console

      Proton main @ 

      Revision 7c593cecb270760af7a05f854fed8078819fd4af

      Test run in Fedora Rawhide container

      test 18
            Start 18: system_tests_autolinks
      
      18: Test command: /usr/bin/python3.6 "/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/build/tests/run.py" "-m" "unittest" "-v" "system_tests_autolinks"
      18: Test timeout computed to be: 600
      18: test_auto_link_reattch (system_tests_autolinks.AutoLinkRetryTest) ... ok
      18: test_01_autolink_attach (system_tests_autolinks.AutolinkTest) ... ok
      18: test_02_autolink_credit (system_tests_autolinks.AutolinkTest) ... ok
      18: test_03_autolink_sender (system_tests_autolinks.AutolinkTest) ... ok
      18: test_04_autolink_receiver (system_tests_autolinks.AutolinkTest) ... ok
      18: test_05_inter_container_transfer (system_tests_autolinks.AutolinkTest) ... ok
      18: test_06_manage_autolinks (system_tests_autolinks.AutolinkTest) ... ok
      18: test_07_autolink_attach_with_ext_addr (system_tests_autolinks.AutolinkTest) ... ok
      18: test_08_autolink_sender_with_ext_addr (system_tests_autolinks.AutolinkTest) ... ok
      18: test_09_autolink_receiver_with_ext_addr (system_tests_autolinks.AutolinkTest) ... ok
      18: test_10_autolink_attach_to_listener (system_tests_autolinks.AutolinkTest) ... ok
      18: test_11_autolink_multiple_receivers_on_listener (system_tests_autolinks.AutolinkTest) ... ok
      18: 
      18: Router QDR output file:
      18: >>>>
      18: 2022-01-07 14:37:55.326051 +0000 AGENT (warning) Attribute 'dir' of entity 'autoLink' has been deprecated. Use 'direction' instead
      18: 2022-01-07 14:37:55.326406 +0000 AGENT (warning) Attribute 'dir' of entity 'autoLink' has been deprecated. Use 'direction' instead
      18: qdrouterd: /home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/proton/c/src/proactor/epoll.c:2481: poller_do_epoll: Assertion `!p->sched_ready_first' failed.
      18: 
      18: <<<<
      18: 
      18: Router QDR output file:
      18: >>>>
      18: 2022-01-07 14:37:55.326051 +0000 AGENT (warning) Attribute 'dir' of entity 'autoLink' has been deprecated. Use 'direction' instead
      18: 2022-01-07 14:37:55.326406 +0000 AGENT (warning) Attribute 'dir' of entity 'autoLink' has been deprecated. Use 'direction' instead
      18: qdrouterd: /home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/proton/c/src/proactor/epoll.c:2481: poller_do_epoll: Assertion `!p->sched_ready_first' failed.
      18: 
      18: <<<<
      18: 
      18: Router QDR command file:
      18: >>>>
      18: qdrouterd -c test-router.conf -I /home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/python
      18: pid=4172
      18: 
      18: <<<<
      18: 
      18: Router QDR log file tail:
      18: >>>>
      18: 2022-01-07 14:37:58.334748 +0000 ROUTER_CORE (info) Auto Link Deactivated 'autoLink/7' on connection myListener (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/route_control.c:137)
      18: 2022-01-07 14:37:58.334770 +0000 ROUTER_CORE (info) [C30] Connection Closed (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/connections.c:1607)
      18: 2022-01-07 14:37:58.334797 +0000 ROUTER_CORE (trace) Core action 'qdr_connection_process - remove from updated list' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.334821 +0000 ROUTER_CORE (debug) [C29][L67][D2787] Delivery outcome : is accepted (0x24) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:394)
      18: 2022-01-07 14:37:58.334841 +0000 ROUTER_CORE (trace) Core action 'qdr_connection_process - remove from updated list' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.334857 +0000 ROUTER_CORE (debug) [C29][L67][D2790] Delivery outcome : is accepted (0x24) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:394)
      18: 2022-01-07 14:37:58.334869 +0000 ROUTER_CORE (trace) Core action 'qdr_connection_process - remove from updated list' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.334883 +0000 ROUTER_CORE (debug) [C29][L67][D2792] Delivery outcome : is accepted (0x24) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:394)
      18: 2022-01-07 14:37:58.334900 +0000 ROUTER_CORE (trace) Core action 'update_delivery' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.334918 +0000 ROUTER_CORE (debug) [C29][L67][D2788] Delivery incref:    rc:3  qdr_delivery_anycast_update_CT - prevent peer from being freed (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:116)
      18: 2022-01-07 14:37:58.334949 +0000 ROUTER_CORE (trace) [C31][L63][D2793] :in qdr_delivery_unlink_peers_CT out: [C29][L67][D2788] (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:543)
      18: 2022-01-07 14:37:58.334962 +0000 ROUTER_CORE (debug) [C31][L63][D2793] Delivery decref_CT: rc:2 qdr_delivery_unlink_peers_CT - unlinked from peer (delivery) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.334974 +0000 ROUTER_CORE (debug) [C29][L67][D2788] Delivery decref_CT: rc:2 qdr_delivery_unlink_peers_CT - unlinked from delivery (peer) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.334986 +0000 ROUTER_CORE (debug) [C29][L67][D2788] Delivery incref:    rc:3  qdr_delivery_push_CT - add to updated list (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:116)
      18: 2022-01-07 14:37:58.335004 +0000 ROUTER_CORE (debug) [C31][L63][D2793] Delivery decref_CT: rc:1 qdr_delivery_anycast_update CT - dlv removed from unsettled (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335020 +0000 ROUTER_CORE (debug) [C29][L67][D2788] Delivery decref_CT: rc:2 qdr_delivery_anycast_update_CT - peer removed from unsettled (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335057 +0000 ROUTER_CORE (debug) [C29][L67][D2788] Delivery decref_CT: rc:1 qdr_delivery_anycast_update_CT - allow free of peer (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335068 +0000 ROUTER_CORE (debug) [C31][L63][D2793] Delivery decref_CT: rc:0 qdr_update_delivery_CT - remove from action (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335083 +0000 ROUTER_CORE (debug) [C31][L63][D2793] Delivery outcome : is accepted (0x24) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:394)
      18: 2022-01-07 14:37:58.335113 +0000 ROUTER_CORE (trace) Core action 'update_delivery' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.335130 +0000 ROUTER_CORE (debug) [C29][L67][D2791] Delivery incref:    rc:3  qdr_delivery_anycast_update_CT - prevent peer from being freed (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:116)
      18: 2022-01-07 14:37:58.335146 +0000 ROUTER_CORE (trace) [C31][L63][D2795] :in qdr_delivery_unlink_peers_CT out: [C29][L67][D2791] (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:543)
      18: 2022-01-07 14:37:58.335162 +0000 ROUTER_CORE (debug) [C31][L63][D2795] Delivery decref_CT: rc:2 qdr_delivery_unlink_peers_CT - unlinked from peer (delivery) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335190 +0000 ROUTER_CORE (debug) [C29][L67][D2791] Delivery decref_CT: rc:2 qdr_delivery_unlink_peers_CT - unlinked from delivery (peer) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335207 +0000 ROUTER_CORE (debug) [C29][L67][D2791] Delivery incref:    rc:3  qdr_delivery_push_CT - add to updated list (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:116)
      18: 2022-01-07 14:37:58.335230 +0000 ROUTER_CORE (debug) [C31][L63][D2795] Delivery decref_CT: rc:1 qdr_delivery_anycast_update CT - dlv removed from unsettled (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335244 +0000 ROUTER_CORE (debug) [C29][L67][D2791] Delivery decref_CT: rc:2 qdr_delivery_anycast_update_CT - peer removed from unsettled (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335263 +0000 ROUTER_CORE (debug) [C29][L67][D2791] Delivery decref_CT: rc:1 qdr_delivery_anycast_update_CT - allow free of peer (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335274 +0000 ROUTER_CORE (debug) [C31][L63][D2795] Delivery decref_CT: rc:0 qdr_update_delivery_CT - remove from action (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.335303 +0000 ROUTER_CORE (debug) [C31][L63][D2795] Delivery outcome : is accepted (0x24) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:394)
      18: 2022-01-07 14:37:58.335320 +0000 ROUTER_CORE (trace) Core action 'link_detach' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.335344 +0000 ROUTER_CORE (info) [C31][L63] Link lost: del=5 presett=0 psdrop=0 acc=5 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/connections.c:1116)
      18: 2022-01-07 14:37:58.335364 +0000 ROUTER_CORE (trace) Core action 'link_detach' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.335539 +0000 ROUTER_CORE (info) Auto Link Activation Failed.  'autoLink/6' on connection myListener (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/route_control.c:137)
      18: 2022-01-07 14:37:58.335594 +0000 ROUTER_CORE (info) [C31][L61] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/connections.c:1116)
      18: 2022-01-07 14:37:58.335614 +0000 ROUTER_CORE (trace) Core action 'link_detach' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.335685 +0000 ROUTER_CORE (info) Auto Link Activation Failed.  'autoLink/7' on connection myListener (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/route_control.c:137)
      18: 2022-01-07 14:37:58.335712 +0000 ROUTER_CORE (info) [C31][L62] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/connections.c:1116)
      18: 2022-01-07 14:37:58.335771 +0000 ROUTER_CORE (trace) Core action 'connection_closed' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.335812 +0000 ROUTER_CORE (info) Auto Link Deactivated 'autoLink/6' on connection myListener (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/route_control.c:137)
      18: 2022-01-07 14:37:58.335826 +0000 ROUTER_CORE (info) Auto Link Deactivated 'autoLink/7' on connection myListener (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/route_control.c:137)
      18: 2022-01-07 14:37:58.335888 +0000 ROUTER_CORE (info) [C31] Connection Closed (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/connections.c:1607)
      18: 2022-01-07 14:37:58.336522 +0000 ROUTER_CORE (trace) Core action 'link_detach' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.336725 +0000 ROUTER_CORE (debug) [C29][L67][D2788] Delivery outcome : is accepted (0x24) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:394)
      18: 2022-01-07 14:37:58.336766 +0000 ROUTER_CORE (debug) [C29][L67][D2788] Delivery decref_CT: rc:0 qdr_link_cleanup_deliveries_CT - remove from updated list (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.336799 +0000 ROUTER_CORE (debug) [C29][L67][D2791] Delivery outcome : is accepted (0x24) (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:394)
      18: 2022-01-07 14:37:58.336818 +0000 ROUTER_CORE (debug) [C29][L67][D2791] Delivery decref_CT: rc:0 qdr_link_cleanup_deliveries_CT - remove from updated list (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/delivery.c:641)
      18: 2022-01-07 14:37:58.336845 +0000 ROUTER_CORE (info) [C29][L67] Link lost: del=10 presett=0 psdrop=0 acc=10 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/connections.c:1116)
      18: 2022-01-07 14:37:58.336864 +0000 ROUTER_CORE (trace) Core action 'connection_closed' (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/router_core_thread.c:235)
      18: 2022-01-07 14:37:58.336882 +0000 ROUTER_CORE (info) [C29] Connection Closed (/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/src/router_core/connections.c:1607)
      18: 
      18: <<<<
      18: ERROR
      18: test_auto_link_attach_detach_reattch (system_tests_autolinks.DetachAfterAttachTest) ... ok
      18: test_name_collision (system_tests_autolinks.NameCollisionTest) ... ok
      18: test_two_router_waypoint_no_tenant_external_addr_phase (system_tests_autolinks.WaypointReceiverPhaseTest) ... ok
      18: 
      18: ======================================================================
      18: ERROR: tearDownClass (system_tests_autolinks.AutolinkTest)
      18: ----------------------------------------------------------------------
      18: Traceback (most recent call last):
      18:   File "/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/tests/system_test.py", line 836, in tearDownClass
      18:     cls.tester.teardown()
      18:   File "/home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/tests/system_test.py", line 779, in teardown
      18:     raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
      18: RuntimeError: Errors during teardown: 
      18: 
      18: Process 4172 error: exit code -6, expected 0
      18: qdrouterd -c test-router.conf -I /home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/python
      18: /home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/dispatch/build/tests/system_test.dir/system_tests_autolinks/AutolinkTest/setUpClass/test-router-3.cmd
      18: >>>>
      18: 2022-01-07 14:37:55.326051 +0000 AGENT (warning) Attribute 'dir' of entity 'autoLink' has been deprecated. Use 'direction' instead
      18: 2022-01-07 14:37:55.326406 +0000 AGENT (warning) Attribute 'dir' of entity 'autoLink' has been deprecated. Use 'direction' instead
      18: qdrouterd: /home/jenkins/workspace/dispatch/dispatch-rhel8-main/source/proton/c/src/proactor/epoll.c:2481: poller_do_epoll: Assertion `!p->sched_ready_first' failed.
      18: <<<<
      18: 
      18: ----------------------------------------------------------------------
      18: Ran 15 tests in 19.943s
      18: 
      18: FAILED (errors=1)
      18/76 Test #18: system_tests_autolinks ............................***Failed   20.19 sec
      

      Attachments

        Issue Links

          Activity

            People

              cliffjansen Clifford Jansen
              kgiusti Ken Giusti
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: