Description
The links for a session are kept in a proton list object, which is essentially an array.
Removing a link from this list involves first searching for the link by iterating over the array comparing it with the desired object using the comparator defined for the class, then cleaning up that element and shuffling up any remaining elements.
In the case of pn_session_free, it loops calling on_link_free on the first element in the list (which involves among other things removing it from the list as described above) until there are no more elements in the list. Looking up the link (though actually unnecessary in the overall scheme of things) is not too expensive as it will always be the first item, however each of the items in the list (which is large) is then shuffled up, which will be relatively expensive.
The removed links are also added to the session free list (which is also a proton list type) and then removed from that again in pn_link_finalize.
(Also, pni_link_remove is then called in the pn_link_finalize (as well as having been called in the pn_link_free), though since the list should be emoty by this point that probably doesn't actually matter, its just looks odd).
In more practical terms, this inefficiency can prevent applications using proton form properly handling sessions with large numbers of links on them. In the case of the dispatch router, if you have N connections (where N == number of work threads) with a large number of links (say 60,000) and suddenly all N connections are disconnected, the worker threads will all be stuck processing pn_session_free for each of the disconnected connections leaving the router unresponsive to any other task for the duration.
Thread 5 (Thread 0x7f920cbb6700 (LWP 3624)): #0 0x00007f9216372e79 in pn_object_reify (object=0x7f91d0e63480) at /home/gordon/projects/proton/c/src/core/object/object.c:214 #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>, a=0x7f91d0e63480, b=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/object/object.c:158 #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>, a=<optimized out>, b=b@entry=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/object/object.c:169 #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f91f801dc40, value=value@entry=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/object/list.c:88 #4 0x00007f92163737a9 in pn_list_remove (list=0x7f91f801dc40, value=value@entry=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/object/list.c:99 #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f91f804b870, link=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/engine.c:311 #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at /home/gordon/projects/proton/c/src/core/engine.c:1119 #7 pn_link_finalize (object=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/engine.c:1098 #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>, clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f91f95f9300) at /home/gordon/projects/proton/c/src/core/object/object.c:95 #9 0x00007f921637326f in pn_decref (object=<optimized out>) at /home/gordon/projects/proton/c/src/core/object/object.c:253 #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at /home/gordon/projects/proton/c/src/core/engine.c:366 #11 0x00007f921637f086 in pn_session_free (session=0x7f91f804b870) at /home/gordon/projects/proton/c/src/core/engine.c:271 #12 0x00007f921637f165 in pn_connection_release (connection=0x7f9204050fa0) at /home/gordon/projects/proton/c/src/core/engine.c:125 #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f9204050fa0) at /home/gordon/projects/proton/c/src/core/engine.c:146 #14 0x00007f921637c296 in pn_connection_driver_destroy (d=d@entry=0x7f920405b4b8) at /home/gordon/projects/proton/c/src/core/connection_driver.c:94 #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f920405af10) at /home/gordon/projects/proton/c/src/proactor/epoll.c:882 #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at /home/gordon/projects/proton/c/src/proactor/epoll.c:898 #17 0x00007f921635cb82 in pconnection_done (pc=0x7f920405af10) at /home/gordon/projects/proton/c/src/proactor/epoll.c:1037 #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f920405b4b0) at /home/gordon/projects/proton/c/src/proactor/epoll.c:2159 #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at /home/gordon/projects/dispatch/src/server.c:1037 #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f920d3b7700 (LWP 3623)): #0 0x00007f9216372e79 in pn_object_reify (object=0x7f9205c699d0) at /home/gordon/projects/proton/c/src/core/object/object.c:214 #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>, a=0x7f9205c699d0, b=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/object/object.c:158 #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>, a=<optimized out>, b=b@entry=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/object/object.c:169 #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f920400bec0, value=value@entry=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/object/list.c:88 #4 0x00007f92163737a9 in pn_list_remove (list=0x7f920400bec0, value=value@entry=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/object/list.c:99 #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f9204077fd0, link=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/engine.c:311 #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at /home/gordon/projects/proton/c/src/core/engine.c:1119 #7 pn_link_finalize (object=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/engine.c:1098 #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>, clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f9201a87ca0) at /home/gordon/projects/proton/c/src/core/object/object.c:95 #9 0x00007f921637326f in pn_decref (object=<optimized out>) at /home/gordon/projects/proton/c/src/core/object/object.c:253 #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at /home/gordon/projects/proton/c/src/core/engine.c:366 #11 0x00007f921637f086 in pn_session_free (session=0x7f9204077fd0) at /home/gordon/projects/proton/c/src/core/engine.c:271 #12 0x00007f921637f165 in pn_connection_release (connection=0x7f9200053520) at /home/gordon/projects/proton/c/src/core/engine.c:125 #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f9200053520) at /home/gordon/projects/proton/c/src/core/engine.c:146 #14 0x00007f921637c296 in pn_connection_driver_destroy (d=d@entry=0x7f9200054258) at /home/gordon/projects/proton/c/src/core/connection_driver.c:94 #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f9200053cb0) at /home/gordon/projects/proton/c/src/proactor/epoll.c:882 #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at /home/gordon/projects/proton/c/src/proactor/epoll.c:898 #17 0x00007f921635cb82 in pconnection_done (pc=0x7f9200053cb0) at /home/gordon/projects/proton/c/src/proactor/epoll.c:1037 #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f9200054250) at /home/gordon/projects/proton/c/src/proactor/epoll.c:2159 #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at /home/gordon/projects/dispatch/src/server.c:1037 #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f920dbb8700 (LWP 3622)): #0 0x00007f9216372e79 in pn_object_reify (object=0x7f91f9de8050) at /home/gordon/projects/proton/c/src/core/object/object.c:214 #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>, a=0x7f91f9de8050, b=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/object/object.c:158 #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>, a=<optimized out>, b=b@entry=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/object/object.c:169 #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f9200036900, value=value@entry=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/object/list.c:88 #4 0x00007f92163737a9 in pn_list_remove (list=0x7f9200036900, value=value@entry=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/object/list.c:99 #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f920005a670, link=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/engine.c:311 #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at /home/gordon/projects/proton/c/src/core/engine.c:1119 #7 pn_link_finalize (object=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/engine.c:1098 #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>, clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f92059f0710) at /home/gordon/projects/proton/c/src/core/object/object.c:95 #9 0x00007f921637326f in pn_decref (object=<optimized out>) at /home/gordon/projects/proton/c/src/core/object/object.c:253 #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at /home/gordon/projects/proton/c/src/core/engine.c:366 #11 0x00007f921637f086 in pn_session_free (session=0x7f920005a670) at /home/gordon/projects/proton/c/src/core/engine.c:271 #12 0x00007f921637f165 in pn_connection_release (connection=0x7f91f8045d00) at /home/gordon/projects/proton/c/src/core/engine.c:125 #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f91f8045d00) at /home/gordon/projects/proton/c/src/core/engine.c:146 #14 0x00007f921637c296 in pn_connection_driver_destroy (d=d@entry=0x7f91f8061898) at /home/gordon/projects/proton/c/src/core/connection_driver.c:94 #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f91f80612f0) at /home/gordon/projects/proton/c/src/proactor/epoll.c:882 #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at /home/gordon/projects/proton/c/src/proactor/epoll.c:898 #17 0x00007f921635cb82 in pconnection_done (pc=0x7f91f80612f0) at /home/gordon/projects/proton/c/src/proactor/epoll.c:1037 #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f91f8061890) at /home/gordon/projects/proton/c/src/proactor/epoll.c:2159 #19 0x00007f9216411bb5 in thread_run (arg=0xaf8380) at /home/gordon/projects/dispatch/src/server.c:1037 #20 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f9215f75553 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f920e3f9700 (LWP 3621)): #0 0x00007f921631f3c5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92163e5199 in sys_cond_wait (cond=<optimized out>, held_mutex=<optimized out>) at /home/gordon/projects/dispatch/src/posix/threading.c:91 #2 0x00007f92163fee77 in router_core_thread (arg=0xba9220) at /home/gordon/projects/dispatch/src/router_core/router_core_thread.c:176 #3 0x00007f92163194c0 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9215f75553 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f92156a5240 (LWP 3620)): #0 0x00007f9216372e79 in pn_object_reify (object=0x7f9202159b80) at /home/gordon/projects/proton/c/src/core/object/object.c:214 #1 0x00007f921637315b in pn_class_compare (clazz=<optimized out>, a=0x7f9202159b80, b=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/object/object.c:158 #2 0x00007f92163731a9 in pn_class_equals (clazz=<optimized out>, a=<optimized out>, b=b@entry=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/object/object.c:169 #3 0x00007f92163736b6 in pn_list_index (list=list@entry=0x7f91f8023a40, value=value@entry=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/object/list.c:88 #4 0x00007f92163737a9 in pn_list_remove (list=0x7f91f8023a40, value=value@entry=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/object/list.c:99 #5 0x00007f921637cb78 in pni_remove_link (ssn=0x7f91f8067cb0, link=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/engine.c:311 #6 0x00007f921637cffe in pn_link_finalize (object=<optimized out>) at /home/gordon/projects/proton/c/src/core/engine.c:1119 #7 pn_link_finalize (object=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/engine.c:1098 #8 0x00007f9216373059 in pn_class_decref (clazz=0x7f921639da20 <clazz>, clazz@entry=0x7f921639d620 <PN_OBJECT>, object=0x7f91f9742490) at /home/gordon/projects/proton/c/src/core/object/object.c:95 #9 0x00007f921637326f in pn_decref (object=<optimized out>) at /home/gordon/projects/proton/c/src/core/object/object.c:253 #10 0x00007f921637f05c in pn_link_free (link=<optimized out>) at /home/gordon/projects/proton/c/src/core/engine.c:366 #11 0x00007f921637f086 in pn_session_free (session=0x7f91f8067cb0) at /home/gordon/projects/proton/c/src/core/engine.c:271 #12 0x00007f921637f165 in pn_connection_release (connection=0x7f91f8044720) at /home/gordon/projects/proton/c/src/core/engine.c:125 #13 0x00007f921637f1b9 in pn_connection_free (connection=0x7f91f8044720) at /home/gordon/projects/proton/c/src/core/engine.c:146 #14 0x00007f921637c296 in pn_connection_driver_destroy (d=d@entry=0x7f91f8045458) at /home/gordon/projects/proton/c/src/core/connection_driver.c:94 #15 0x00007f9216359b85 in pconnection_final_free (pc=0x7f91f8044eb0) at /home/gordon/projects/proton/c/src/proactor/epoll.c:882 #16 0x00007f921635aa7c in pconnection_cleanup (pc=<optimized out>) at /home/gordon/projects/proton/c/src/proactor/epoll.c:898 #17 0x00007f921635cb82 in pconnection_done (pc=0x7f91f8044eb0) at /home/gordon/projects/proton/c/src/proactor/epoll.c:1037 #18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f91f8045450) at /home/gordon/projects/proton/c/src/proactor/epoll.c:2159 #19 0x00007f9216411bb5 in thread_run (arg=arg@entry=0xaf8380) at /home/gordon/projects/dispatch/src/server.c:1037 #20 0x00007f9216411e60 in qd_server_run (qd=<optimized out>) at /home/gordon/projects/dispatch/src/server.c:1297 #21 0x00000000004026b2 in main_process (config_path=0x7ffe78e17155 "/home/gordon/projects/router-config-examples/router_a.conf", python_pkgdir=<optimized out>, test_hooks=<optimized out>, fd=2) at /home/gordon/projects/dispatch/router/src/main.c:113 #22 0x0000000000402400 in main (argc=3, argv=0x7ffe78e16cb8) at /home/gordon/projects/dispatch/router/src/main.c:367
It would seem that pn_session_free could be more efficient in cleaning up the links since it knows it is doing them all at once (even if there are other references, itwill still remove them all from the session list of active links).
Attachments
Issue Links
- is cloned by
-
DISPATCH-1504 CLONE - pn_session_free is inefficient if handling large numbers of links
-
- Closed
-