Uploaded image for project: 'Qpid Dispatch'
  1. Qpid Dispatch
  2. DISPATCH-1926

system_tests_http failing on Fedora 33 when trying to delete listeners

Details

    • Test
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.15.0
    • Container
    • None

    Description

      ======================================================================
      FAIL: test_http_listener_delete (system_tests_http.RouterTestHttp)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_http.py", line 157, in test_http_listener_delete
          self.assertTrue(ret_val)
      AssertionError: None is not true
      ======================================================================
      FAIL: test_https_get (system_tests_http.RouterTestHttp)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_http.py", line 332, in test_https_get
          self.assertTrue(ret_val)
      AssertionError: None is not true
      ----------------------------------------------------------------------
      Ran 6 tests in 21.374sFAILED (failures=2)
      [gmurthy@localhost build]$ 
       

      Attachments

        Issue Links

          Activity

            gmurthy Ganesh Murthy added a comment -

            Fedora 32 comes with libwebsockets-3.2.2. I have a server that is listening on localhost port 9000. I can send a management request to my server to delete the listener on port 9000. When I send that management request to the server to delete the listener, the server calls lws_vhost_destroy(vhost). This would close the listening port and I could no longer run a curl command against that port (curl would simply hang and timeout which is what I expect).

            On moving to Fedora 33, which comes with libwebsockets-4.1.2, calling the lws_vhost_destroy(vhost) no longer shuts down the listening port on calling lws_vhost_destroy(vhost). I git bisected the libwebsockets source code and found that the following commit [1] is responsible. Sent an email to the libwebsockets mailing list and got a response and was asked to try a couple of things. Investigation still in progress.

             
            [1]
            [gmurthy@localhost libwebsockets]$ git bisect bad
            c327c7fdb7a65ecdace4f6e9694256c30b545252 is the first bad commit
            commit c327c7fdb7a65ecdace4f6e9694256c30b545252
            Author: Andy Green <andy@warmcat.com>
            Date:   Sat Dec 21 12:13:14 2019 +0000

                vhost destruction: dont allow all wsi closures to kill vh we are already in process of destroying
               
                Saw this on travis selftests during context destroy
               
                ==18895== Invalid read of size 8
                ==18895==    at 0x415909: __lws_vhost_destroy2 (vhost.c:1063)
                ==18895==    by 0x40E65B: lws_context_destroy2 (context.c:929)
                ==18895==    by 0x40EBE5: lws_context_destroy (context.c:1128)
                ==18895==    by 0x40CC41: main (minimal-http-client-post.c:267)
                ==18895==  Address 0x6168688 is 728 bytes inside a block of size 792 free'd
                ==18895==    at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
                ==18895==    by 0x45B29E: _realloc (alloc.c:120)
                ==18895==    by 0x45B2D6: lws_realloc (alloc.c:130)
                ==18895==    by 0x415ED7: __lws_vhost_destroy2 (vhost.c:1204)
                ==18895==    by 0x419164: lws_vhost_unbind_wsi (wsi.c:82)
                ==18895==    by 0x41236B: __lws_free_wsi (close.c:154)
                ==18895==    by 0x4134CF: __lws_close_free_wsi_final (close.c:650)
                ==18895==    by 0x4133BA: __lws_close_free_wsi (close.c:610)
                ==18895==    by 0x413528: lws_close_free_wsi (close.c:660)
                ==18895==    by 0x4158C7: __lws_vhost_destroy2 (vhost.c:1053)
                ==18895==    by 0x40E65B: lws_context_destroy2 (context.c:929)
                ==18895==    by 0x40EBE5: lws_context_destroy (context.c:1128)
               
                Removing the last wsi from the vhost we started to destroy finalized the
                vhost destruction, which is aimed at libuv async close cleanup.  But if
                we already entered __lws_vhost_destroy2, we will definitely destroy the vhost
                ourselves at the end of that function already.  So defeat the wsi close
                triggering it.

             lib/core-net/vhost.c | 3 ++-
             1 file changed, 2 insertions, 1 deletion

            gmurthy Ganesh Murthy added a comment - Fedora 32 comes with libwebsockets-3.2.2. I have a server that is listening on localhost port 9000. I can send a management request to my server to delete the listener on port 9000. When I send that management request to the server to delete the listener, the server calls lws_vhost_destroy(vhost). This would close the listening port and I could no longer run a curl command against that port (curl would simply hang and timeout which is what I expect). On moving to Fedora 33, which comes with libwebsockets-4.1.2, calling the lws_vhost_destroy(vhost) no longer shuts down the listening port on calling lws_vhost_destroy(vhost). I git bisected the libwebsockets source code and found that the following commit [1] is responsible. Sent an email to the libwebsockets mailing list and got a response and was asked to try a couple of things. Investigation still in progress.   [1] [gmurthy@localhost libwebsockets] $ git bisect bad c327c7fdb7a65ecdace4f6e9694256c30b545252 is the first bad commit commit c327c7fdb7a65ecdace4f6e9694256c30b545252 Author: Andy Green < andy@warmcat.com > Date:   Sat Dec 21 12:13:14 2019 +0000     vhost destruction: dont allow all wsi closures to kill vh we are already in process of destroying         Saw this on travis selftests during context destroy         ==18895== Invalid read of size 8     ==18895==    at 0x415909: __lws_vhost_destroy2 (vhost.c:1063)     ==18895==    by 0x40E65B: lws_context_destroy2 (context.c:929)     ==18895==    by 0x40EBE5: lws_context_destroy (context.c:1128)     ==18895==    by 0x40CC41: main (minimal-http-client-post.c:267)     ==18895==  Address 0x6168688 is 728 bytes inside a block of size 792 free'd     ==18895==    at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)     ==18895==    by 0x45B29E: _realloc (alloc.c:120)     ==18895==    by 0x45B2D6: lws_realloc (alloc.c:130)     ==18895==    by 0x415ED7: __lws_vhost_destroy2 (vhost.c:1204)     ==18895==    by 0x419164: lws_vhost_unbind_wsi (wsi.c:82)     ==18895==    by 0x41236B: __lws_free_wsi (close.c:154)     ==18895==    by 0x4134CF: __lws_close_free_wsi_final (close.c:650)     ==18895==    by 0x4133BA: __lws_close_free_wsi (close.c:610)     ==18895==    by 0x413528: lws_close_free_wsi (close.c:660)     ==18895==    by 0x4158C7: __lws_vhost_destroy2 (vhost.c:1053)     ==18895==    by 0x40E65B: lws_context_destroy2 (context.c:929)     ==18895==    by 0x40EBE5: lws_context_destroy (context.c:1128)         Removing the last wsi from the vhost we started to destroy finalized the     vhost destruction, which is aimed at libuv async close cleanup.  But if     we already entered __lws_vhost_destroy2, we will definitely destroy the vhost     ourselves at the end of that function already.  So defeat the wsi close     triggering it.  lib/core-net/vhost.c | 3 ++-  1 file changed, 2 insertions , 1 deletion
            githubbot ASF GitHub Bot added a comment -

            ganeshmurthy opened a new pull request #995:
            URL: https://github.com/apache/qpid-dispatch/pull/995

            …isteners based on available libwebsockets version

            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on to GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org

            githubbot ASF GitHub Bot added a comment - ganeshmurthy opened a new pull request #995: URL: https://github.com/apache/qpid-dispatch/pull/995 …isteners based on available libwebsockets version ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org

            Commit 0c27fdb9f5016258422f2902f98b27985e31dec9 in qpid-dispatch's branch refs/heads/master from Ganesh Murthy
            [ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=0c27fdb ]

            DISPATCH-1926: Added code to selectively allow deletion of http:yes listeners based on available libwebsockets version. This closes #995.

            jira-bot ASF subversion and git services added a comment - Commit 0c27fdb9f5016258422f2902f98b27985e31dec9 in qpid-dispatch's branch refs/heads/master from Ganesh Murthy [ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=0c27fdb ] DISPATCH-1926 : Added code to selectively allow deletion of http:yes listeners based on available libwebsockets version. This closes #995.
            githubbot ASF GitHub Bot added a comment -

            asfgit closed pull request #995:
            URL: https://github.com/apache/qpid-dispatch/pull/995

            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on to GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org

            githubbot ASF GitHub Bot added a comment - asfgit closed pull request #995: URL: https://github.com/apache/qpid-dispatch/pull/995 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
            githubbot ASF GitHub Bot added a comment -

            asfgit closed pull request #995:
            URL: https://github.com/apache/qpid-dispatch/pull/995

            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on to GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org

            githubbot ASF GitHub Bot added a comment - asfgit closed pull request #995: URL: https://github.com/apache/qpid-dispatch/pull/995 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org

            People

              gmurthy Ganesh Murthy
              gmurthy Ganesh Murthy
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: