Uploaded image for project: 'Qpid Dispatch'
  1. Qpid Dispatch
  2. DISPATCH-2323

[http1] system_tests_http1_adaptor failure due to Address already in use issue (second round)

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • 1.15.0
    • None
    • Protocol Adaptors
    • None

    Description

      https://github.com/apache/qpid-dispatch/runs/4993651356?check_suite_focus=true#step:27:1904

      36/37 Test #70: system_tests_http1_adaptor ........................***Failed  351.39 sec
      ============================= test session starts ==============================
      platform linux -- Python 3.6.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/bin/python3
      cachedir: .pytest_cache
      rootdir: /__w/qpid-dispatch/qpid-dispatch/qpid-dispatch/build/tests, configfile: tox.ini
      collecting ... collected 28 items
      
      ::Http1AdaptorManagementTest::test_01_create_delete FAILED
      ::Http1AdaptorManagementTest::test_01_delete_active_connector FAILED
      ::Http1AdaptorOneRouterTest::test_000_stats PASSED
      ::Http1AdaptorOneRouterTest::test_001_get PASSED
      ::Http1AdaptorOneRouterTest::test_002_head PASSED
      ::Http1AdaptorOneRouterTest::test_003_post PASSED
      ::Http1AdaptorOneRouterTest::test_004_put PASSED
      ::Http1AdaptorOneRouterTest::test_005_get_10 PASSED
      ::Http1AdaptorOneRouterTest::test_006_head_10 PASSED
      ::Http1AdaptorOneRouterTest::test_007_post_10 PASSED
      ::Http1AdaptorOneRouterTest::test_008_put_10 PASSED
      ::Http1AdaptorEdge2EdgeTest::test_01_concurrent_requests ERROR
      ::Http1AdaptorEdge2EdgeTest::test_02_credit_replenish ERROR
      ::Http1AdaptorEdge2EdgeTest::test_03_server_reconnect ERROR
      ::Http1AdaptorEdge2EdgeTest::test_04_server_pining_for_the_fjords ERROR
      ::Http1AdaptorEdge2EdgeTest::test_05_large_streaming_msg ERROR
      ::Http1AdaptorEdge2EdgeTest::test_1001_client_request_close ERROR
      ::Http1AdaptorEdge2EdgeTest::test_1002_client_response_close ERROR
      ::Http1AdaptorEdge2EdgeTest::test_2000_curl_get ERROR
      ::Http1AdaptorEdge2EdgeTest::test_2001_curl_put ERROR
      ::Http1AdaptorEdge2EdgeTest::test_2002_curl_post ERROR
      ::Http1AdaptorBadEndpointsTest::test_01_unsolicited_response PASSED
      ::Http1AdaptorBadEndpointsTest::test_02_bad_request_message PASSED
      ::Http1AdaptorBadEndpointsTest::test_03_bad_response_message PASSED
      ::Http1AdaptorBadEndpointsTest::test_04_client_request_close PASSED
      ::Http1AdaptorBadEndpointsTest::test_05_client_response_close PASSED
      ::Http1AdaptorQ2Standalone::test_01_backpressure_client PASSED
      ::Http1AdaptorQ2Standalone::test_02_backpressure_server PASSED
      
      ==================================== ERRORS ====================================
      ___ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_01_concurrent_requests ____
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      name = 'EA2', mode = 'edge'
      extra = [('connector', {'host': '127.0.0.1', 'idleTimeoutSeconds': '120', 'name': 'uplink', 'port': 25519, ...}), ('httpConnec...protocolVersion': 'HTTP1'}), ('httpConnector', {'address': 'testServer10', 'port': 25522, 'protocolVersion': 'HTTP1'})]
      
          @classmethod
          def router(cls, name, mode, extra):
              config = [
                  ('router', {'mode': mode,
                              'id': name,
                              'allowUnsettledMulticast': 'yes'}),
                  ('listener', {'role': 'normal',
                                'port': cls.tester.get_port()}),
                  ('address', {'prefix': 'closest', 'distribution': 'closest'}),
                  ('address', {'prefix': 'multicast', 'distribution': 'multicast'}),
              ]
          
              if extra:
                  config.extend(extra)
              config = Qdrouterd.Config(config)
      >       cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      _____ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_02_credit_replenish _____
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      name = 'EA2', mode = 'edge'
      extra = [('connector', {'host': '127.0.0.1', 'idleTimeoutSeconds': '120', 'name': 'uplink', 'port': 25519, ...}), ('httpConnec...protocolVersion': 'HTTP1'}), ('httpConnector', {'address': 'testServer10', 'port': 25522, 'protocolVersion': 'HTTP1'})]
      
          @classmethod
          def router(cls, name, mode, extra):
              config = [
                  ('router', {'mode': mode,
                              'id': name,
                              'allowUnsettledMulticast': 'yes'}),
                  ('listener', {'role': 'normal',
                                'port': cls.tester.get_port()}),
                  ('address', {'prefix': 'closest', 'distribution': 'closest'}),
                  ('address', {'prefix': 'multicast', 'distribution': 'multicast'}),
              ]
          
              if extra:
                  config.extend(extra)
              config = Qdrouterd.Config(config)
      >       cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      _____ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_03_server_reconnect _____
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      name = 'EA2', mode = 'edge'
      extra = [('connector', {'host': '127.0.0.1', 'idleTimeoutSeconds': '120', 'name': 'uplink', 'port': 25519, ...}), ('httpConnec...protocolVersion': 'HTTP1'}), ('httpConnector', {'address': 'testServer10', 'port': 25522, 'protocolVersion': 'HTTP1'})]
      
          @classmethod
          def router(cls, name, mode, extra):
              config = [
                  ('router', {'mode': mode,
                              'id': name,
                              'allowUnsettledMulticast': 'yes'}),
                  ('listener', {'role': 'normal',
                                'port': cls.tester.get_port()}),
                  ('address', {'prefix': 'closest', 'distribution': 'closest'}),
                  ('address', {'prefix': 'multicast', 'distribution': 'multicast'}),
              ]
          
              if extra:
                  config.extend(extra)
              config = Qdrouterd.Config(config)
      >       cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      _ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_04_server_pining_for_the_fjords _
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      name = 'EA2', mode = 'edge'
      extra = [('connector', {'host': '127.0.0.1', 'idleTimeoutSeconds': '120', 'name': 'uplink', 'port': 25519, ...}), ('httpConnec...protocolVersion': 'HTTP1'}), ('httpConnector', {'address': 'testServer10', 'port': 25522, 'protocolVersion': 'HTTP1'})]
      
          @classmethod
          def router(cls, name, mode, extra):
              config = [
                  ('router', {'mode': mode,
                              'id': name,
                              'allowUnsettledMulticast': 'yes'}),
                  ('listener', {'role': 'normal',
                                'port': cls.tester.get_port()}),
                  ('address', {'prefix': 'closest', 'distribution': 'closest'}),
                  ('address', {'prefix': 'multicast', 'distribution': 'multicast'}),
              ]
          
              if extra:
                  config.extend(extra)
              config = Qdrouterd.Config(config)
      >       cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      ___ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_05_large_streaming_msg ____
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      name = 'EA2', mode = 'edge'
      extra = [('connector', {'host': '127.0.0.1', 'idleTimeoutSeconds': '120', 'name': 'uplink', 'port': 25519, ...}), ('httpConnec...protocolVersion': 'HTTP1'}), ('httpConnector', {'address': 'testServer10', 'port': 25522, 'protocolVersion': 'HTTP1'})]
      
          @classmethod
          def router(cls, name, mode, extra):
              config = [
                  ('router', {'mode': mode,
                              'id': name,
                              'allowUnsettledMulticast': 'yes'}),
                  ('listener', {'role': 'normal',
                                'port': cls.tester.get_port()}),
                  ('address', {'prefix': 'closest', 'distribution': 'closest'}),
                  ('address', {'prefix': 'multicast', 'distribution': 'multicast'}),
              ]
          
              if extra:
                  config.extend(extra)
              config = Qdrouterd.Config(config)
      >       cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      __ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_1001_client_request_close __
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      
          @classmethod
          def setUpClass(cls):
              """Start a router"""
              super(Http1AdaptorEdge2EdgeTest, cls).setUpClass()
          
              # configuration:
              # one edge, one interior
              #
              #  +-------+    +---------+    +-------+
              #  |  EA1  |<==>|  INT.A  |<==>|  EA2  |
              #  +-------+    +---------+    +-------+
              #      ^                           ^
              #      |                           |
              #      V                           V
              #  <clients>                   <servers>
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('INT.A', 'interior', [('listener', {'role': 'edge', 'port': cls.INTA_edge1_port}),
                                               ('listener', {'role': 'edge', 'port': cls.INTA_edge2_port}),
                                               ])
              cls.INT_A = cls.routers[0]
              cls.INT_A.listener = cls.INT_A.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA1', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge1_port}),
                          ('httpListener', {'port': cls.http_listener11_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer11'}),
                          ('httpListener', {'port': cls.http_listener10_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer10'})
                          ])
              cls.EA1 = cls.routers[1]
              cls.EA1.listener = cls.EA1.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA2', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge2_port}),
                          ('httpConnector', {'port': cls.http_server11_port,
                                             'protocolVersion': 'HTTP1',
                                             'address': 'testServer11'}),
                          ('httpConnector', {'port': cls.http_server10_port,
                                             'protocolVersion': 'HTTP1',
      >                                      'address': 'testServer10'})
                          ])
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_tests_http1_adaptor.py:391: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: in router
          cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      _ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_1002_client_response_close __
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      
          @classmethod
          def setUpClass(cls):
              """Start a router"""
              super(Http1AdaptorEdge2EdgeTest, cls).setUpClass()
          
              # configuration:
              # one edge, one interior
              #
              #  +-------+    +---------+    +-------+
              #  |  EA1  |<==>|  INT.A  |<==>|  EA2  |
              #  +-------+    +---------+    +-------+
              #      ^                           ^
              #      |                           |
              #      V                           V
              #  <clients>                   <servers>
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('INT.A', 'interior', [('listener', {'role': 'edge', 'port': cls.INTA_edge1_port}),
                                               ('listener', {'role': 'edge', 'port': cls.INTA_edge2_port}),
                                               ])
              cls.INT_A = cls.routers[0]
              cls.INT_A.listener = cls.INT_A.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA1', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge1_port}),
                          ('httpListener', {'port': cls.http_listener11_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer11'}),
                          ('httpListener', {'port': cls.http_listener10_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer10'})
                          ])
              cls.EA1 = cls.routers[1]
              cls.EA1.listener = cls.EA1.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA2', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge2_port}),
                          ('httpConnector', {'port': cls.http_server11_port,
                                             'protocolVersion': 'HTTP1',
                                             'address': 'testServer11'}),
                          ('httpConnector', {'port': cls.http_server10_port,
                                             'protocolVersion': 'HTTP1',
      >                                      'address': 'testServer10'})
                          ])
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_tests_http1_adaptor.py:391: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: in router
          cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      ________ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_2000_curl_get ________
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      
          @classmethod
          def setUpClass(cls):
              """Start a router"""
              super(Http1AdaptorEdge2EdgeTest, cls).setUpClass()
          
              # configuration:
              # one edge, one interior
              #
              #  +-------+    +---------+    +-------+
              #  |  EA1  |<==>|  INT.A  |<==>|  EA2  |
              #  +-------+    +---------+    +-------+
              #      ^                           ^
              #      |                           |
              #      V                           V
              #  <clients>                   <servers>
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('INT.A', 'interior', [('listener', {'role': 'edge', 'port': cls.INTA_edge1_port}),
                                               ('listener', {'role': 'edge', 'port': cls.INTA_edge2_port}),
                                               ])
              cls.INT_A = cls.routers[0]
              cls.INT_A.listener = cls.INT_A.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA1', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge1_port}),
                          ('httpListener', {'port': cls.http_listener11_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer11'}),
                          ('httpListener', {'port': cls.http_listener10_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer10'})
                          ])
              cls.EA1 = cls.routers[1]
              cls.EA1.listener = cls.EA1.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA2', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge2_port}),
                          ('httpConnector', {'port': cls.http_server11_port,
                                             'protocolVersion': 'HTTP1',
                                             'address': 'testServer11'}),
                          ('httpConnector', {'port': cls.http_server10_port,
                                             'protocolVersion': 'HTTP1',
      >                                      'address': 'testServer10'})
                          ])
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_tests_http1_adaptor.py:391: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: in router
          cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      ________ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_2001_curl_put ________
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      
          @classmethod
          def setUpClass(cls):
              """Start a router"""
              super(Http1AdaptorEdge2EdgeTest, cls).setUpClass()
          
              # configuration:
              # one edge, one interior
              #
              #  +-------+    +---------+    +-------+
              #  |  EA1  |<==>|  INT.A  |<==>|  EA2  |
              #  +-------+    +---------+    +-------+
              #      ^                           ^
              #      |                           |
              #      V                           V
              #  <clients>                   <servers>
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('INT.A', 'interior', [('listener', {'role': 'edge', 'port': cls.INTA_edge1_port}),
                                               ('listener', {'role': 'edge', 'port': cls.INTA_edge2_port}),
                                               ])
              cls.INT_A = cls.routers[0]
              cls.INT_A.listener = cls.INT_A.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA1', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge1_port}),
                          ('httpListener', {'port': cls.http_listener11_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer11'}),
                          ('httpListener', {'port': cls.http_listener10_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer10'})
                          ])
              cls.EA1 = cls.routers[1]
              cls.EA1.listener = cls.EA1.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA2', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge2_port}),
                          ('httpConnector', {'port': cls.http_server11_port,
                                             'protocolVersion': 'HTTP1',
                                             'address': 'testServer11'}),
                          ('httpConnector', {'port': cls.http_server10_port,
                                             'protocolVersion': 'HTTP1',
      >                                      'address': 'testServer10'})
                          ])
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_tests_http1_adaptor.py:391: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: in router
          cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      _______ ERROR at setup of Http1AdaptorEdge2EdgeTest.test_2002_curl_post ________
      
      cls = <class 'system_tests_http1_adaptor.Http1AdaptorEdge2EdgeTest'>
      
          @classmethod
          def setUpClass(cls):
              """Start a router"""
              super(Http1AdaptorEdge2EdgeTest, cls).setUpClass()
          
              # configuration:
              # one edge, one interior
              #
              #  +-------+    +---------+    +-------+
              #  |  EA1  |<==>|  INT.A  |<==>|  EA2  |
              #  +-------+    +---------+    +-------+
              #      ^                           ^
              #      |                           |
              #      V                           V
              #  <clients>                   <servers>
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('INT.A', 'interior', [('listener', {'role': 'edge', 'port': cls.INTA_edge1_port}),
                                               ('listener', {'role': 'edge', 'port': cls.INTA_edge2_port}),
                                               ])
              cls.INT_A = cls.routers[0]
              cls.INT_A.listener = cls.INT_A.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA1', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge1_port}),
                          ('httpListener', {'port': cls.http_listener11_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer11'}),
                          ('httpListener', {'port': cls.http_listener10_port,
                                            'protocolVersion': 'HTTP1',
                                            'address': 'testServer10'})
                          ])
              cls.EA1 = cls.routers[1]
              cls.EA1.listener = cls.EA1.addresses[0]
          
              super(Http1AdaptorEdge2EdgeTest, cls).\
                  router('EA2', 'edge',
                         [('connector', {'name': 'uplink', 'role': 'edge',
                                         'port': cls.INTA_edge2_port}),
                          ('httpConnector', {'port': cls.http_server11_port,
                                             'protocolVersion': 'HTTP1',
                                             'address': 'testServer11'}),
                          ('httpConnector', {'port': cls.http_server10_port,
                                             'protocolVersion': 'HTTP1',
      >                                      'address': 'testServer10'})
                          ])
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_tests_http1_adaptor.py:391: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/http1_tests.py:1198: in router
          cls.routers.append(cls.tester.qdrouterd(name, config, wait=True))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:795: in qdrouterd
          return self.cleanup(Qdrouterd(*args, **kwargs))
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:481: in __init__
          self.wait_ready()
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:705: in wait_ready
          self.wait_connectors(**retry_kwargs)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <system_test.Qdrouterd object at 0x7fed8feb5390>, retry_kwargs = {}
      @py_assert1 = <function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>
      @py_assert4 = None
      @py_format6 = 'Port not connected 25519\n>assert None\n{None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})\n}'
      
          def wait_connectors(self, **retry_kwargs):
              """
              Wait for all connectors to be connected
              @param retry_kwargs: keyword args for L{retry}
              """
              for c in self.config.sections('connector'):
      >           assert retry(lambda: self.is_connected(port=c['port'], host=self.get_host(c.get('protocolFamily'))),
                               **retry_kwargs), "Port not connected %s" % c['port']
      E           AssertionError: Port not connected 25519
      E           assert None
      E            +  where None = retry(<function Qdrouterd.wait_connectors.<locals>.<lambda> at 0x7fed8fe2e9d8>, **{})
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_test.py:697: AssertionError
      =================================== FAILURES ===================================
      _______________ Http1AdaptorManagementTest.test_01_create_delete _______________
      
      self = <system_tests_http1_adaptor.Http1AdaptorManagementTest testMethod=test_01_create_delete>
      
          def test_01_create_delete(self):
              """ Create and delete HTTP1 connectors and listeners.  The
              connectors/listeners are created on the edge router.  Verify that the
              adaptor properly notifies the interior of the subscribers/producers.
              """
              e_mgmt = self.e_router.management
              self.assertEqual(0, len(e_mgmt.query(type=self.LISTENER_TYPE).results))
              self.assertEqual(0, len(e_mgmt.query(type=self.CONNECTOR_TYPE).results))
          
              e_mgmt.create(type=self.CONNECTOR_TYPE,
                            name="ServerConnector",
                            attributes={'address': 'closest/http1Service',
                                        'port': self.http_server_port,
                                        'protocolVersion': 'HTTP1'})
          
              e_mgmt.create(type=self.LISTENER_TYPE,
                            name="ClientListener",
                            attributes={'address': 'closest/http1Service',
                                        'port': self.http_listener_port,
                                        'protocolVersion': 'HTTP1'})
          
              # verify the entities have been created and http traffic works
          
              self.assertEqual(1, len(e_mgmt.query(type=self.LISTENER_TYPE).results))
              self.assertEqual(1, len(e_mgmt.query(type=self.CONNECTOR_TYPE).results))
          
              count, error = http1_ping(sport=self.http_server_port,
                                        cport=self.http_listener_port)
              self.assertIsNone(error)
              self.assertEqual(1, count)
          
              # now check the interior router for the closest/http1Service address
              self.i_router.wait_address("closest/http1Service", subscribers=1)
          
              #
              # delete the connector and listener; wait for the associated connection
              # to be removed
              #
              e_mgmt.delete(type=self.CONNECTOR_TYPE, name="ServerConnector")
              self.assertEqual(0, len(e_mgmt.query(type=self.CONNECTOR_TYPE).results))
              e_mgmt.delete(type=self.LISTENER_TYPE, name="ClientListener")
              self.assertEqual(0, len(e_mgmt.query(type=self.LISTENER_TYPE).results))
          
              # will hit test timeout on failure:
              while True:
                  hconns = 0
                  obj = e_mgmt.query(type=self.CONNECTION_TYPE,
                                     attribute_names=["protocol"])
                  for item in obj.get_dicts():
                      if "http/1.x" in item["protocol"]:
                          hconns += 1
                  if hconns == 0:
                      break
                  sleep(0.25)
          
              # When a connector is configured the router will periodically attempt
              # to connect to the server address. To prove that the connector has
              # been completely removed listen for connection attempts on the server
              # port.
              s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
              s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
              s.bind(("", self.http_server_port))
              s.setblocking(True)
              s.settimeout(3)  # reconnect attempts every 2.5 seconds
              s.listen(1)
              with self.assertRaises(socket.timeout):
      >           conn, addr = s.accept()
      E           AssertionError: timeout not raised
      
      /home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/tests/system_tests_http1_adaptor.py:158: AssertionError
      __________ Http1AdaptorManagementTest.test_01_delete_active_connector __________
      
      self = <system_tests_http1_adaptor.Http1AdaptorManagementTest testMethod=test_01_delete_active_connector>
      
          def test_01_delete_active_connector(self):
              """Delete an HTTP1 connector that is currently connected to a server.
              Verify the connection is dropped.
              """
              e_mgmt = self.e_router.management
              self.assertEqual(0, len(e_mgmt.query(type=self.CONNECTOR_TYPE).results))
          
              e_mgmt.create(type=self.CONNECTOR_TYPE,
                            name="ServerConnector",
                            attributes={'address': 'closest/http1Service',
                                        'port': self.http_server_port,
                                        'protocolVersion': 'HTTP1'})
          
              # verify the connector has been created and attach a dummy server
              self.assertEqual(1, len(e_mgmt.query(type=self.CONNECTOR_TYPE).results))
          
              server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
              server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
      >       server.bind(("", self.http_server_port))
      E       OSError: [Errno 98] Address already in use
      

      One scenario against which the tests are not protecting is that something without SO_REUSEADDR was listening on the socket before. In that case, attempt to connect will fail (nothing is there listening now) but a new bind will fail too (we are lingering after use). The port range dispatch is using is chosen not to fall into

      $ cat /proc/sys/net/ipv4/ip_local_port_range
      32768   60999
      

      so we should be safe in that respect (local randomly assigned ports don't clash). Still, it seems sensible to me to try checking a bind() before we hand over the port.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jdanek Jiri Daněk
            jdanek Jiri Daněk
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment