Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-7453

Duplex networkConnector failure with mKahaDB after inactive destinations deleted

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.15.12
    • Fix Version/s: None
    • Component/s: Broker, KahaDB
    • Labels:
      None
    • Environment:

      Windows 10 Home 64-bit
      Java 1.8.0_45-b14
      ActiveMQ 5.15.12

      Description

      Cannot re-establish duplex network connection on remote broker with following error:

      INFO | jvm 1 | 2020/03/21 14:41:32 | ERROR | Failed to create responder end of duplex network bridge Q-bridge@ID:LHC-59471-1584794492297-0:1
      INFO | jvm 1 | 2020/03/21 14:41:32 | java.lang.IllegalStateException: PageFile is not loaded
      INFO | jvm 1 | 2020/03/21 14:41:32 | at org.apache.activemq.store.kahadb.disk.page.PageFile.assertLoaded(PageFile.java:906)[activemq-kahadb-store-5.15.12.jar:5.15.12]
      ...

      Remote broker is configured to delete inactive destinations and uses mKahaDB persistence adapters for different destinations (as described here: Automatic Per Destination Persistence Adapter).

      Same setup, but single kahaDB persistence adapter on remote broker is not causing the issue.

      See attached files for detailed configuration and logs (configuration allows to run both brokers on same PC):

       

      Use case

      Simulate network connection loss and then re-establish duplex communication after remote broker destinations were purged due to inactivity:

      1. clean installation of apache-activemq-5.15.12-bin.zip
      2. start remote broker
      3. start local broker
      4. destination queue created automatically on remote (active consumer from local broker is also shown correctly in web-console)
      5. stop local broker
      6. wait for a while until destination is deleted on remote due to inactivity
      7. start local broker again

      Steps 6.-7. can be repeated multiple times with the same result. However, if required queue is created through web-console on remote broker, duplex bridge establishes successfully, but as soon as destination is purged, problem repeats: bad-activemq.debug.log

      8. Problem disappears if remote broker is restarted, but comes back whenever inactive destinations are purged once again: bad-activemq-restart.debug.log

       

      Some observations

      The main difference I see in logs (good vs bad situation), that in bad situation following messages appear after inactive destinations deleted:

      INFO | jvm 1 | 2020/03/22 11:51:20 | INFO | Stopping async queue tasks
      INFO | jvm 1 | 2020/03/22 11:51:20 | INFO | Stopping async topic tasks
      INFO | jvm 1 | 2020/03/22 11:51:20 | INFO | Stopped KahaDB

      Since this moment local broker cannot establish duplex connection any more, and it doesn't matter which destinations have been purged – with the same name (App.Data) or any other. Also it doesn't matter whether local broker already had any successful communication with the remote. As soon as these messages appear, broker cannot "create responder end of duplex network bridge" because of "PageFile is not loaded".

      When I try to do the same with single kahadb instance, these messages do not appear and no such problem.

       

      Links

      Original discussion here: http://activemq.2283324.n4.nabble.com/Duplex-networkConnector-error-with-mKahaDB-after-inactive-destinations-deleted-td4755827.html

        Attachments

        1. bad-remote-activemq.xml
          2 kB
          Anthony Kocherov
        2. bad-remote-wrapper.log
          10 kB
          Anthony Kocherov
        3. good-remote-activemq.xml
          2 kB
          Anthony Kocherov
        4. good-remote-wrapper.log
          7 kB
          Anthony Kocherov
        5. local-activemq.xml
          2 kB
          Anthony Kocherov
        6. bad-activemq.debug.log
          115 kB
          Anthony Kocherov
        7. bad-activemq-restart.debug.log
          97 kB
          Anthony Kocherov

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Nexus85 Anthony Kocherov
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: