Uploaded image for project: 'ActiveMQ Classic'
  1. ActiveMQ Classic
  2. AMQ-5082

ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Won't Fix
    • 5.9.0, 5.10.0
    • None
    • LevelDB
    • None

    Description

      I have a 3 node amq cluster and one zookeeper node using a replicatedLevelDB persistence adapter.

              <persistenceAdapter>
                  <replicatedLevelDB
                    directory="${activemq.data}/leveldb"
                    replicas="3"
                    bind="tcp://0.0.0.0:0"
                    zkAddress="zookeep0:2181"
                    zkPath="/activemq/leveldb-stores"/>
              </persistenceAdapter>
      

      After about a day or so of sitting idle there are cascading failures and the cluster completely stops listening all together.

      I can reproduce this consistently on 5.9 and the latest 5.10 (commit 2360fb859694bacac1e48092e53a56b388e1d2f0). I am going to attach logs from the three mq nodes and the zookeeper logs that reflect the time where the cluster starts having issues.

      The cluster stops listening Mar 4, 2014 4:56:50 AM (within 5 seconds).

      The OSs are all centos 5.9 on one esx server, so I doubt networking is an issue.

      If you need more data it should be pretty easy to get whatever is needed since it is consistently reproducible.

      This bug may be related to AMQ-5026, but looks different enough to file a separate issue.

      Attachments

        1. 03-07.tgz
          634 kB
          Scott Feldstein
        2. amq_5082_threads.tar.gz
          4 kB
          Kevin McLaughlin
        3. mq-node1-cluster.failure
          13 kB
          Scott Feldstein
        4. mq-node2-cluster.failure
          47 kB
          Scott Feldstein
        5. mq-node3-cluster.failure
          101 kB
          Scott Feldstein
        6. zookeeper.out-cluster.failure
          14 kB
          Scott Feldstein
        7. zookeeper-failover-logs.7z
          269 kB
          Gregor Stephen

        Activity

          People

            ceposta Christian Posta
            scottmf Scott Feldstein
            Votes:
            12 Vote for this issue
            Watchers:
            19 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: