Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Won't Fix
-
None
-
None
-
None
-
Junit
Description
I created the following unit test to restart master repeatedly in ReplicatedLevelDBBrokerTest.java. Occasionally it hung up and reported "Not enough cluster members have reported their update positions yet.". It can't be re-produced every time as it depends largely on timing.
I think it happened due to an issue of hawtdispatch-transport. I have created a pull request for that. The issue caused the "next" callback of wal_session_copy.transport.stop(next) not being triggered.
I'm not sure when the pull request will be merged and released as hawtdispatch-transport doesn't seem to be maintained actively. That would be great if some work-around could be done in ActiveMQ.
ReplicatedLevelDBBrokerTest.java
@Test(timeout = 1000*60*10)
public void test() throws Throwable {
boolean jmx = false;
try {
System.out.println("======================================");
System.out.println("1. Start 3 activemq nodes.");
System.out.println("======================================");
startBrokerAsync(createBrokerNode("node-1"));
startBrokerAsync(createBrokerNode("node-2"));
startBrokerAsync(createBrokerNode("node-3"));
BrokerService master = waitForNextMaster();
System.out.println("======================================");
System.out.println("2. Push a message to the master and browse the queue");
System.out.println("======================================");
sendMessage(master, pad("Hello World #1", 1024));
assertEquals(1, browseMessages(master, jmx).size());
int times = 200;
System.out.println("iteration " + times + " times.");
for (int i = 0; i < times; i++) {
System.out.println("======================================");
System.out.println( i + ".3. Restart master node");
System.out.println("======================================");
stop(master);
brokers.remove(master);
master = createBrokerNode(master.getBrokerName());
startBrokerAsync(master);
master = waitForNextMaster();
System.out.println("======================================");
System.out.println(i + ".4. Browse the queue on new master");
System.out.println("======================================");
assertEquals(1, browseMessages(master, jmx).size());
System.out.println("======================================");
System.out.println(i + ".5. Found");
System.out.println("======================================");
}
} catch (Throwable e) {
e.printStackTrace();
throw e;
}
}
Starting ZooKeeper 2016-08-25 09:01:04,848 | INFO | main | Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-08-25 09:01:04,853 | INFO | main | Server environment:host.name=192.168.1.118 2016-08-25 09:01:04,853 | INFO | main | Server environment:java.version=1.7.0_79 2016-08-25 09:01:04,853 | INFO | main | Server environment:java.vendor=Oracle Corporation 2016-08-25 09:01:04,853 | INFO | main | Server environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre 2016-08-25 09:01:04,854 | INFO | main | Server environment:java.class.path=/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/test-classes:/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/classes:/Users/samuel.ma/.m2/repository/org/scala-lang/scala-library/2.11.0/scala-library-2.11.0.jar:/Users/samuel.ma/git/lab/activemq/activemq-broker/target/classes:/Users/samuel.ma/git/lab/activemq/activemq-client/target/classes:/Users/samuel.ma/.m2/repository/javax/jmdns/jmdns/3.4.1/jmdns-3.4.1.jar:/Users/samuel.ma/.m2/repository/org/slf4j/slf4j-api/1.7.13/slf4j-api-1.7.13.jar:/Users/samuel.ma/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1_spec-1.1.1.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtbuf/hawtbuf/1.11/hawtbuf-1.11.jar:/Users/samuel.ma/.m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/Users/samuel.ma/.m2/repository/commons-net/commons-net/3.5/commons-net-3.5.jar:/Users/samuel.ma/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/samuel.ma/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Users/samuel.ma/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar:/Users/samuel.ma/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/samuel.ma/git/lab/activemq/activemq-openwire-legacy/target/classes:/Users/samuel.ma/.m2/repository/org/fusesource/hawtbuf/hawtbuf-proto/1.11/hawtbuf-proto-1.11.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtdispatch/hawtdispatch-scala-2.11/1.22/hawtdispatch-scala-2.11-1.22.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtdispatch/hawtdispatch/1.22/hawtdispatch-1.22.jar:/Users/samuel.ma/.m2/repository/org/iq80/leveldb/leveldb-api/0.6/leveldb-api-0.6.jar:/Users/samuel.ma/.m2/repository/org/iq80/leveldb/leveldb/0.6/leveldb-0.6.jar:/Users/samuel.ma/.m2/repository/com/google/guava/guava/12.0/guava-12.0.jar:/Users/samuel.ma/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/samuel.ma/.m2/repository/org/fusesource/leveldbjni/leveldbjni/1.8/leveldbjni-1.8.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtjni/hawtjni-runtime/1.9/hawtjni-runtime-1.9.jar:/Users/samuel.ma/git/lab/hawtdispatch/hawtdispatch-transport/target/classes:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.zookeeper-impl/1.4.0/org.linkedin.zookeeper-impl-1.4.0.jar:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.util-groovy/1.7.1/org.linkedin.util-groovy-1.7.1.jar:/Users/samuel.ma/.m2/repository/org/slf4j/jul-to-slf4j/1.5.8/jul-to-slf4j-1.5.8.jar:/Users/samuel.ma/.m2/repository/org/apache/ant/ant/1.8.4/ant-1.8.4.jar:/Users/samuel.ma/.m2/repository/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.jar:/Users/samuel.ma/.m2/repository/org/json/json/20090211/json-20090211.jar:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.util-core/1.4.0/org.linkedin.util-core-1.4.0.jar:/Users/samuel.ma/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/samuel.ma/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/samuel.ma/.m2/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar:/Users/samuel.ma/.m2/repository/org/osgi/org.osgi.core/4.3.1/org.osgi.core-4.3.1.jar:/Users/samuel.ma/.m2/repository/org/osgi/org.osgi.compendium/4.3.1/org.osgi.compendium-4.3.1.jar:/Users/samuel.ma/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar:/Users/samuel.ma/.m2/repository/org/xerial/snappy/snappy-java/1.1.2/snappy-java-1.1.2.jar:/Users/samuel.ma/.m2/repository/org/iq80/snappy/snappy/0.2/snappy-0.2.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.3/jackson-core-2.6.3.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.3/jackson-annotations-2.6.3.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.3/jackson-databind-2.6.3.jar:/Users/samuel.ma/.m2/repository/org/apache/hadoop/hadoop-core/1.0.0/hadoop-core-1.0.0.jar:/Users/samuel.ma/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/samuel.ma/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/samuel.ma/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/samuel.ma/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/samuel.ma/.m2/repository/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar:/Users/samuel.ma/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/samuel.ma/git/lab/activemq/activemq-broker/target/test-classes:/Users/samuel.ma/git/lab/activemq/activemq-kahadb-store/target/classes:/Users/samuel.ma/.m2/repository/org/apache/activemq/protobuf/activemq-protobuf/1.1/activemq-protobuf-1.1.jar:/Users/samuel.ma/.m2/repository/org/apache/hadoop/hadoop-test/1.0.0/hadoop-test-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftplet-api/1.0.0/ftplet-api-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/mina/mina-core/2.0.0-M5/mina-core-2.0.0-M5.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftpserver-core/1.0.0/ftpserver-core-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftpserver-deprecated/1.0.0-M2/ftpserver-deprecated-1.0.0-M2.jar:/Users/samuel.ma/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/samuel.ma/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/samuel.ma/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/samuel.ma/.m2/repository/org/apache/commons/commons-math/2.2/commons-math-2.2.jar:/Users/samuel.ma/.m2/repository/org/scalatest/scalatest_2.11/2.1.5/scalatest_2.11-2.1.5.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/scala-reflect/2.11.0/scala-reflect-2.11.0.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.1/scala-parser-combinators_2.11-1.0.1.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.1/scala-xml_2.11-1.0.1.jar:/Applications/Eclipse-Mars-4.5.app/Contents/Eclipse/configuration/org.eclipse.osgi/379/0/.cp/:/Applications/Eclipse-Mars-4.5.app/Contents/Eclipse/configuration/org.eclipse.osgi/378/0/.cp/ 2016-08-25 09:01:04,854 | INFO | main | Server environment:java.library.path=/Users/samuel.ma/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. 2016-08-25 09:01:04,855 | INFO | main | Server environment:java.io.tmpdir=/var/folders/7k/ds05vsr900q_s75x5wq2q9g40000gp/T/ 2016-08-25 09:01:04,855 | INFO | main | Server environment:java.compiler=<NA> 2016-08-25 09:01:04,855 | INFO | main | Server environment:os.name=Mac OS X 2016-08-25 09:01:04,855 | INFO | main | Server environment:os.arch=x86_64 2016-08-25 09:01:04,855 | INFO | main | Server environment:os.version=10.10.5 2016-08-25 09:01:04,856 | INFO | main | Server environment:user.name=samuel.ma 2016-08-25 09:01:04,856 | INFO | main | Server environment:user.home=/Users/samuel.ma 2016-08-25 09:01:04,856 | INFO | main | Server environment:user.dir=/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store 2016-08-25 09:01:04,857 | INFO | main | tickTime set to 500 2016-08-25 09:01:04,905 | INFO | main | binding to port 0.0.0.0/0.0.0.0:0 ZooKeeper started ====================================== 1. Start 3 activemq nodes. ====================================== Wait for master to start up... 2016-08-25 09:01:05,242 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:05,242 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:05,242 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:05,771 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:05,772 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:05,771 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:host.name=192.168.1.118 2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.version=1.7.0_79 2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.vendor=Oracle Corporation 2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre 2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.class.path=/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/test-classes:/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/classes:/Users/samuel.ma/.m2/repository/org/scala-lang/scala-library/2.11.0/scala-library-2.11.0.jar:/Users/samuel.ma/git/lab/activemq/activemq-broker/target/classes:/Users/samuel.ma/git/lab/activemq/activemq-client/target/classes:/Users/samuel.ma/.m2/repository/javax/jmdns/jmdns/3.4.1/jmdns-3.4.1.jar:/Users/samuel.ma/.m2/repository/org/slf4j/slf4j-api/1.7.13/slf4j-api-1.7.13.jar:/Users/samuel.ma/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1_spec-1.1.1.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtbuf/hawtbuf/1.11/hawtbuf-1.11.jar:/Users/samuel.ma/.m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/Users/samuel.ma/.m2/repository/commons-net/commons-net/3.5/commons-net-3.5.jar:/Users/samuel.ma/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/samuel.ma/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Users/samuel.ma/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar:/Users/samuel.ma/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/samuel.ma/git/lab/activemq/activemq-openwire-legacy/target/classes:/Users/samuel.ma/.m2/repository/org/fusesource/hawtbuf/hawtbuf-proto/1.11/hawtbuf-proto-1.11.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtdispatch/hawtdispatch-scala-2.11/1.22/hawtdispatch-scala-2.11-1.22.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtdispatch/hawtdispatch/1.22/hawtdispatch-1.22.jar:/Users/samuel.ma/.m2/repository/org/iq80/leveldb/leveldb-api/0.6/leveldb-api-0.6.jar:/Users/samuel.ma/.m2/repository/org/iq80/leveldb/leveldb/0.6/leveldb-0.6.jar:/Users/samuel.ma/.m2/repository/com/google/guava/guava/12.0/guava-12.0.jar:/Users/samuel.ma/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/samuel.ma/.m2/repository/org/fusesource/leveldbjni/leveldbjni/1.8/leveldbjni-1.8.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtjni/hawtjni-runtime/1.9/hawtjni-runtime-1.9.jar:/Users/samuel.ma/git/lab/hawtdispatch/hawtdispatch-transport/target/classes:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.zookeeper-impl/1.4.0/org.linkedin.zookeeper-impl-1.4.0.jar:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.util-groovy/1.7.1/org.linkedin.util-groovy-1.7.1.jar:/Users/samuel.ma/.m2/repository/org/slf4j/jul-to-slf4j/1.5.8/jul-to-slf4j-1.5.8.jar:/Users/samuel.ma/.m2/repository/org/apache/ant/ant/1.8.4/ant-1.8.4.jar:/Users/samuel.ma/.m2/repository/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.jar:/Users/samuel.ma/.m2/repository/org/json/json/20090211/json-20090211.jar:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.util-core/1.4.0/org.linkedin.util-core-1.4.0.jar:/Users/samuel.ma/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/samuel.ma/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/samuel.ma/.m2/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar:/Users/samuel.ma/.m2/repository/org/osgi/org.osgi.core/4.3.1/org.osgi.core-4.3.1.jar:/Users/samuel.ma/.m2/repository/org/osgi/org.osgi.compendium/4.3.1/org.osgi.compendium-4.3.1.jar:/Users/samuel.ma/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar:/Users/samuel.ma/.m2/repository/org/xerial/snappy/snappy-java/1.1.2/snappy-java-1.1.2.jar:/Users/samuel.ma/.m2/repository/org/iq80/snappy/snappy/0.2/snappy-0.2.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.3/jackson-core-2.6.3.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.3/jackson-annotations-2.6.3.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.3/jackson-databind-2.6.3.jar:/Users/samuel.ma/.m2/repository/org/apache/hadoop/hadoop-core/1.0.0/hadoop-core-1.0.0.jar:/Users/samuel.ma/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/samuel.ma/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/samuel.ma/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/samuel.ma/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/samuel.ma/.m2/repository/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar:/Users/samuel.ma/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/samuel.ma/git/lab/activemq/activemq-broker/target/test-classes:/Users/samuel.ma/git/lab/activemq/activemq-kahadb-store/target/classes:/Users/samuel.ma/.m2/repository/org/apache/activemq/protobuf/activemq-protobuf/1.1/activemq-protobuf-1.1.jar:/Users/samuel.ma/.m2/repository/org/apache/hadoop/hadoop-test/1.0.0/hadoop-test-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftplet-api/1.0.0/ftplet-api-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/mina/mina-core/2.0.0-M5/mina-core-2.0.0-M5.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftpserver-core/1.0.0/ftpserver-core-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftpserver-deprecated/1.0.0-M2/ftpserver-deprecated-1.0.0-M2.jar:/Users/samuel.ma/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/samuel.ma/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/samuel.ma/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/samuel.ma/.m2/repository/org/apache/commons/commons-math/2.2/commons-math-2.2.jar:/Users/samuel.ma/.m2/repository/org/scalatest/scalatest_2.11/2.1.5/scalatest_2.11-2.1.5.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/scala-reflect/2.11.0/scala-reflect-2.11.0.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.1/scala-parser-combinators_2.11-1.0.1.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.1/scala-xml_2.11-1.0.1.jar:/Applications/Eclipse-Mars-4.5.app/Contents/Eclipse/configuration/org.eclipse.osgi/379/0/.cp/:/Applications/Eclipse-Mars-4.5.app/Contents/Eclipse/configuration/org.eclipse.osgi/378/0/.cp/ 2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.library.path=/Users/samuel.ma/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. 2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.io.tmpdir=/var/folders/7k/ds05vsr900q_s75x5wq2q9g40000gp/T/ 2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:java.compiler=<NA> 2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:os.name=Mac OS X 2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:os.arch=x86_64 2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:os.version=10.10.5 2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:user.name=samuel.ma 2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:user.home=/Users/samuel.ma 2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:user.dir=/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store 2016-08-25 09:01:05,783 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@57f6eb51 2016-08-25 09:01:05,783 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@13d33ff5 2016-08-25 09:01:05,783 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@57f6eb51 2016-08-25 09:01:05,803 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:05,803 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:05,803 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:05,807 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:05,807 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54918 2016-08-25 09:01:05,808 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:05,808 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:05,812 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54919 2016-08-25 09:01:05,815 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54918 2016-08-25 09:01:05,816 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54920 2016-08-25 09:01:05,816 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54919 2016-08-25 09:01:05,817 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54920 2016-08-25 09:01:05,818 | INFO | SyncThread:0 | Creating new log file: log.1 2016-08-25 09:01:05,827 | INFO | SyncThread:0 | Established session 0x156bec917250000 with negotiated timeout 10000 for client /127.0.0.1:54918 2016-08-25 09:01:05,828 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250000, negotiated timeout = 10000 2016-08-25 09:01:05,828 | INFO | SyncThread:0 | Established session 0x156bec917250001 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54919 2016-08-25 09:01:05,828 | INFO | SyncThread:0 | Established session 0x156bec917250002 with negotiated timeout 10000 for client /127.0.0.1:54920 2016-08-25 09:01:05,829 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250001, negotiated timeout = 10000 2016-08-25 09:01:05,829 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250002, negotiated timeout = 10000 2016-08-25 09:01:05,860 | INFO | ProcessThread(sid:0 cport:-1): | Got user-level KeeperException when processing sessionid:0x156bec917250001 type:create cxid:0x3 zxid:0x5 txntype:-1 reqpath:n/a Error Path:/default Error:KeeperErrorCode = NodeExists for /default 2016-08-25 09:01:05,861 | INFO | ProcessThread(sid:0 cport:-1): | Got user-level KeeperException when processing sessionid:0x156bec917250002 type:create cxid:0x3 zxid:0x6 txntype:-1 reqpath:n/a Error Path:/default Error:KeeperErrorCode = NodeExists for /default 2016-08-25 09:01:06,391 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:06,393 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:06,395 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:06,405 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:06,524 | WARN | Starting broker node: node-2-EventThread | listeners are taking too long to process the events 2016-08-25 09:01:06,524 | WARN | Starting broker node: node-2-EventThread | listeners are taking too long to process the events 2016-08-25 09:01:06,568 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:06,932 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:54921 2016-08-25 09:01:06,943 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:06,943 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:06,965 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54921 2016-08-25 09:01:06,965 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54921 2016-08-25 09:01:06,982 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:06,982 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:07,784 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:07,784 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:07,852 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.00/0.00 kb and 1/1 files 2016-08-25 09:01:07,852 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.00/0.00 kb and 1/1 files 2016-08-25 09:01:07,854 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:07,854 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:07,890 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:07,890 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:08,132 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) is starting 2016-08-25 09:01:08,167 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54927 2016-08-25 09:01:08,168 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:08,168 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) started 2016-08-25 09:01:08,168 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:08,169 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 0 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30381 mb of usable space. - resetting to maximum available disk space: 30381 mb 2016-08-25 09:01:08,170 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 0 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30381 mb of usable space. - resetting to maximum available disk space: 30381 mb Master started: node-2 ====================================== 2. Push a message to the master and browse the queue ====================================== iteration 200 times. ====================================== 0.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:08,709 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) is shutting down 2016-08-25 09:01:08,711 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:08,731 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:08,731 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:08,731 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250000 2016-08-25 09:01:08,732 | INFO | Time-limited test | Session: 0x156bec917250000 closed 2016-08-25 09:01:08,732 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:08,733 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54918 which had sessionid 0x156bec917250000 2016-08-25 09:01:08,737 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:08,737 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:08,738 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:08,738 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:09,193 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:09,193 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:09,194 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:09,228 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:09,234 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:09,241 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:09,325 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:09,341 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) uptime 4.193 seconds 2016-08-25 09:01:09,341 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) is shutdown Wait for master to start up... 2016-08-25 09:01:09,350 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:09,358 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@22ea26c0 2016-08-25 09:01:09,358 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:09,359 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:09,360 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:09,360 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54930 2016-08-25 09:01:09,361 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54930 2016-08-25 09:01:09,362 | INFO | SyncThread:0 | Established session 0x156bec917250003 with negotiated timeout 10000 for client /127.0.0.1:54930 2016-08-25 09:01:09,362 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250003, negotiated timeout = 10000 2016-08-25 09:01:09,390 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:54931 2016-08-25 09:01:09,396 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:09,396 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:09,410 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:54931 2016-08-25 09:01:09,410 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54931 2016-08-25 09:01:09,410 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:09,410 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:09,422 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:09,765 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:09,767 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:09,769 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000002 2016-08-25 09:01:09,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000002 2016-08-25 09:01:09,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:09,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000003.log 2016-08-25 09:01:09,771 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000003.log 2016-08-25 09:01:09,771 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:09,776 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.36/2.70 kb and 1/4 files 2016-08-25 09:01:09,778 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.05/1.34 kb and 1/3 files 2016-08-25 09:01:09,778 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.41/2.70 kb and 2/4 files 2016-08-25 09:01:09,779 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.68/2.70 kb and 3/4 files 2016-08-25 09:01:09,779 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.06/1.34 kb and 2/3 files 2016-08-25 09:01:09,780 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.34/1.34 kb and 3/3 files 2016-08-25 09:01:09,780 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/2.70 kb and 4/4 files 2016-08-25 09:01:09,780 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:09,780 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:09,790 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:09,790 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:09,792 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) is starting 2016-08-25 09:01:09,821 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54936 2016-08-25 09:01:09,822 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:09,822 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) started 2016-08-25 09:01:09,822 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:09,822 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30783 mb of usable space. - resetting to maximum available disk space: 30783 mb 2016-08-25 09:01:09,822 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30783 mb of usable space. - resetting to maximum available disk space: 30783 mb Master started: node-3 ====================================== 0.4. Browse the queue on new master ====================================== ====================================== 0.5. Found ====================================== ====================================== 1.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:09,877 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) is shutting down 2016-08-25 09:01:09,878 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:09,881 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:09,881 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:09,881 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250001 2016-08-25 09:01:09,882 | INFO | Time-limited test | Session: 0x156bec917250001 closed 2016-08-25 09:01:09,882 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54919 which had sessionid 0x156bec917250001 2016-08-25 09:01:09,882 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:09,882 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:09,882 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:09,883 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:09,883 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:09,917 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:09,917 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:09,917 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:09,920 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:09,924 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:09,930 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:09,990 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:09,992 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) uptime 4.846 seconds 2016-08-25 09:01:09,992 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) is shutdown Wait for master to start up... 2016-08-25 09:01:09,997 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:54938 2016-08-25 09:01:10,000 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:10,006 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@58053ef7 2016-08-25 09:01:10,006 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:10,006 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,007 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:10,007 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:10,007 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54939 2016-08-25 09:01:10,007 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:54938 2016-08-25 09:01:10,008 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54939 2016-08-25 09:01:10,008 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:10,008 | INFO | SyncThread:0 | Established session 0x156bec917250004 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54939 2016-08-25 09:01:10,009 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250004, negotiated timeout = 10000 2016-08-25 09:01:10,021 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,024 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54938 2016-08-25 09:01:10,025 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:10,027 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:10,031 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:10,032 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000004 2016-08-25 09:01:10,032 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:10,032 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:10,032 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000006.log 2016-08-25 09:01:10,035 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:10,037 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.10/2.25 kb and 1/4 files 2016-08-25 09:01:10,038 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.05/2.25 kb and 2/4 files 2016-08-25 09:01:10,038 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.07/2.25 kb and 3/4 files 2016-08-25 09:01:10,039 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.25/2.25 kb and 4/4 files 2016-08-25 09:01:10,040 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:10,041 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:10,042 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:10,043 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000006.log 2016-08-25 09:01:10,043 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:10,043 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:10,044 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000004 2016-08-25 09:01:10,044 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) is starting 2016-08-25 09:01:10,053 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.95/2.25 kb and 1/4 files 2016-08-25 09:01:10,053 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.13/2.25 kb and 2/4 files 2016-08-25 09:01:10,054 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.15/2.25 kb and 3/4 files 2016-08-25 09:01:10,055 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.25/2.25 kb and 4/4 files 2016-08-25 09:01:10,056 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:10,060 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:10,063 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54944 2016-08-25 09:01:10,063 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:10,064 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) started 2016-08-25 09:01:10,064 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:10,064 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:10,064 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-1 ====================================== 1.4. Browse the queue on new master ====================================== ====================================== 1.5. Found ====================================== ====================================== 2.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:10,115 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) is shutting down 2016-08-25 09:01:10,116 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:10,119 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:10,120 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250002 2016-08-25 09:01:10,120 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,120 | INFO | Time-limited test | Session: 0x156bec917250002 closed 2016-08-25 09:01:10,120 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:10,121 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54920 which had sessionid 0x156bec917250002 2016-08-25 09:01:10,122 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:10,122 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:10,124 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:10,125 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:10,130 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:10,130 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,134 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,143 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:10,148 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:10,157 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,204 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:54946 2016-08-25 09:01:10,210 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,212 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:54946 2016-08-25 09:01:10,212 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:10,229 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:10,240 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:10,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000006.log 2016-08-25 09:01:10,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000004 2016-08-25 09:01:10,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:10,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:10,244 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:10,245 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) uptime 5.100 seconds 2016-08-25 09:01:10,245 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) is shutdown Wait for master to start up... 2016-08-25 09:01:10,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.18/2.25 kb and 1/4 files 2016-08-25 09:01:10,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.29/2.25 kb and 2/4 files 2016-08-25 09:01:10,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.24/2.25 kb and 3/4 files 2016-08-25 09:01:10,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.25/2.25 kb and 4/4 files 2016-08-25 09:01:10,250 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:10,263 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:10,267 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:10,268 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@626495c2 2016-08-25 09:01:10,268 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:10,269 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:10,270 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) is starting 2016-08-25 09:01:10,270 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:10,271 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54949 2016-08-25 09:01:10,271 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54949 2016-08-25 09:01:10,279 | INFO | SyncThread:0 | Established session 0x156bec917250005 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54949 2016-08-25 09:01:10,280 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250005, negotiated timeout = 10000 2016-08-25 09:01:10,305 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54950 2016-08-25 09:01:10,305 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,306 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:10,307 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) started 2016-08-25 09:01:10,307 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:10,307 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:10,307 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:10,307 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54946 2016-08-25 09:01:10,308 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started Master started: node-2 ====================================== 2.4. Browse the queue on new master ====================================== 2016-08-25 09:01:10,351 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:10,359 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:10,367 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000004 ====================================== 2.5. Found ====================================== ====================================== 3.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:10,368 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) is shutting down 2016-08-25 09:01:10,368 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:10,369 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000006.log 2016-08-25 09:01:10,369 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:10,369 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:10,372 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:10,373 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250003 2016-08-25 09:01:10,373 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,374 | INFO | Time-limited test | Session: 0x156bec917250003 closed 2016-08-25 09:01:10,374 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:10,375 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54930 which had sessionid 0x156bec917250003 2016-08-25 09:01:10,377 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:10,378 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:10,380 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:10,381 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:10,400 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:10,400 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,404 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,407 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:10,414 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:10,419 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,455 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:54954 2016-08-25 09:01:10,461 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,461 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54954 2016-08-25 09:01:10,461 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:10,479 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:10,480 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) uptime 1.138 seconds 2016-08-25 09:01:10,480 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) is shutdown Wait for master to start up... 2016-08-25 09:01:10,488 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:10,492 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:10,492 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7171ff47 2016-08-25 09:01:10,493 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:10,494 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:10,494 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:10,495 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54956 2016-08-25 09:01:10,495 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54956 2016-08-25 09:01:10,497 | INFO | SyncThread:0 | Established session 0x156bec917250006 with negotiated timeout 10000 for client /127.0.0.1:54956 2016-08-25 09:01:10,497 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250006, negotiated timeout = 10000 2016-08-25 09:01:10,500 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:10,501 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:10,501 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:10,502 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000007 2016-08-25 09:01:10,502 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000009.log 2016-08-25 09:01:10,502 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:10,511 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.95/3.21 kb and 1/5 files 2016-08-25 09:01:10,511 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,511 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.85/3.21 kb and 2/5 files 2016-08-25 09:01:10,512 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.01/3.21 kb and 3/5 files 2016-08-25 09:01:10,513 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:54954 2016-08-25 09:01:10,513 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:10,513 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.19/3.21 kb and 4/5 files 2016-08-25 09:01:10,514 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.21/3.21 kb and 5/5 files 2016-08-25 09:01:10,514 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:10,518 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:10,520 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) is starting 2016-08-25 09:01:10,535 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54959 2016-08-25 09:01:10,536 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:10,536 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) started 2016-08-25 09:01:10,536 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:10,536 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:10,537 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:10,537 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 Master started: node-3 ====================================== 3.4. Browse the queue on new master ====================================== 2016-08-25 09:01:10,549 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:10,551 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000009.log 2016-08-25 09:01:10,551 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000007 2016-08-25 09:01:10,552 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:10,552 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:10,552 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:10,573 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/3.21 kb and 1/5 files 2016-08-25 09:01:10,574 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.34/3.21 kb and 2/5 files 2016-08-25 09:01:10,574 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.29/3.21 kb and 3/5 files 2016-08-25 09:01:10,575 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.19/3.21 kb and 4/5 files 2016-08-25 09:01:10,576 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.21/3.21 kb and 5/5 files 2016-08-25 09:01:10,576 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:10,582 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 3.5. Found ====================================== ====================================== 4.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:10,588 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) is shutting down 2016-08-25 09:01:10,589 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:10,592 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:10,593 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250004 2016-08-25 09:01:10,593 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,594 | INFO | Time-limited test | Session: 0x156bec917250004 closed 2016-08-25 09:01:10,594 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:10,594 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54939 which had sessionid 0x156bec917250004 2016-08-25 09:01:10,596 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:10,596 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:10,597 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:10,597 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:10,626 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:10,627 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,630 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,653 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:10,658 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:10,665 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,704 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:54962 2016-08-25 09:01:10,710 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,711 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:54962 2016-08-25 09:01:10,712 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:10,723 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:10,724 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) uptime 0.731 seconds 2016-08-25 09:01:10,724 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) is shutdown Wait for master to start up... 2016-08-25 09:01:10,732 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:10,735 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@17c1fbff 2016-08-25 09:01:10,735 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:10,736 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:10,737 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:10,737 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54964 2016-08-25 09:01:10,738 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54964 2016-08-25 09:01:10,739 | INFO | SyncThread:0 | Established session 0x156bec917250007 with negotiated timeout 10000 for client /127.0.0.1:54964 2016-08-25 09:01:10,739 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250007, negotiated timeout = 10000 2016-08-25 09:01:10,742 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:10,765 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:10,765 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:10,768 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54962 2016-08-25 09:01:10,768 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:10,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:10,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:10,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000007 2016-08-25 09:01:10,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:10,771 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000009.log 2016-08-25 09:01:10,778 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/3.21 kb and 1/5 files 2016-08-25 09:01:10,779 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/3.21 kb and 2/5 files 2016-08-25 09:01:10,780 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.07/3.21 kb and 3/5 files 2016-08-25 09:01:10,780 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.02/3.21 kb and 4/5 files 2016-08-25 09:01:10,781 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.21/3.21 kb and 5/5 files 2016-08-25 09:01:10,781 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:10,785 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:10,787 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) is starting 2016-08-25 09:01:10,799 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54967 2016-08-25 09:01:10,799 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:10,799 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) started 2016-08-25 09:01:10,799 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:10,800 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:10,800 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-1 ====================================== 4.4. Browse the queue on new master ====================================== 2016-08-25 09:01:10,857 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:10,867 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:10,868 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000007 2016-08-25 09:01:10,868 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000009.log 2016-08-25 09:01:10,868 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:10,869 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:10,869 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:10,878 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.15/3.21 kb and 1/5 files 2016-08-25 09:01:10,880 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.34/3.21 kb and 2/5 files 2016-08-25 09:01:10,881 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.24/3.21 kb and 3/5 files 2016-08-25 09:01:10,882 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.19/3.21 kb and 4/5 files 2016-08-25 09:01:10,883 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.21/3.21 kb and 5/5 files 2016-08-25 09:01:10,884 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:10,896 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 4.5. Found ====================================== ====================================== 5.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:10,927 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) is shutting down 2016-08-25 09:01:10,930 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:10,935 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,936 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:10,938 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250005 2016-08-25 09:01:10,939 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54949 which had sessionid 0x156bec917250005 2016-08-25 09:01:10,939 | INFO | Time-limited test | Session: 0x156bec917250005 closed 2016-08-25 09:01:10,940 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:10,941 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:10,942 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:10,952 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:10,953 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:10,984 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:10,985 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:10,988 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,005 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:11,010 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:11,016 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,060 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:11,061 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) uptime 0.815 seconds 2016-08-25 09:01:11,061 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) is shutdown Wait for master to start up... 2016-08-25 09:01:11,063 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:54970 2016-08-25 09:01:11,067 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:11,068 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,070 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6d2eb2e 2016-08-25 09:01:11,071 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54970 2016-08-25 09:01:11,071 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:11,071 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:11,072 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:11,072 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:11,072 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54972 2016-08-25 09:01:11,072 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54972 2016-08-25 09:01:11,073 | INFO | SyncThread:0 | Established session 0x156bec917250008 with negotiated timeout 10000 for client /127.0.0.1:54972 2016-08-25 09:01:11,074 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250008, negotiated timeout = 10000 2016-08-25 09:01:11,089 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,091 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54970 2016-08-25 09:01:11,091 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:11,108 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:11,117 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,119 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:11,119 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:11,119 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:11,120 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:11,120 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000010 2016-08-25 09:01:11,120 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:11,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000012.log 2016-08-25 09:01:11,123 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:11,123 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000010 2016-08-25 09:01:11,124 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:11,124 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:11,125 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:11,125 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:11,125 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000012.log 2016-08-25 09:01:11,128 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/4.16 kb and 1/6 files 2016-08-25 09:01:11,130 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/4.16 kb and 2/6 files 2016-08-25 09:01:11,131 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.86/4.16 kb and 3/6 files 2016-08-25 09:01:11,131 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.07/4.16 kb and 4/6 files 2016-08-25 09:01:11,132 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.97/4.16 kb and 5/6 files 2016-08-25 09:01:11,133 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.16/4.16 kb and 6/6 files 2016-08-25 09:01:11,133 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:11,136 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:11,138 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.21/4.16 kb and 1/6 files 2016-08-25 09:01:11,138 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) is starting 2016-08-25 09:01:11,139 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.11/4.16 kb and 2/6 files 2016-08-25 09:01:11,140 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.01/4.16 kb and 3/6 files 2016-08-25 09:01:11,140 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.96/4.16 kb and 4/6 files 2016-08-25 09:01:11,141 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.97/4.16 kb and 5/6 files 2016-08-25 09:01:11,142 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.16/4.16 kb and 6/6 files 2016-08-25 09:01:11,142 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:11,146 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,148 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54976 2016-08-25 09:01:11,148 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:11,148 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) started 2016-08-25 09:01:11,148 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:11,148 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:11,149 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 5.4. Browse the queue on new master ====================================== ====================================== 5.5. Found ====================================== ====================================== 6.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:11,184 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) is shutting down 2016-08-25 09:01:11,185 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:11,189 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,190 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:11,191 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250006 2016-08-25 09:01:11,192 | INFO | Time-limited test | Session: 0x156bec917250006 closed 2016-08-25 09:01:11,192 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:11,192 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54956 which had sessionid 0x156bec917250006 2016-08-25 09:01:11,196 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:11,196 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:11,197 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,197 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:11,243 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:11,243 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,244 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:11,246 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,250 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:11,258 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,306 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:11,307 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) uptime 0.826 seconds 2016-08-25 09:01:11,307 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) is shutdown Wait for master to start up... 2016-08-25 09:01:11,313 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:11,316 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6086510a 2016-08-25 09:01:11,316 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:54978 2016-08-25 09:01:11,316 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:11,317 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:11,318 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:11,318 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54979 2016-08-25 09:01:11,318 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54979 2016-08-25 09:01:11,319 | INFO | SyncThread:0 | Established session 0x156bec917250009 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54979 2016-08-25 09:01:11,319 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250009, negotiated timeout = 10000 2016-08-25 09:01:11,323 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,324 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:54978 2016-08-25 09:01:11,324 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:11,330 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,332 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:54978 2016-08-25 09:01:11,332 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:11,346 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:11,350 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:11,351 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:11,351 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:11,352 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:11,352 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000012.log 2016-08-25 09:01:11,352 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000010 2016-08-25 09:01:11,352 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:11,355 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,358 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.02/4.16 kb and 1/6 files 2016-08-25 09:01:11,359 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.92/4.16 kb and 2/6 files 2016-08-25 09:01:11,359 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.82/4.16 kb and 3/6 files 2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000010 2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000012.log 2016-08-25 09:01:11,361 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.00/4.16 kb and 4/6 files 2016-08-25 09:01:11,361 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:11,361 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:11,361 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.21/4.16 kb and 5/6 files 2016-08-25 09:01:11,362 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.16/4.16 kb and 6/6 files 2016-08-25 09:01:11,362 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:11,365 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:11,366 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) is starting 2016-08-25 09:01:11,366 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.95/4.16 kb and 1/6 files 2016-08-25 09:01:11,367 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.96/4.16 kb and 2/6 files 2016-08-25 09:01:11,368 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.17/4.16 kb and 3/6 files 2016-08-25 09:01:11,369 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.35/4.16 kb and 4/6 files 2016-08-25 09:01:11,370 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.26/4.16 kb and 5/6 files 2016-08-25 09:01:11,371 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.16/4.16 kb and 6/6 files 2016-08-25 09:01:11,371 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:11,382 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,384 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54984 2016-08-25 09:01:11,384 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:11,385 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) started 2016-08-25 09:01:11,385 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:11,385 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:11,385 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 6.4. Browse the queue on new master ====================================== ====================================== 6.5. Found ====================================== ====================================== 7.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:11,421 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) is shutting down 2016-08-25 09:01:11,422 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:11,426 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:11,427 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,427 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250007 2016-08-25 09:01:11,428 | INFO | Time-limited test | Session: 0x156bec917250007 closed 2016-08-25 09:01:11,428 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:11,428 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54964 which had sessionid 0x156bec917250007 2016-08-25 09:01:11,432 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:11,432 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:11,433 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:11,433 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,486 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:11,488 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,506 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:11,507 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,511 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:11,515 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,531 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:11,532 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) uptime 0.806 seconds 2016-08-25 09:01:11,532 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) is shutdown Wait for master to start up... 2016-08-25 09:01:11,536 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:11,538 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@28289bdd 2016-08-25 09:01:11,539 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:11,540 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:11,540 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:11,540 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54986 2016-08-25 09:01:11,541 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54986 2016-08-25 09:01:11,542 | INFO | SyncThread:0 | Established session 0x156bec91725000a with negotiated timeout 10000 for client /127.0.0.1:54986 2016-08-25 09:01:11,542 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725000a, negotiated timeout = 10000 2016-08-25 09:01:11,567 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:54987 2016-08-25 09:01:11,572 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,572 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,574 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54987 2016-08-25 09:01:11,574 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:54987 2016-08-25 09:01:11,574 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:11,574 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:11,601 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:11,612 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:11,612 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:11,613 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:11,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000013 2016-08-25 09:01:11,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:11,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000015.log 2016-08-25 09:01:11,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:11,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000016.sst 2016-08-25 09:01:11,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:11,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:11,623 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:11,623 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/5.11 kb and 1/8 files 2016-08-25 09:01:11,623 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000015.log 2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000013 2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000016.sst 2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.16/5.11 kb and 2/8 files 2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:11,625 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:11,625 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:11,625 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:11,625 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.17/5.11 kb and 3/8 files 2016-08-25 09:01:11,626 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.36/5.11 kb and 4/8 files 2016-08-25 09:01:11,628 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.26/5.11 kb and 5/8 files 2016-08-25 09:01:11,629 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.26/5.11 kb and 6/8 files 2016-08-25 09:01:11,630 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.21/5.11 kb and 7/8 files 2016-08-25 09:01:11,630 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.11/5.11 kb and 8/8 files 2016-08-25 09:01:11,631 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:11,632 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/5.11 kb and 1/8 files 2016-08-25 09:01:11,633 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.44/5.11 kb and 2/8 files 2016-08-25 09:01:11,634 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.34/5.11 kb and 3/8 files 2016-08-25 09:01:11,635 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.34/5.11 kb and 4/8 files 2016-08-25 09:01:11,636 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.36/5.11 kb and 5/8 files 2016-08-25 09:01:11,638 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.26/5.11 kb and 6/8 files 2016-08-25 09:01:11,639 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.21/5.11 kb and 7/8 files 2016-08-25 09:01:11,640 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.11/5.11 kb and 8/8 files 2016-08-25 09:01:11,640 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:11,644 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:11,646 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:11,646 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) is starting 2016-08-25 09:01:11,656 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54992 2016-08-25 09:01:11,656 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:11,657 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) started 2016-08-25 09:01:11,657 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:11,657 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:11,657 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-1 ====================================== 7.4. Browse the queue on new master ====================================== ====================================== 7.5. Found ====================================== ====================================== 8.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:11,690 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) is shutting down 2016-08-25 09:01:11,691 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:11,696 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,697 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:11,698 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250008 2016-08-25 09:01:11,699 | INFO | Time-limited test | Session: 0x156bec917250008 closed 2016-08-25 09:01:11,699 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54972 which had sessionid 0x156bec917250008 2016-08-25 09:01:11,699 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:11,704 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:11,704 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:11,704 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:11,705 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:11,756 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:11,760 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,779 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:11,779 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,785 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:11,790 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,819 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:11,820 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) uptime 0.758 seconds 2016-08-25 09:01:11,820 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) is shutdown Wait for master to start up... 2016-08-25 09:01:11,825 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:11,830 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@15bfc674 2016-08-25 09:01:11,830 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:11,831 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:11,831 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:11,831 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54994 2016-08-25 09:01:11,832 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54994 2016-08-25 09:01:11,833 | INFO | SyncThread:0 | Established session 0x156bec91725000b with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54994 2016-08-25 09:01:11,833 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725000b, negotiated timeout = 10000 2016-08-25 09:01:11,874 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:54995 2016-08-25 09:01:11,879 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,879 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:11,881 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54995 2016-08-25 09:01:11,881 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:11,881 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:54995 2016-08-25 09:01:11,881 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:11,896 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,906 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:11,907 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000013 2016-08-25 09:01:11,907 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000016.sst 2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000015.log 2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:11,909 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:11,914 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.26/5.11 kb and 1/8 files 2016-08-25 09:01:11,915 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.16/5.11 kb and 2/8 files 2016-08-25 09:01:11,916 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.06/5.11 kb and 3/8 files 2016-08-25 09:01:11,916 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.06/5.11 kb and 4/8 files 2016-08-25 09:01:11,917 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.24/5.11 kb and 5/8 files 2016-08-25 09:01:11,918 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.26/5.11 kb and 6/8 files 2016-08-25 09:01:11,919 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.16/5.11 kb and 7/8 files 2016-08-25 09:01:11,920 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.11/5.11 kb and 8/8 files 2016-08-25 09:01:11,920 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:11,923 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,924 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:11,924 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) is starting 2016-08-25 09:01:11,928 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:11,929 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:11,929 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000015.log 2016-08-25 09:01:11,929 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000013 2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000016.sst 2016-08-25 09:01:11,933 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55000 2016-08-25 09:01:11,933 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:11,933 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) started 2016-08-25 09:01:11,933 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:11,934 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:11,934 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 8.4. Browse the queue on new master ====================================== 2016-08-25 09:01:11,938 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/5.11 kb and 1/8 files 2016-08-25 09:01:11,939 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/5.11 kb and 2/8 files 2016-08-25 09:01:11,940 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.03/5.11 kb and 3/8 files 2016-08-25 09:01:11,941 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.94/5.11 kb and 4/8 files 2016-08-25 09:01:11,942 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.84/5.11 kb and 5/8 files 2016-08-25 09:01:11,942 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.85/5.11 kb and 6/8 files 2016-08-25 09:01:11,944 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.11/5.11 kb and 7/8 files 2016-08-25 09:01:11,945 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.11/5.11 kb and 8/8 files 2016-08-25 09:01:11,945 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:11,950 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 8.5. Found ====================================== ====================================== 9.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:11,967 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) is shutting down 2016-08-25 09:01:11,968 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:11,976 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:11,976 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:11,977 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:11,977 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250009 2016-08-25 09:01:11,978 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:11,978 | INFO | Time-limited test | Session: 0x156bec917250009 closed 2016-08-25 09:01:11,979 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:11,979 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54979 which had sessionid 0x156bec917250009 2016-08-25 09:01:11,980 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:11,980 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:12,082 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:12,083 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) uptime 0.775 seconds 2016-08-25 09:01:12,083 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) is shutdown Wait for master to start up... 2016-08-25 09:01:12,088 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:12,093 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7dc082e2 2016-08-25 09:01:12,093 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:12,094 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:12,094 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:12,094 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55002 2016-08-25 09:01:12,095 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55002 2016-08-25 09:01:12,096 | INFO | SyncThread:0 | Established session 0x156bec91725000c with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55002 2016-08-25 09:01:12,096 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725000c, negotiated timeout = 10000 2016-08-25 09:01:12,110 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,125 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:12,129 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,161 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:12,162 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,167 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:12,170 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,225 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55003 2016-08-25 09:01:12,232 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,233 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,234 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55003 2016-08-25 09:01:12,235 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:12,235 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55003 2016-08-25 09:01:12,235 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:12,270 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:12,304 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:12,313 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:12,313 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000018.log 2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000018.log 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000016 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000016 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:12,323 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/6.06 kb and 1/8 files 2016-08-25 09:01:12,323 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/6.06 kb and 1/8 files 2016-08-25 09:01:12,324 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/6.06 kb and 2/8 files 2016-08-25 09:01:12,324 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/6.06 kb and 2/8 files 2016-08-25 09:01:12,325 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/6.06 kb and 3/8 files 2016-08-25 09:01:12,325 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.98/6.06 kb and 3/8 files 2016-08-25 09:01:12,326 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.00/6.06 kb and 4/8 files 2016-08-25 09:01:12,326 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.72/6.06 kb and 4/8 files 2016-08-25 09:01:12,327 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.90/6.06 kb and 5/8 files 2016-08-25 09:01:12,327 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.31/6.06 kb and 5/8 files 2016-08-25 09:01:12,328 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.21/6.06 kb and 6/8 files 2016-08-25 09:01:12,328 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.25/6.06 kb and 6/8 files 2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.16/6.06 kb and 7/8 files 2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.15/6.06 kb and 7/8 files 2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.06/6.06 kb and 8/8 files 2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.06/6.06 kb and 8/8 files 2016-08-25 09:01:12,331 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:12,337 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:12,338 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) is starting 2016-08-25 09:01:12,341 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:12,350 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55008 2016-08-25 09:01:12,350 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:12,350 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) started 2016-08-25 09:01:12,350 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:12,351 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:12,351 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 9.4. Browse the queue on new master ====================================== ====================================== 9.5. Found ====================================== ====================================== 10.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:12,385 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) is shutting down 2016-08-25 09:01:12,386 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:12,386 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | socketQueue interrupted - stopping 2016-08-25 09:01:12,387 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | Could not accept connection during shutdown : java.lang.InterruptedException 2016-08-25 09:01:12,392 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,393 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:12,394 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000a 2016-08-25 09:01:12,395 | INFO | Time-limited test | Session: 0x156bec91725000a closed 2016-08-25 09:01:12,395 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:12,395 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54986 which had sessionid 0x156bec91725000a 2016-08-25 09:01:12,399 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:12,399 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:12,399 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:12,400 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:12,478 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:12,479 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,482 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,500 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:12,506 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:12,512 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,514 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:12,515 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) uptime 0.983 seconds 2016-08-25 09:01:12,515 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) is shutdown Wait for master to start up... 2016-08-25 09:01:12,520 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:12,523 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@c09a095 2016-08-25 09:01:12,523 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:12,524 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:12,525 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55010 2016-08-25 09:01:12,525 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:12,525 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55010 2016-08-25 09:01:12,526 | INFO | SyncThread:0 | Established session 0x156bec91725000d with negotiated timeout 10000 for client /127.0.0.1:55010 2016-08-25 09:01:12,526 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725000d, negotiated timeout = 10000 2016-08-25 09:01:12,592 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55011 2016-08-25 09:01:12,599 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,599 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,601 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55011 2016-08-25 09:01:12,601 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:12,602 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55011 2016-08-25 09:01:12,602 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:12,618 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:12,655 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:12,657 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000018.log 2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000016 2016-08-25 09:01:12,659 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:12,659 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:12,672 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/6.06 kb and 1/8 files 2016-08-25 09:01:12,673 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.85/6.06 kb and 2/8 files 2016-08-25 09:01:12,674 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.75/6.06 kb and 3/8 files 2016-08-25 09:01:12,674 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.65/6.06 kb and 4/8 files 2016-08-25 09:01:12,675 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/6.06 kb and 5/8 files 2016-08-25 09:01:12,675 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:12,676 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.14/6.06 kb and 6/8 files 2016-08-25 09:01:12,676 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:12,676 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000018.log 2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.04/6.06 kb and 7/8 files 2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000016 2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:12,678 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.06/6.06 kb and 8/8 files 2016-08-25 09:01:12,678 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:12,681 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:12,682 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) is starting 2016-08-25 09:01:12,686 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/6.06 kb and 1/8 files 2016-08-25 09:01:12,687 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/6.06 kb and 2/8 files 2016-08-25 09:01:12,687 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/6.06 kb and 3/8 files 2016-08-25 09:01:12,688 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.89/6.06 kb and 4/8 files 2016-08-25 09:01:12,689 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.79/6.06 kb and 5/8 files 2016-08-25 09:01:12,690 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.74/6.06 kb and 6/8 files 2016-08-25 09:01:12,691 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.04/6.06 kb and 7/8 files 2016-08-25 09:01:12,692 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.06/6.06 kb and 8/8 files 2016-08-25 09:01:12,692 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:12,696 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55016 2016-08-25 09:01:12,696 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:12,696 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) started 2016-08-25 09:01:12,696 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:12,696 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:12,696 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:12,697 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 Master started: node-1 ====================================== 10.4. Browse the queue on new master ====================================== ====================================== 10.5. Found ====================================== ====================================== 11.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:12,734 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) is shutting down 2016-08-25 09:01:12,735 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:12,738 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:12,740 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000b 2016-08-25 09:01:12,740 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,743 | INFO | Time-limited test | Session: 0x156bec91725000b closed 2016-08-25 09:01:12,743 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:12,743 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54994 which had sessionid 0x156bec91725000b 2016-08-25 09:01:12,744 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:12,744 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:12,744 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:12,745 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:12,847 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:12,847 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) uptime 1.026 seconds 2016-08-25 09:01:12,847 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) is shutdown Wait for master to start up... 2016-08-25 09:01:12,853 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:12,857 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@b52dc3 2016-08-25 09:01:12,857 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:12,858 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:12,859 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:12,859 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55018 2016-08-25 09:01:12,859 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55018 2016-08-25 09:01:12,860 | INFO | SyncThread:0 | Established session 0x156bec91725000e with negotiated timeout 10000 for client /127.0.0.1:55018 2016-08-25 09:01:12,861 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725000e, negotiated timeout = 10000 2016-08-25 09:01:12,867 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:12,871 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,875 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,884 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:12,885 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:12,890 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:12,898 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,974 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55019 2016-08-25 09:01:12,980 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,980 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:12,982 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55019 2016-08-25 09:01:12,982 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:12,982 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55019 2016-08-25 09:01:12,982 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:13,027 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:13,044 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:13,044 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:13,045 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:13,045 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000019 2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000021.log 2016-08-25 09:01:13,047 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:13,048 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:13,049 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000021.log 2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000019 2016-08-25 09:01:13,051 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:13,051 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:13,051 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:13,054 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/7.01 kb and 1/9 files 2016-08-25 09:01:13,055 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.92/7.01 kb and 2/9 files 2016-08-25 09:01:13,056 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/7.01 kb and 3/9 files 2016-08-25 09:01:13,056 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/7.01 kb and 4/9 files 2016-08-25 09:01:13,057 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/7.01 kb and 5/9 files 2016-08-25 09:01:13,058 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.56/7.01 kb and 6/9 files 2016-08-25 09:01:13,059 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.92/7.01 kb and 7/9 files 2016-08-25 09:01:13,060 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.10/7.01 kb and 8/9 files 2016-08-25 09:01:13,060 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/7.01 kb and 1/9 files 2016-08-25 09:01:13,061 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.01/7.01 kb and 9/9 files 2016-08-25 09:01:13,061 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/7.01 kb and 2/9 files 2016-08-25 09:01:13,061 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:13,062 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.98/7.01 kb and 3/9 files 2016-08-25 09:01:13,063 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.88/7.01 kb and 4/9 files 2016-08-25 09:01:13,064 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.79/7.01 kb and 5/9 files 2016-08-25 09:01:13,065 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.14/7.01 kb and 6/9 files 2016-08-25 09:01:13,065 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:13,065 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.09/7.01 kb and 7/9 files 2016-08-25 09:01:13,066 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.11/7.01 kb and 8/9 files 2016-08-25 09:01:13,067 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) is starting 2016-08-25 09:01:13,067 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.01/7.01 kb and 9/9 files 2016-08-25 09:01:13,068 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:13,073 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:13,075 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55024 2016-08-25 09:01:13,075 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:13,076 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) started 2016-08-25 09:01:13,076 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:13,076 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:13,076 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 11.4. Browse the queue on new master ====================================== ====================================== 11.5. Found ====================================== ====================================== 12.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:13,109 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) is shutting down 2016-08-25 09:01:13,110 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:13,113 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:13,114 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:13,114 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000c 2016-08-25 09:01:13,115 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55002 which had sessionid 0x156bec91725000c 2016-08-25 09:01:13,115 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:13,115 | INFO | Time-limited test | Session: 0x156bec91725000c closed 2016-08-25 09:01:13,121 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:13,121 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:13,121 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:13,122 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:13,244 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:13,245 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) uptime 1.161 seconds 2016-08-25 09:01:13,245 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) is shutdown Wait for master to start up... 2016-08-25 09:01:13,249 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:13,256 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4d995fb7 2016-08-25 09:01:13,256 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:13,257 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:13,258 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:13,258 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55026 2016-08-25 09:01:13,258 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55026 2016-08-25 09:01:13,259 | INFO | SyncThread:0 | Established session 0x156bec91725000f with negotiated timeout 10000 for client /127.0.0.1:55026 2016-08-25 09:01:13,259 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725000f, negotiated timeout = 10000 2016-08-25 09:01:13,262 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:13,263 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:13,264 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:13,265 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:13,272 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:13,281 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:13,360 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55027 2016-08-25 09:01:13,366 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:13,366 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:13,368 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55027 2016-08-25 09:01:13,369 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:13,369 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55027 2016-08-25 09:01:13,369 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:13,427 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:13,427 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:13,431 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:13,431 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:13,431 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000021.log 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000019 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000021.log 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000019 2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:13,440 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/7.01 kb and 1/9 files 2016-08-25 09:01:13,441 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/7.01 kb and 2/9 files 2016-08-25 09:01:13,441 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.44/7.01 kb and 3/9 files 2016-08-25 09:01:13,442 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.34/7.01 kb and 4/9 files 2016-08-25 09:01:13,443 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.24/7.01 kb and 5/9 files 2016-08-25 09:01:13,444 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.26/7.01 kb and 6/9 files 2016-08-25 09:01:13,445 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.21/7.01 kb and 7/9 files 2016-08-25 09:01:13,446 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.11/7.01 kb and 8/9 files 2016-08-25 09:01:13,447 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.01/7.01 kb and 9/9 files 2016-08-25 09:01:13,448 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:13,452 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:13,452 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/7.01 kb and 1/9 files 2016-08-25 09:01:13,453 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) is starting 2016-08-25 09:01:13,454 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/7.01 kb and 2/9 files 2016-08-25 09:01:13,455 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.81/7.01 kb and 3/9 files 2016-08-25 09:01:13,456 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.00/7.01 kb and 4/9 files 2016-08-25 09:01:13,457 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.90/7.01 kb and 5/9 files 2016-08-25 09:01:13,458 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.80/7.01 kb and 6/9 files 2016-08-25 09:01:13,459 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.16/7.01 kb and 7/9 files 2016-08-25 09:01:13,460 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.11/7.01 kb and 8/9 files 2016-08-25 09:01:13,461 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.01/7.01 kb and 9/9 files 2016-08-25 09:01:13,462 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:13,462 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55032 2016-08-25 09:01:13,462 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:13,463 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) started 2016-08-25 09:01:13,463 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:13,463 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:13,463 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 12.4. Browse the queue on new master ====================================== 2016-08-25 09:01:13,466 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d ====================================== 12.5. Found ====================================== ====================================== 13.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:13,484 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) is shutting down 2016-08-25 09:01:13,485 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:13,489 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:13,489 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000d 2016-08-25 09:01:13,489 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:13,490 | INFO | Time-limited test | Session: 0x156bec91725000d closed 2016-08-25 09:01:13,490 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:13,490 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55010 which had sessionid 0x156bec91725000d 2016-08-25 09:01:13,494 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:13,494 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:13,494 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:13,494 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:13,594 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:13,594 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) uptime 1.078 seconds 2016-08-25 09:01:13,595 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) is shutdown Wait for master to start up... 2016-08-25 09:01:13,598 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:13,609 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@41991669 2016-08-25 09:01:13,610 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:13,611 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:13,611 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:13,611 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55034 2016-08-25 09:01:13,611 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55034 2016-08-25 09:01:13,612 | INFO | SyncThread:0 | Established session 0x156bec917250010 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55034 2016-08-25 09:01:13,613 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250010, negotiated timeout = 10000 2016-08-25 09:01:13,627 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:13,667 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:13,672 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:13,678 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:13,678 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:13,683 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:13,688 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:13,771 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55035 2016-08-25 09:01:13,777 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:13,777 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:13,779 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55035 2016-08-25 09:01:13,779 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:13,780 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55035 2016-08-25 09:01:13,780 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:13,842 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:13,842 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:13,846 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:13,847 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:13,847 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000022 2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000022 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000024.log 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000024.log 2016-08-25 09:01:13,850 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:13,850 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:13,857 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/7.96 kb and 1/10 files 2016-08-25 09:01:13,858 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/7.96 kb and 2/10 files 2016-08-25 09:01:13,859 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/7.96 kb and 3/10 files 2016-08-25 09:01:13,859 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.11/7.96 kb and 4/10 files 2016-08-25 09:01:13,860 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.01/7.96 kb and 5/10 files 2016-08-25 09:01:13,861 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.91/7.96 kb and 6/10 files 2016-08-25 09:01:13,862 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.81/7.96 kb and 7/10 files 2016-08-25 09:01:13,863 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.82/7.96 kb and 8/10 files 2016-08-25 09:01:13,864 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.77/7.96 kb and 9/10 files 2016-08-25 09:01:13,864 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/7.96 kb and 1/10 files 2016-08-25 09:01:13,864 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.96/7.96 kb and 10/10 files 2016-08-25 09:01:13,864 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/7.96 kb and 2/10 files 2016-08-25 09:01:13,865 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:13,866 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.86/7.96 kb and 3/10 files 2016-08-25 09:01:13,867 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.76/7.96 kb and 4/10 files 2016-08-25 09:01:13,868 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.66/7.96 kb and 5/10 files 2016-08-25 09:01:13,869 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.07/7.96 kb and 6/10 files 2016-08-25 09:01:13,870 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:13,870 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.97/7.96 kb and 7/10 files 2016-08-25 09:01:13,870 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.16/7.96 kb and 8/10 files 2016-08-25 09:01:13,871 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) is starting 2016-08-25 09:01:13,871 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.06/7.96 kb and 9/10 files 2016-08-25 09:01:13,872 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.96/7.96 kb and 10/10 files 2016-08-25 09:01:13,872 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:13,878 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:13,878 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55040 2016-08-25 09:01:13,879 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:13,879 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) started 2016-08-25 09:01:13,879 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:13,879 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:13,879 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-1 ====================================== 13.4. Browse the queue on new master ====================================== ====================================== 13.5. Found ====================================== ====================================== 14.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:13,907 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) is shutting down 2016-08-25 09:01:13,908 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:13,913 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:13,914 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:13,915 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:13,915 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:13,916 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:13,918 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:13,919 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000e 2016-08-25 09:01:13,920 | INFO | Time-limited test | Session: 0x156bec91725000e closed 2016-08-25 09:01:13,920 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55018 which had sessionid 0x156bec91725000e 2016-08-25 09:01:13,920 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:14,038 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:14,039 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:14,039 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,041 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,045 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:14,046 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:14,047 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) uptime 1.199 seconds 2016-08-25 09:01:14,047 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) is shutdown Wait for master to start up... 2016-08-25 09:01:14,050 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:14,052 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,053 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@102873a6 2016-08-25 09:01:14,053 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:14,054 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:14,055 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:14,055 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55042 2016-08-25 09:01:14,055 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55042 2016-08-25 09:01:14,064 | INFO | SyncThread:0 | Established session 0x156bec917250011 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55042 2016-08-25 09:01:14,064 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250011, negotiated timeout = 10000 2016-08-25 09:01:14,133 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55043 2016-08-25 09:01:14,139 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,139 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,140 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55043 2016-08-25 09:01:14,141 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:14,141 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55043 2016-08-25 09:01:14,141 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:14,154 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:14,165 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:14,167 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000022 2016-08-25 09:01:14,167 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:14,167 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000024.log 2016-08-25 09:01:14,175 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.41/7.96 kb and 1/10 files 2016-08-25 09:01:14,175 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.31/7.96 kb and 2/10 files 2016-08-25 09:01:14,176 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.33/7.96 kb and 3/10 files 2016-08-25 09:01:14,177 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.23/7.96 kb and 4/10 files 2016-08-25 09:01:14,177 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.13/7.96 kb and 5/10 files 2016-08-25 09:01:14,178 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.03/7.96 kb and 6/10 files 2016-08-25 09:01:14,179 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.97/7.96 kb and 7/10 files 2016-08-25 09:01:14,180 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.87/7.96 kb and 8/10 files 2016-08-25 09:01:14,181 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.77/7.96 kb and 9/10 files 2016-08-25 09:01:14,182 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.96/7.96 kb and 10/10 files 2016-08-25 09:01:14,182 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:14,185 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:14,187 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) is starting 2016-08-25 09:01:14,196 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55047 2016-08-25 09:01:14,196 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:14,197 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) started 2016-08-25 09:01:14,197 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:14,197 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:01:14,197 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 14.4. Browse the queue on new master ====================================== 2016-08-25 09:01:14,203 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:14,219 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000022 2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000024.log 2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst ====================================== 14.5. Found ====================================== ====================================== 15.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:14,234 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) is shutting down 2016-08-25 09:01:14,234 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:14,239 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,240 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:14,240 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:14,240 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:14,241 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:14,242 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:14,242 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000f 2016-08-25 09:01:14,253 | INFO | Time-limited test | Session: 0x156bec91725000f closed 2016-08-25 09:01:14,253 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55026 which had sessionid 0x156bec91725000f 2016-08-25 09:01:14,253 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:14,327 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:14,327 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,330 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,357 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:14,358 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) uptime 1.112 seconds 2016-08-25 09:01:14,358 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) is shutdown Wait for master to start up... 2016-08-25 09:01:14,361 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:14,365 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6ac47476 2016-08-25 09:01:14,365 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:14,366 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:14,366 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:14,366 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55050 2016-08-25 09:01:14,366 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55050 2016-08-25 09:01:14,367 | INFO | SyncThread:0 | Established session 0x156bec917250012 with negotiated timeout 10000 for client /127.0.0.1:55050 2016-08-25 09:01:14,367 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250012, negotiated timeout = 10000 2016-08-25 09:01:14,376 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,425 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:14,430 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:14,436 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,519 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55051 2016-08-25 09:01:14,525 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,525 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,525 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55051 2016-08-25 09:01:14,525 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:14,526 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55051 2016-08-25 09:01:14,526 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:14,577 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:14,581 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:14,587 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:14,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000025 2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000027.log 2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:14,591 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:14,593 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:14,593 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:14,593 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000025 2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000027.log 2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:14,595 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:14,595 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:14,597 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/8.91 kb and 1/11 files 2016-08-25 09:01:14,598 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/8.91 kb and 2/11 files 2016-08-25 09:01:14,599 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/8.91 kb and 3/11 files 2016-08-25 09:01:14,599 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/8.91 kb and 4/11 files 2016-08-25 09:01:14,600 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.62/8.91 kb and 5/11 files 2016-08-25 09:01:14,601 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.08/8.91 kb and 6/11 files 2016-08-25 09:01:14,602 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.26/8.91 kb and 7/11 files 2016-08-25 09:01:14,603 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.21/8.91 kb and 8/11 files 2016-08-25 09:01:14,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/8.91 kb and 1/11 files 2016-08-25 09:01:14,603 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.11/8.91 kb and 9/11 files 2016-08-25 09:01:14,604 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/8.91 kb and 2/11 files 2016-08-25 09:01:14,604 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.01/8.91 kb and 10/11 files 2016-08-25 09:01:14,605 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.91/8.91 kb and 11/11 files 2016-08-25 09:01:14,605 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:14,605 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/8.91 kb and 3/11 files 2016-08-25 09:01:14,606 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.27/8.91 kb and 4/11 files 2016-08-25 09:01:14,607 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.17/8.91 kb and 5/11 files 2016-08-25 09:01:14,608 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.08/8.91 kb and 6/11 files 2016-08-25 09:01:14,609 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.26/8.91 kb and 7/11 files 2016-08-25 09:01:14,610 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.21/8.91 kb and 8/11 files 2016-08-25 09:01:14,610 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:14,611 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.11/8.91 kb and 9/11 files 2016-08-25 09:01:14,611 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.01/8.91 kb and 10/11 files 2016-08-25 09:01:14,611 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) is starting 2016-08-25 09:01:14,612 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.91/8.91 kb and 11/11 files 2016-08-25 09:01:14,612 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:14,618 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55056 2016-08-25 09:01:14,618 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:14,618 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) started 2016-08-25 09:01:14,618 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:14,618 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:14,619 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:14,620 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 Master started: node-1 ====================================== 15.4. Browse the queue on new master ====================================== ====================================== 15.5. Found ====================================== ====================================== 16.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:14,646 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) is shutting down 2016-08-25 09:01:14,646 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:14,652 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:14,653 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250011 2016-08-25 09:01:14,653 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:01:14,653 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,653 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55042 which had sessionid 0x156bec917250011 2016-08-25 09:01:14,654 | INFO | Time-limited test | Session: 0x156bec917250011 closed 2016-08-25 09:01:14,654 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:14,656 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:14,656 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:14,656 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:14,656 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:14,658 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,783 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:14,784 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) uptime 0.736 seconds 2016-08-25 09:01:14,784 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) is shutdown Wait for master to start up... 2016-08-25 09:01:14,788 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:14,789 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:14,794 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,795 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@37f832df 2016-08-25 09:01:14,795 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:14,796 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:14,796 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:14,796 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:14,797 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:14,797 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55058 2016-08-25 09:01:14,797 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55058 2016-08-25 09:01:14,798 | INFO | SyncThread:0 | Established session 0x156bec917250013 with negotiated timeout 10000 for client /127.0.0.1:55058 2016-08-25 09:01:14,798 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250013, negotiated timeout = 10000 2016-08-25 09:01:14,802 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:14,813 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,895 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55059 2016-08-25 09:01:14,901 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,901 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:14,902 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55059 2016-08-25 09:01:14,902 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:14,903 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55059 2016-08-25 09:01:14,903 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:14,917 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:14,968 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:14,968 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:14,969 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:14,971 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:14,971 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000027.log 2016-08-25 09:01:14,971 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:14,971 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000025 2016-08-25 09:01:14,977 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/8.91 kb and 1/11 files 2016-08-25 09:01:14,978 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:14,978 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.92/8.91 kb and 2/11 files 2016-08-25 09:01:14,978 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.82/8.91 kb and 3/11 files 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000025 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000027.log 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.72/8.91 kb and 4/11 files 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.62/8.91 kb and 5/11 files 2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.52/8.91 kb and 6/11 files 2016-08-25 09:01:14,981 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.42/8.91 kb and 7/11 files 2016-08-25 09:01:14,981 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.32/8.91 kb and 8/11 files 2016-08-25 09:01:14,982 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.50/8.91 kb and 9/11 files 2016-08-25 09:01:14,982 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.45/8.91 kb and 10/11 files 2016-08-25 09:01:14,983 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.91/8.91 kb and 11/11 files 2016-08-25 09:01:14,983 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:14,986 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/8.91 kb and 1/11 files 2016-08-25 09:01:14,987 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:14,988 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.36/8.91 kb and 2/11 files 2016-08-25 09:01:14,989 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.26/8.91 kb and 3/11 files 2016-08-25 09:01:14,990 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) is starting 2016-08-25 09:01:14,990 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.44/8.91 kb and 4/11 files 2016-08-25 09:01:14,991 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.34/8.91 kb and 5/11 files 2016-08-25 09:01:14,991 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.36/8.91 kb and 6/11 files 2016-08-25 09:01:14,993 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.26/8.91 kb and 7/11 files 2016-08-25 09:01:14,993 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.16/8.91 kb and 8/11 files 2016-08-25 09:01:14,995 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.06/8.91 kb and 9/11 files 2016-08-25 09:01:14,995 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.96/8.91 kb and 10/11 files 2016-08-25 09:01:14,996 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55064 2016-08-25 09:01:14,996 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.91/8.91 kb and 11/11 files 2016-08-25 09:01:14,996 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:14,996 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:14,996 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) started 2016-08-25 09:01:14,997 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:14,997 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:14,997 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-3 ====================================== 16.4. Browse the queue on new master ====================================== 2016-08-25 09:01:15,002 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 16.5. Found ====================================== ====================================== 17.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:15,023 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) is shutting down 2016-08-25 09:01:15,024 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:15,029 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,032 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:15,034 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250010 2016-08-25 09:01:15,035 | INFO | Time-limited test | Session: 0x156bec917250010 closed 2016-08-25 09:01:15,035 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55034 which had sessionid 0x156bec917250010 2016-08-25 09:01:15,035 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:15,037 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:15,037 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:15,037 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:15,038 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:15,140 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:15,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) uptime 1.546 seconds 2016-08-25 09:01:15,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) is shutdown Wait for master to start up... 2016-08-25 09:01:15,145 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:15,150 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@184797a 2016-08-25 09:01:15,151 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:15,152 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:15,152 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:15,152 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55066 2016-08-25 09:01:15,153 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55066 2016-08-25 09:01:15,153 | INFO | SyncThread:0 | Established session 0x156bec917250014 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55066 2016-08-25 09:01:15,154 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250014, negotiated timeout = 10000 2016-08-25 09:01:15,174 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,176 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:15,179 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,232 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:15,232 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,240 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:15,245 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:15,337 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55067 2016-08-25 09:01:15,343 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:15,343 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:15,344 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55067 2016-08-25 09:01:15,344 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55067 2016-08-25 09:01:15,344 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:15,344 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:15,359 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:15,400 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:15,410 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:15,411 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:15,412 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:15,412 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000030.log 2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000028 2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000030.log 2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000031.sst 2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000031.sst 2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000028 2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:15,424 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/9.86 kb and 1/13 files 2016-08-25 09:01:15,424 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.86 kb and 1/13 files 2016-08-25 09:01:15,424 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/9.86 kb and 2/13 files 2016-08-25 09:01:15,424 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.92/9.86 kb and 2/13 files 2016-08-25 09:01:15,425 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/9.86 kb and 3/13 files 2016-08-25 09:01:15,425 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.82/9.86 kb and 3/13 files 2016-08-25 09:01:15,426 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.65/9.86 kb and 4/13 files 2016-08-25 09:01:15,426 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.72/9.86 kb and 4/13 files 2016-08-25 09:01:15,427 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.90/9.86 kb and 5/13 files 2016-08-25 09:01:15,427 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.55/9.86 kb and 5/13 files 2016-08-25 09:01:15,428 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.56/9.86 kb and 6/13 files 2016-08-25 09:01:15,428 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.41/9.86 kb and 6/13 files 2016-08-25 09:01:15,428 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.41/9.86 kb and 7/13 files 2016-08-25 09:01:15,428 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.47/9.86 kb and 7/13 files 2016-08-25 09:01:15,429 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.37/9.86 kb and 8/13 files 2016-08-25 09:01:15,429 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.31/9.86 kb and 8/13 files 2016-08-25 09:01:15,430 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.55/9.86 kb and 9/13 files 2016-08-25 09:01:15,430 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.21/9.86 kb and 9/13 files 2016-08-25 09:01:15,430 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.11/9.86 kb and 10/13 files 2016-08-25 09:01:15,430 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.45/9.86 kb and 10/13 files 2016-08-25 09:01:15,431 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.06/9.86 kb and 11/13 files 2016-08-25 09:01:15,431 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.35/9.86 kb and 11/13 files 2016-08-25 09:01:15,432 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.96/9.86 kb and 12/13 files 2016-08-25 09:01:15,432 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.35/9.86 kb and 12/13 files 2016-08-25 09:01:15,432 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.86/9.86 kb and 13/13 files 2016-08-25 09:01:15,432 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.86/9.86 kb and 13/13 files 2016-08-25 09:01:15,433 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:15,433 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:15,437 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:15,437 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:15,438 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) is starting 2016-08-25 09:01:15,444 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55072 2016-08-25 09:01:15,452 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:15,452 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) started 2016-08-25 09:01:15,452 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:15,452 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:15,453 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-2 ====================================== 17.4. Browse the queue on new master ====================================== ====================================== 17.5. Found ====================================== ====================================== 18.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:15,486 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) is shutting down 2016-08-25 09:01:15,488 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:15,494 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,494 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:15,495 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250012 2016-08-25 09:01:15,496 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55050 which had sessionid 0x156bec917250012 2016-08-25 09:01:15,496 | INFO | Time-limited test | Session: 0x156bec917250012 closed 2016-08-25 09:01:15,496 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:15,500 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:15,500 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:15,501 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:15,501 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:15,616 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:15,616 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) uptime 1.258 seconds 2016-08-25 09:01:15,617 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) is shutdown Wait for master to start up... 2016-08-25 09:01:15,620 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:15,624 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@519b1696 2016-08-25 09:01:15,624 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:15,626 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:15,626 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:15,626 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55074 2016-08-25 09:01:15,627 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55074 2016-08-25 09:01:15,628 | INFO | SyncThread:0 | Established session 0x156bec917250015 with negotiated timeout 10000 for client /127.0.0.1:55074 2016-08-25 09:01:15,628 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250015, negotiated timeout = 10000 2016-08-25 09:01:15,647 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,679 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:15,679 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,686 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,691 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:15,697 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:15,702 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:15,788 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55075 2016-08-25 09:01:15,792 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:15,794 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:15,794 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55075 2016-08-25 09:01:15,795 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:15,795 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55075 2016-08-25 09:01:15,795 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:15,858 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:15,870 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:15,870 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:15,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:15,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000030.log 2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000031.sst 2016-08-25 09:01:15,873 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:15,873 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:15,873 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000028 2016-08-25 09:01:15,874 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:15,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:15,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000031.sst 2016-08-25 09:01:15,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:15,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000028 2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:15,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:15,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:15,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000030.log 2016-08-25 09:01:15,882 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.86 kb and 1/13 files 2016-08-25 09:01:15,883 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/9.86 kb and 2/13 files 2016-08-25 09:01:15,884 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.99/9.86 kb and 3/13 files 2016-08-25 09:01:15,884 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.00/9.86 kb and 4/13 files 2016-08-25 09:01:15,885 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/9.86 kb and 1/13 files 2016-08-25 09:01:15,885 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.95/9.86 kb and 5/13 files 2016-08-25 09:01:15,886 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/9.86 kb and 2/13 files 2016-08-25 09:01:15,886 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.85/9.86 kb and 6/13 files 2016-08-25 09:01:15,887 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/9.86 kb and 3/13 files 2016-08-25 09:01:15,887 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.75/9.86 kb and 7/13 files 2016-08-25 09:01:15,887 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/9.86 kb and 4/13 files 2016-08-25 09:01:15,888 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.65/9.86 kb and 8/13 files 2016-08-25 09:01:15,888 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/9.86 kb and 5/13 files 2016-08-25 09:01:15,888 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.55/9.86 kb and 9/13 files 2016-08-25 09:01:15,889 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.11/9.86 kb and 6/13 files 2016-08-25 09:01:15,889 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.55/9.86 kb and 10/13 files 2016-08-25 09:01:15,890 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.45/9.86 kb and 11/13 files 2016-08-25 09:01:15,890 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.01/9.86 kb and 7/13 files 2016-08-25 09:01:15,890 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.35/9.86 kb and 12/13 files 2016-08-25 09:01:15,890 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.91/9.86 kb and 8/13 files 2016-08-25 09:01:15,891 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.86/9.86 kb and 13/13 files 2016-08-25 09:01:15,891 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:15,891 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.81/9.86 kb and 9/13 files 2016-08-25 09:01:15,892 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.83/9.86 kb and 10/13 files 2016-08-25 09:01:15,893 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.72/9.86 kb and 11/13 files 2016-08-25 09:01:15,894 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.67/9.86 kb and 12/13 files 2016-08-25 09:01:15,895 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.86/9.86 kb and 13/13 files 2016-08-25 09:01:15,895 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:15,896 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:15,898 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:15,898 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) is starting 2016-08-25 09:01:15,903 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55080 2016-08-25 09:01:15,903 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:15,903 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) started 2016-08-25 09:01:15,903 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:15,904 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:15,904 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-1 ====================================== 18.4. Browse the queue on new master ====================================== ====================================== 18.5. Found ====================================== ====================================== 19.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:15,928 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) is shutting down 2016-08-25 09:01:15,929 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:15,934 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:15,935 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:15,936 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:15,936 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:15,936 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:15,936 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:15,938 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250013 2016-08-25 09:01:15,939 | INFO | Time-limited test | Session: 0x156bec917250013 closed 2016-08-25 09:01:15,939 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55058 which had sessionid 0x156bec917250013 2016-08-25 09:01:15,939 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:16,042 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:16,043 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) uptime 1.258 seconds 2016-08-25 09:01:16,043 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) is shutdown Wait for master to start up... 2016-08-25 09:01:16,046 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:16,048 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7505d7e7 2016-08-25 09:01:16,048 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:16,049 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:16,049 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:16,049 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55082 2016-08-25 09:01:16,049 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55082 2016-08-25 09:01:16,050 | INFO | SyncThread:0 | Established session 0x156bec917250016 with negotiated timeout 10000 for client /127.0.0.1:55082 2016-08-25 09:01:16,050 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250016, negotiated timeout = 10000 2016-08-25 09:01:16,068 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:16,140 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:16,140 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:16,141 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:16,144 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:16,147 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:16,152 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:16,237 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55083 2016-08-25 09:01:16,242 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:16,242 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:16,243 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55083 2016-08-25 09:01:16,243 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:16,244 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55083 2016-08-25 09:01:16,244 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:16,319 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:16,319 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:16,325 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:16,325 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:16,326 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000031 2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000033.log 2016-08-25 09:01:16,329 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:16,329 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:16,330 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:16,330 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000031 2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:16,332 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000033.log 2016-08-25 09:01:16,332 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:16,332 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:16,348 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/10.81 kb and 1/13 files 2016-08-25 09:01:16,348 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/10.81 kb and 1/13 files 2016-08-25 09:01:16,349 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.92/10.81 kb and 2/13 files 2016-08-25 09:01:16,349 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.46/10.81 kb and 2/13 files 2016-08-25 09:01:16,350 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.82/10.81 kb and 3/13 files 2016-08-25 09:01:16,350 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.36/10.81 kb and 3/13 files 2016-08-25 09:01:16,351 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.72/10.81 kb and 4/13 files 2016-08-25 09:01:16,351 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.26/10.81 kb and 4/13 files 2016-08-25 09:01:16,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.62/10.81 kb and 5/13 files 2016-08-25 09:01:16,352 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.16/10.81 kb and 5/13 files 2016-08-25 09:01:16,352 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.06/10.81 kb and 6/13 files 2016-08-25 09:01:16,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.52/10.81 kb and 6/13 files 2016-08-25 09:01:16,353 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.96/10.81 kb and 7/13 files 2016-08-25 09:01:16,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.08/10.81 kb and 7/13 files 2016-08-25 09:01:16,353 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.86/10.81 kb and 8/13 files 2016-08-25 09:01:16,354 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.98/10.81 kb and 8/13 files 2016-08-25 09:01:16,354 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.76/10.81 kb and 9/13 files 2016-08-25 09:01:16,354 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.88/10.81 kb and 9/13 files 2016-08-25 09:01:16,355 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.71/10.81 kb and 10/13 files 2016-08-25 09:01:16,355 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.78/10.81 kb and 10/13 files 2016-08-25 09:01:16,355 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.89/10.81 kb and 11/13 files 2016-08-25 09:01:16,356 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.96/10.81 kb and 11/13 files 2016-08-25 09:01:16,356 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.91/10.81 kb and 12/13 files 2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.81/10.81 kb and 13/13 files 2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.91/10.81 kb and 12/13 files 2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.81/10.81 kb and 13/13 files 2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:16,361 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:16,362 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) is starting 2016-08-25 09:01:16,362 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:16,367 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55088 2016-08-25 09:01:16,367 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:16,367 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) started 2016-08-25 09:01:16,367 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:16,368 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:16,368 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-3 ====================================== 19.4. Browse the queue on new master ====================================== ====================================== 19.5. Found ====================================== ====================================== 20.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:16,389 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) is shutting down 2016-08-25 09:01:16,390 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:16,395 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:16,396 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:16,398 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250014 2016-08-25 09:01:16,399 | INFO | Time-limited test | Session: 0x156bec917250014 closed 2016-08-25 09:01:16,399 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:16,399 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55066 which had sessionid 0x156bec917250014 2016-08-25 09:01:16,400 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:16,400 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:16,404 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:16,404 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:16,525 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:16,526 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) uptime 1.382 seconds 2016-08-25 09:01:16,526 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) is shutdown Wait for master to start up... 2016-08-25 09:01:16,529 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:16,532 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@177a26e 2016-08-25 09:01:16,533 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:16,533 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:16,534 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:16,534 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55090 2016-08-25 09:01:16,535 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55090 2016-08-25 09:01:16,536 | INFO | SyncThread:0 | Established session 0x156bec917250017 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55090 2016-08-25 09:01:16,536 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250017, negotiated timeout = 10000 2016-08-25 09:01:16,550 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:16,556 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:16,556 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:16,561 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:16,578 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:16,586 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:16,592 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:16,675 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55091 2016-08-25 09:01:16,680 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:16,680 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:16,681 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55091 2016-08-25 09:01:16,681 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55091 2016-08-25 09:01:16,681 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:16,682 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:16,696 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:16,749 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:16,757 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:16,758 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:16,758 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000033.log 2016-08-25 09:01:16,758 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000031 2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:16,760 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:16,763 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:16,764 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000033.log 2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000031 2016-08-25 09:01:16,770 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/10.81 kb and 1/13 files 2016-08-25 09:01:16,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.08/10.81 kb and 2/13 files 2016-08-25 09:01:16,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.03/10.81 kb and 3/13 files 2016-08-25 09:01:16,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.93/10.81 kb and 4/13 files 2016-08-25 09:01:16,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/10.81 kb and 5/13 files 2016-08-25 09:01:16,774 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/10.81 kb and 6/13 files 2016-08-25 09:01:16,775 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/10.81 kb and 7/13 files 2016-08-25 09:01:16,776 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.53/10.81 kb and 8/13 files 2016-08-25 09:01:16,776 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/10.81 kb and 1/13 files 2016-08-25 09:01:16,776 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.43/10.81 kb and 9/13 files 2016-08-25 09:01:16,777 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/10.81 kb and 10/13 files 2016-08-25 09:01:16,778 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.89/10.81 kb and 11/13 files 2016-08-25 09:01:16,778 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/10.81 kb and 2/13 files 2016-08-25 09:01:16,779 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.79/10.81 kb and 12/13 files 2016-08-25 09:01:16,779 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.81/10.81 kb and 13/13 files 2016-08-25 09:01:16,780 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:16,779 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/10.81 kb and 3/13 files 2016-08-25 09:01:16,780 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.60/10.81 kb and 4/13 files 2016-08-25 09:01:16,781 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.50/10.81 kb and 5/13 files 2016-08-25 09:01:16,782 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.40/10.81 kb and 6/13 files 2016-08-25 09:01:16,783 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:16,783 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.30/10.81 kb and 7/13 files 2016-08-25 09:01:16,784 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.32/10.81 kb and 8/13 files 2016-08-25 09:01:16,784 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) is starting 2016-08-25 09:01:16,785 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.26/10.81 kb and 9/13 files 2016-08-25 09:01:16,786 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.16/10.81 kb and 10/13 files 2016-08-25 09:01:16,786 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.06/10.81 kb and 11/13 files 2016-08-25 09:01:16,787 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.25/10.81 kb and 12/13 files 2016-08-25 09:01:16,788 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.81/10.81 kb and 13/13 files 2016-08-25 09:01:16,788 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:16,790 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55096 2016-08-25 09:01:16,790 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:16,791 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) started 2016-08-25 09:01:16,791 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:16,791 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:16,791 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:16,793 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d Master started: node-2 ====================================== 20.4. Browse the queue on new master ====================================== ====================================== 20.5. Found ====================================== ====================================== 21.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:16,813 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) is shutting down 2016-08-25 09:01:16,814 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:16,814 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | socketQueue interrupted - stopping 2016-08-25 09:01:16,815 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | Could not accept connection during shutdown : java.lang.InterruptedException 2016-08-25 09:01:16,822 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:16,827 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:16,828 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250015 2016-08-25 09:01:16,829 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55074 which had sessionid 0x156bec917250015 2016-08-25 09:01:16,830 | INFO | Time-limited test | Session: 0x156bec917250015 closed 2016-08-25 09:01:16,830 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:16,833 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:16,833 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:16,833 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:16,833 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:16,933 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:16,934 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) uptime 1.316 seconds 2016-08-25 09:01:16,934 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) is shutdown Wait for master to start up... 2016-08-25 09:01:16,936 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:16,940 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6f900132 2016-08-25 09:01:16,940 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:16,943 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:16,944 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:16,944 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55098 2016-08-25 09:01:16,944 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55098 2016-08-25 09:01:16,945 | INFO | SyncThread:0 | Established session 0x156bec917250018 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55098 2016-08-25 09:01:16,945 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250018, negotiated timeout = 10000 2016-08-25 09:01:16,956 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:17,124 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:17,128 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:17,147 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:17,148 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:17,153 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:17,170 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:17,312 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55099 2016-08-25 09:01:17,317 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:17,317 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:17,319 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55099 2016-08-25 09:01:17,319 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:17,320 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55099 2016-08-25 09:01:17,321 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:17,412 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:17,431 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:17,441 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:17,441 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:17,441 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:17,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000037.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000036.log 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000034 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000036.log 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000034 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000037.sst 2016-08-25 09:01:17,452 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/11.75 kb and 1/15 files 2016-08-25 09:01:17,453 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/11.75 kb and 2/15 files 2016-08-25 09:01:17,454 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/11.75 kb and 3/15 files 2016-08-25 09:01:17,454 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/11.75 kb and 4/15 files 2016-08-25 09:01:17,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/11.75 kb and 5/15 files 2016-08-25 09:01:17,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/11.75 kb and 6/15 files 2016-08-25 09:01:17,456 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/11.75 kb and 7/15 files 2016-08-25 09:01:17,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.74/11.75 kb and 8/15 files 2016-08-25 09:01:17,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.64/11.75 kb and 9/15 files 2016-08-25 09:01:17,458 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.25/11.75 kb and 10/15 files 2016-08-25 09:01:17,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.15/11.75 kb and 11/15 files 2016-08-25 09:01:17,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.06/11.75 kb and 12/15 files 2016-08-25 09:01:17,460 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/11.75 kb and 13/15 files 2016-08-25 09:01:17,460 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/11.75 kb and 1/15 files 2016-08-25 09:01:17,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/11.75 kb and 14/15 files 2016-08-25 09:01:17,461 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/11.75 kb and 2/15 files 2016-08-25 09:01:17,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.75/11.75 kb and 15/15 files 2016-08-25 09:01:17,461 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:17,462 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/11.75 kb and 3/15 files 2016-08-25 09:01:17,462 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.31/11.75 kb and 4/15 files 2016-08-25 09:01:17,464 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.21/11.75 kb and 5/15 files 2016-08-25 09:01:17,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.11/11.75 kb and 6/15 files 2016-08-25 09:01:17,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/11.75 kb and 7/15 files 2016-08-25 09:01:17,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.24/11.75 kb and 8/15 files 2016-08-25 09:01:17,466 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:17,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.14/11.75 kb and 9/15 files 2016-08-25 09:01:17,468 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) is starting 2016-08-25 09:01:17,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.16/11.75 kb and 10/15 files 2016-08-25 09:01:17,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.06/11.75 kb and 11/15 files 2016-08-25 09:01:17,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.96/11.75 kb and 12/15 files 2016-08-25 09:01:17,473 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55104 2016-08-25 09:01:17,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.86/11.75 kb and 13/15 files 2016-08-25 09:01:17,473 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:17,473 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) started 2016-08-25 09:01:17,474 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:17,474 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:17,474 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:17,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/11.75 kb and 14/15 files 2016-08-25 09:01:17,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/11.75 kb and 15/15 files 2016-08-25 09:01:17,475 | INFO | hawtdispatch-DEFAULT-1 | Attached Master started: node-1 ====================================== 21.4. Browse the queue on new master ====================================== 2016-08-25 09:01:17,481 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 21.5. Found ====================================== ====================================== 22.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:17,494 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) is shutting down 2016-08-25 09:01:17,494 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:17,496 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:17,497 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250016 2016-08-25 09:01:17,497 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:17,498 | INFO | Time-limited test | Session: 0x156bec917250016 closed 2016-08-25 09:01:17,498 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:17,498 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55082 which had sessionid 0x156bec917250016 2016-08-25 09:01:17,505 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:17,505 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:17,505 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:17,505 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:17,627 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:17,628 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) uptime 1.584 seconds 2016-08-25 09:01:17,628 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) is shutdown Wait for master to start up... 2016-08-25 09:01:17,631 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:17,634 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6e57f107 2016-08-25 09:01:17,634 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:17,635 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:17,636 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55106 2016-08-25 09:01:17,636 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:17,636 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55106 2016-08-25 09:01:17,637 | INFO | SyncThread:0 | Established session 0x156bec917250019 with negotiated timeout 10000 for client /127.0.0.1:55106 2016-08-25 09:01:17,637 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250019, negotiated timeout = 10000 2016-08-25 09:01:17,649 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:17,735 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:17,739 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:17,746 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:17,746 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:17,756 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:17,764 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:17,909 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55107 2016-08-25 09:01:17,916 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:17,917 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:17,918 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55107 2016-08-25 09:01:17,918 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55107 2016-08-25 09:01:17,918 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:17,918 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:17,933 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:18,014 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:18,015 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:18,015 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:18,016 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000034 2016-08-25 09:01:18,016 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:18,016 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:18,016 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000036.log 2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000037.sst 2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:18,019 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:18,020 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000034 2016-08-25 09:01:18,020 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:18,020 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:18,020 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000036.log 2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000037.sst 2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:18,022 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:18,022 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:18,022 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:18,022 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:18,027 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/11.75 kb and 1/15 files 2016-08-25 09:01:18,028 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.51/11.75 kb and 2/15 files 2016-08-25 09:01:18,028 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.41/11.75 kb and 3/15 files 2016-08-25 09:01:18,029 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.31/11.75 kb and 4/15 files 2016-08-25 09:01:18,030 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.21/11.75 kb and 5/15 files 2016-08-25 09:01:18,030 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.16/11.75 kb and 6/15 files 2016-08-25 09:01:18,031 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.34/11.75 kb and 7/15 files 2016-08-25 09:01:18,032 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.25/11.75 kb and 8/15 files 2016-08-25 09:01:18,032 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/11.75 kb and 1/15 files 2016-08-25 09:01:18,033 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.25/11.75 kb and 9/15 files 2016-08-25 09:01:18,034 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.51/11.75 kb and 2/15 files 2016-08-25 09:01:18,034 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.26/11.75 kb and 10/15 files 2016-08-25 09:01:18,035 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.53/11.75 kb and 3/15 files 2016-08-25 09:01:18,035 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.16/11.75 kb and 11/15 files 2016-08-25 09:01:18,035 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.06/11.75 kb and 12/15 files 2016-08-25 09:01:18,035 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.43/11.75 kb and 4/15 files 2016-08-25 09:01:18,036 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.96/11.75 kb and 13/15 files 2016-08-25 09:01:18,036 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/11.75 kb and 5/15 files 2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.86/11.75 kb and 14/15 files 2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.51/11.75 kb and 6/15 files 2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.75/11.75 kb and 15/15 files 2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.41/11.75 kb and 7/15 files 2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:18,038 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.35/11.75 kb and 8/15 files 2016-08-25 09:01:18,039 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.35/11.75 kb and 9/15 files 2016-08-25 09:01:18,040 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.25/11.75 kb and 10/15 files 2016-08-25 09:01:18,040 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.15/11.75 kb and 11/15 files 2016-08-25 09:01:18,041 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.05/11.75 kb and 12/15 files 2016-08-25 09:01:18,043 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.96/11.75 kb and 13/15 files 2016-08-25 09:01:18,044 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:18,044 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.86/11.75 kb and 14/15 files 2016-08-25 09:01:18,046 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/11.75 kb and 15/15 files 2016-08-25 09:01:18,046 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:18,047 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) is starting 2016-08-25 09:01:18,052 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55112 2016-08-25 09:01:18,052 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:18,053 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:18,053 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) started 2016-08-25 09:01:18,053 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:18,053 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:18,053 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-3 ====================================== 22.4. Browse the queue on new master ====================================== ====================================== 22.5. Found ====================================== ====================================== 23.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:18,071 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) is shutting down 2016-08-25 09:01:18,072 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:18,078 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:18,079 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:18,080 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:18,080 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:18,080 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:18,081 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:18,081 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250017 2016-08-25 09:01:18,082 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55090 which had sessionid 0x156bec917250017 2016-08-25 09:01:18,083 | INFO | Time-limited test | Session: 0x156bec917250017 closed 2016-08-25 09:01:18,083 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:18,188 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:18,189 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) uptime 1.662 seconds 2016-08-25 09:01:18,189 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) is shutdown Wait for master to start up... 2016-08-25 09:01:18,192 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:18,194 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@727c55d0 2016-08-25 09:01:18,194 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:18,195 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:18,195 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:18,195 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55114 2016-08-25 09:01:18,196 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55114 2016-08-25 09:01:18,196 | INFO | SyncThread:0 | Established session 0x156bec91725001a with negotiated timeout 10000 for client /127.0.0.1:55114 2016-08-25 09:01:18,197 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725001a, negotiated timeout = 10000 2016-08-25 09:01:18,205 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:18,338 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:18,340 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:18,348 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:18,348 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:18,353 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:18,358 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:18,486 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55115 2016-08-25 09:01:18,492 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:18,492 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:18,494 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55115 2016-08-25 09:01:18,494 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:18,495 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55115 2016-08-25 09:01:18,495 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:18,599 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:18,601 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:18,604 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:18,605 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:18,605 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000037 2016-08-25 09:01:18,605 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000039.log 2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000039.log 2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000037 2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:18,617 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/12.71 kb and 1/15 files 2016-08-25 09:01:18,617 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.56/12.71 kb and 2/15 files 2016-08-25 09:01:18,618 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.46/12.71 kb and 3/15 files 2016-08-25 09:01:18,619 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/12.71 kb and 1/15 files 2016-08-25 09:01:18,619 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.36/12.71 kb and 4/15 files 2016-08-25 09:01:18,619 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.26/12.71 kb and 5/15 files 2016-08-25 09:01:18,619 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/12.71 kb and 2/15 files 2016-08-25 09:01:18,620 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/12.71 kb and 3/15 files 2016-08-25 09:01:18,620 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.16/12.71 kb and 6/15 files 2016-08-25 09:01:18,621 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/12.71 kb and 4/15 files 2016-08-25 09:01:18,621 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.06/12.71 kb and 7/15 files 2016-08-25 09:01:18,621 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.96/12.71 kb and 8/15 files 2016-08-25 09:01:18,621 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.78/12.71 kb and 5/15 files 2016-08-25 09:01:18,622 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.86/12.71 kb and 9/15 files 2016-08-25 09:01:18,622 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.73/12.71 kb and 6/15 files 2016-08-25 09:01:18,622 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.76/12.71 kb and 10/15 files 2016-08-25 09:01:18,622 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.63/12.71 kb and 7/15 files 2016-08-25 09:01:18,623 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.53/12.71 kb and 8/15 files 2016-08-25 09:01:18,623 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.66/12.71 kb and 11/15 files 2016-08-25 09:01:18,624 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.43/12.71 kb and 9/15 files 2016-08-25 09:01:18,624 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.84/12.71 kb and 12/15 files 2016-08-25 09:01:18,624 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.33/12.71 kb and 10/15 files 2016-08-25 09:01:18,624 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.74/12.71 kb and 13/15 files 2016-08-25 09:01:18,625 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.69/12.71 kb and 14/15 files 2016-08-25 09:01:18,625 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.22/12.71 kb and 11/15 files 2016-08-25 09:01:18,626 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.71/12.71 kb and 15/15 files 2016-08-25 09:01:18,626 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.89/12.71 kb and 12/15 files 2016-08-25 09:01:18,626 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:18,626 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.79/12.71 kb and 13/15 files 2016-08-25 09:01:18,627 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.69/12.71 kb and 14/15 files 2016-08-25 09:01:18,628 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.71/12.71 kb and 15/15 files 2016-08-25 09:01:18,628 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:18,630 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:18,631 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) is starting 2016-08-25 09:01:18,632 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:18,636 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55120 2016-08-25 09:01:18,636 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:18,636 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) started 2016-08-25 09:01:18,636 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:18,637 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:18,637 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-2 ====================================== 23.4. Browse the queue on new master ====================================== ====================================== 23.5. Found ====================================== ====================================== 24.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:18,663 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) is shutting down 2016-08-25 09:01:18,663 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:18,666 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:18,667 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:18,667 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:18,667 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250018 2016-08-25 09:01:18,667 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:18,668 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:18,668 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:18,668 | INFO | Time-limited test | Session: 0x156bec917250018 closed 2016-08-25 09:01:18,668 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:18,668 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55098 which had sessionid 0x156bec917250018 2016-08-25 09:01:18,783 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:18,784 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) uptime 1.850 seconds 2016-08-25 09:01:18,784 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) is shutdown Wait for master to start up... 2016-08-25 09:01:18,787 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:18,790 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3d31af5f 2016-08-25 09:01:18,790 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:18,791 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:18,791 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:18,791 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55122 2016-08-25 09:01:18,792 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55122 2016-08-25 09:01:18,793 | INFO | SyncThread:0 | Established session 0x156bec91725001b with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55122 2016-08-25 09:01:18,793 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725001b, negotiated timeout = 10000 2016-08-25 09:01:18,802 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:18,862 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:18,864 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:18,865 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:18,865 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:18,870 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:18,873 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:18,992 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55123 2016-08-25 09:01:18,997 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:18,997 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:18,999 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55123 2016-08-25 09:01:18,999 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:18,999 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55123 2016-08-25 09:01:18,999 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:19,016 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:19,113 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:19,115 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:19,115 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000037 2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000039.log 2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:19,119 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:19,120 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:19,120 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000037 2016-08-25 09:01:19,120 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:19,120 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000039.log 2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:19,123 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:19,128 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.66/12.71 kb and 1/15 files 2016-08-25 09:01:19,128 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.61/12.71 kb and 2/15 files 2016-08-25 09:01:19,129 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.51/12.71 kb and 3/15 files 2016-08-25 09:01:19,130 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.70/12.71 kb and 4/15 files 2016-08-25 09:01:19,130 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.59/12.71 kb and 5/15 files 2016-08-25 09:01:19,131 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.49/12.71 kb and 6/15 files 2016-08-25 09:01:19,132 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.39/12.71 kb and 7/15 files 2016-08-25 09:01:19,133 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.29/12.71 kb and 8/15 files 2016-08-25 09:01:19,133 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.19/12.71 kb and 9/15 files 2016-08-25 09:01:19,134 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.09/12.71 kb and 10/15 files 2016-08-25 09:01:19,135 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.99/12.71 kb and 11/15 files 2016-08-25 09:01:19,136 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.89/12.71 kb and 12/15 files 2016-08-25 09:01:19,136 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.90/12.71 kb and 13/15 files 2016-08-25 09:01:19,137 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/12.71 kb and 1/15 files 2016-08-25 09:01:19,137 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.80/12.71 kb and 14/15 files 2016-08-25 09:01:19,137 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.56/12.71 kb and 2/15 files 2016-08-25 09:01:19,137 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.71/12.71 kb and 15/15 files 2016-08-25 09:01:19,138 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:19,138 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.46/12.71 kb and 3/15 files 2016-08-25 09:01:19,138 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.36/12.71 kb and 4/15 files 2016-08-25 09:01:19,139 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.54/12.71 kb and 5/15 files 2016-08-25 09:01:19,140 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.44/12.71 kb and 6/15 files 2016-08-25 09:01:19,141 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.34/12.71 kb and 7/15 files 2016-08-25 09:01:19,142 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.24/12.71 kb and 8/15 files 2016-08-25 09:01:19,142 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.14/12.71 kb and 9/15 files 2016-08-25 09:01:19,143 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.15/12.71 kb and 10/15 files 2016-08-25 09:01:19,144 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.06/12.71 kb and 11/15 files 2016-08-25 09:01:19,145 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.01/12.71 kb and 12/15 files 2016-08-25 09:01:19,145 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.91/12.71 kb and 13/15 files 2016-08-25 09:01:19,146 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.81/12.71 kb and 14/15 files 2016-08-25 09:01:19,148 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.71/12.71 kb and 15/15 files 2016-08-25 09:01:19,148 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:19,152 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:19,152 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:19,154 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) is starting 2016-08-25 09:01:19,159 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55128 2016-08-25 09:01:19,159 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:19,159 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) started 2016-08-25 09:01:19,159 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:19,160 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:19,160 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-1 ====================================== 24.4. Browse the queue on new master ====================================== ====================================== 24.5. Found ====================================== ====================================== 25.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:19,179 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) is shutting down 2016-08-25 09:01:19,179 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:19,183 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:19,183 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:19,183 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:19,184 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:19,184 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:19,185 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:19,185 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250019 2016-08-25 09:01:19,186 | INFO | Time-limited test | Session: 0x156bec917250019 closed 2016-08-25 09:01:19,186 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:19,186 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55106 which had sessionid 0x156bec917250019 2016-08-25 09:01:19,290 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:19,291 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) uptime 1.661 seconds 2016-08-25 09:01:19,291 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) is shutdown Wait for master to start up... 2016-08-25 09:01:19,294 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:19,298 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@78f004c9 2016-08-25 09:01:19,298 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:19,299 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:19,299 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:19,300 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55130 2016-08-25 09:01:19,300 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55130 2016-08-25 09:01:19,301 | INFO | SyncThread:0 | Established session 0x156bec91725001c with negotiated timeout 10000 for client /127.0.0.1:55130 2016-08-25 09:01:19,301 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725001c, negotiated timeout = 10000 2016-08-25 09:01:19,314 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:19,407 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:19,409 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:19,507 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:19,507 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:19,513 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:19,522 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:19,631 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55131 2016-08-25 09:01:19,636 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:19,636 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:19,638 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55131 2016-08-25 09:01:19,638 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55131 2016-08-25 09:01:19,638 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:19,638 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:19,720 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:19,720 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:19,724 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:19,724 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:19,725 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:19,725 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:19,725 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:19,725 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000042.log 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000042.log 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000040 2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000040 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:19,736 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/13.66 kb and 1/16 files 2016-08-25 09:01:19,737 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/13.66 kb and 2/16 files 2016-08-25 09:01:19,737 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/13.66 kb and 3/16 files 2016-08-25 09:01:19,738 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.88/13.66 kb and 4/16 files 2016-08-25 09:01:19,739 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.79/13.66 kb and 5/16 files 2016-08-25 09:01:19,739 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.80/13.66 kb and 6/16 files 2016-08-25 09:01:19,740 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.70/13.66 kb and 7/16 files 2016-08-25 09:01:19,741 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.65/13.66 kb and 8/16 files 2016-08-25 09:01:19,742 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.55/13.66 kb and 9/16 files 2016-08-25 09:01:19,743 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.45/13.66 kb and 10/16 files 2016-08-25 09:01:19,744 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.34/13.66 kb and 11/16 files 2016-08-25 09:01:19,745 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.06/13.66 kb and 12/16 files 2016-08-25 09:01:19,746 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.96/13.66 kb and 13/16 files 2016-08-25 09:01:19,747 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.86/13.66 kb and 14/16 files 2016-08-25 09:01:19,748 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.76/13.66 kb and 15/16 files 2016-08-25 09:01:19,749 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.66/13.66 kb and 16/16 files 2016-08-25 09:01:19,749 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:19,753 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:19,753 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/13.66 kb and 1/16 files 2016-08-25 09:01:19,754 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/13.66 kb and 2/16 files 2016-08-25 09:01:19,754 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) is starting 2016-08-25 09:01:19,755 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.69/13.66 kb and 3/16 files 2016-08-25 09:01:19,755 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.59/13.66 kb and 4/16 files 2016-08-25 09:01:19,756 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.50/13.66 kb and 5/16 files 2016-08-25 09:01:19,757 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.40/13.66 kb and 6/16 files 2016-08-25 09:01:19,758 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.35/13.66 kb and 7/16 files 2016-08-25 09:01:19,758 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.53/13.66 kb and 8/16 files 2016-08-25 09:01:19,759 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55136 2016-08-25 09:01:19,759 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.24/13.66 kb and 9/16 files 2016-08-25 09:01:19,759 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:19,759 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) started 2016-08-25 09:01:19,759 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:19,759 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:19,760 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.14/13.66 kb and 10/16 files 2016-08-25 09:01:19,760 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:19,760 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.04/13.66 kb and 11/16 files 2016-08-25 09:01:19,761 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.94/13.66 kb and 12/16 files 2016-08-25 09:01:19,762 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.84/13.66 kb and 13/16 files Master started: node-3 ====================================== 25.4. Browse the queue on new master ====================================== 2016-08-25 09:01:19,762 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.74/13.66 kb and 14/16 files 2016-08-25 09:01:19,763 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.76/13.66 kb and 15/16 files 2016-08-25 09:01:19,764 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.66/13.66 kb and 16/16 files 2016-08-25 09:01:19,764 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:19,768 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 25.5. Found ====================================== ====================================== 26.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:19,781 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) is shutting down 2016-08-25 09:01:19,783 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:19,788 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:19,789 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:19,789 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:19,789 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:19,790 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:19,790 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:19,791 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001a 2016-08-25 09:01:19,792 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55114 which had sessionid 0x156bec91725001a 2016-08-25 09:01:19,792 | INFO | Time-limited test | Session: 0x156bec91725001a closed 2016-08-25 09:01:19,792 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:19,913 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:19,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) uptime 1.723 seconds 2016-08-25 09:01:19,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) is shutdown Wait for master to start up... 2016-08-25 09:01:19,916 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:19,919 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3512e6d9 2016-08-25 09:01:19,919 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:19,920 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:19,920 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:19,921 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55138 2016-08-25 09:01:19,921 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55138 2016-08-25 09:01:19,922 | INFO | SyncThread:0 | Established session 0x156bec91725001d with negotiated timeout 10000 for client /127.0.0.1:55138 2016-08-25 09:01:19,922 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725001d, negotiated timeout = 10000 2016-08-25 09:01:19,932 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,033 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:20,035 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,045 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:20,045 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,050 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:20,056 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:20,178 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55139 2016-08-25 09:01:20,185 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:20,186 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55139 2016-08-25 09:01:20,186 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:20,187 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:20,190 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55139 2016-08-25 09:01:20,191 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:20,207 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:20,212 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:20,212 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000042.log 2016-08-25 09:01:20,213 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:20,213 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:20,213 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:20,213 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000040 2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:20,215 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:20,215 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:20,215 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:20,225 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/13.66 kb and 1/16 files 2016-08-25 09:01:20,226 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/13.66 kb and 2/16 files 2016-08-25 09:01:20,227 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.98/13.66 kb and 3/16 files 2016-08-25 09:01:20,228 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.88/13.66 kb and 4/16 files 2016-08-25 09:01:20,231 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.78/13.66 kb and 5/16 files 2016-08-25 09:01:20,232 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.80/13.66 kb and 6/16 files 2016-08-25 09:01:20,232 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.70/13.66 kb and 7/16 files 2016-08-25 09:01:20,233 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.65/13.66 kb and 8/16 files 2016-08-25 09:01:20,234 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.55/13.66 kb and 9/16 files 2016-08-25 09:01:20,235 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.45/13.66 kb and 10/16 files 2016-08-25 09:01:20,236 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.35/13.66 kb and 11/16 files 2016-08-25 09:01:20,236 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.06/13.66 kb and 12/16 files 2016-08-25 09:01:20,237 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.96/13.66 kb and 13/16 files 2016-08-25 09:01:20,238 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.86/13.66 kb and 14/16 files 2016-08-25 09:01:20,238 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.76/13.66 kb and 15/16 files 2016-08-25 09:01:20,239 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.66/13.66 kb and 16/16 files 2016-08-25 09:01:20,239 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:20,242 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:20,243 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) is starting 2016-08-25 09:01:20,248 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55143 2016-08-25 09:01:20,248 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:20,248 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) started 2016-08-25 09:01:20,249 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:20,249 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:20,249 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-2 ====================================== 26.4. Browse the queue on new master ====================================== ====================================== 26.5. Found ====================================== ====================================== 27.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:20,267 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) is shutting down 2016-08-25 09:01:20,268 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:20,271 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,271 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:20,272 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001b 2016-08-25 09:01:20,272 | INFO | Time-limited test | Session: 0x156bec91725001b closed 2016-08-25 09:01:20,272 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:20,273 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55122 which had sessionid 0x156bec91725001b 2016-08-25 09:01:20,275 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:20,275 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:20,338 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:20,377 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:20,378 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) uptime 1.593 seconds 2016-08-25 09:01:20,378 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) is shutdown Wait for master to start up... 2016-08-25 09:01:20,381 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:20,383 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@72133137 2016-08-25 09:01:20,383 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:20,385 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:20,385 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:20,385 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55145 2016-08-25 09:01:20,386 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55145 2016-08-25 09:01:20,386 | INFO | SyncThread:0 | Established session 0x156bec91725001e with negotiated timeout 10000 for client /127.0.0.1:55145 2016-08-25 09:01:20,386 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725001e, negotiated timeout = 10000 2016-08-25 09:01:20,394 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,462 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:20,463 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,465 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,620 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:20,624 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:20,629 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:20,757 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55146 2016-08-25 09:01:20,762 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:20,762 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55146 2016-08-25 09:01:20,762 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:20,764 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:20,765 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55146 2016-08-25 09:01:20,765 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:20,866 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:20,867 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:20,873 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:20,873 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:20,874 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:20,874 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:20,874 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:20,874 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000045.log 2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000045.log 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000043 2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000043 2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:20,891 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/14.61 kb and 1/17 files 2016-08-25 09:01:20,891 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/14.61 kb and 1/17 files 2016-08-25 09:01:20,891 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/14.61 kb and 2/17 files 2016-08-25 09:01:20,892 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/14.61 kb and 2/17 files 2016-08-25 09:01:20,892 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/14.61 kb and 3/17 files 2016-08-25 09:01:20,892 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/14.61 kb and 3/17 files 2016-08-25 09:01:20,893 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/14.61 kb and 4/17 files 2016-08-25 09:01:20,893 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/14.61 kb and 4/17 files 2016-08-25 09:01:20,894 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/14.61 kb and 5/17 files 2016-08-25 09:01:20,894 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.78/14.61 kb and 5/17 files 2016-08-25 09:01:20,894 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.68/14.61 kb and 6/17 files 2016-08-25 09:01:20,894 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/14.61 kb and 6/17 files 2016-08-25 09:01:20,895 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.35/14.61 kb and 7/17 files 2016-08-25 09:01:20,895 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.70/14.61 kb and 7/17 files 2016-08-25 09:01:20,896 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.25/14.61 kb and 8/17 files 2016-08-25 09:01:20,896 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.60/14.61 kb and 8/17 files 2016-08-25 09:01:20,896 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.26/14.61 kb and 9/17 files 2016-08-25 09:01:20,896 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.50/14.61 kb and 9/17 files 2016-08-25 09:01:20,897 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.45/14.61 kb and 10/17 files 2016-08-25 09:01:20,897 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.39/14.61 kb and 10/17 files 2016-08-25 09:01:20,898 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.16/14.61 kb and 11/17 files 2016-08-25 09:01:20,898 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.35/14.61 kb and 11/17 files 2016-08-25 09:01:20,899 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.24/14.61 kb and 12/17 files 2016-08-25 09:01:20,899 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.10/14.61 kb and 12/17 files 2016-08-25 09:01:20,899 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.14/14.61 kb and 13/17 files 2016-08-25 09:01:20,899 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.00/14.61 kb and 13/17 files 2016-08-25 09:01:20,900 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.04/14.61 kb and 14/17 files 2016-08-25 09:01:20,900 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.90/14.61 kb and 14/17 files 2016-08-25 09:01:20,901 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.94/14.61 kb and 15/17 files 2016-08-25 09:01:20,901 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.80/14.61 kb and 15/17 files 2016-08-25 09:01:20,901 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.71/14.61 kb and 16/17 files 2016-08-25 09:01:20,901 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.84/14.61 kb and 16/17 files 2016-08-25 09:01:20,902 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.61/14.61 kb and 17/17 files 2016-08-25 09:01:20,902 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.61/14.61 kb and 17/17 files 2016-08-25 09:01:20,902 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:20,903 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:20,908 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:20,908 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:20,909 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) is starting 2016-08-25 09:01:20,914 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55151 2016-08-25 09:01:20,914 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:20,915 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) started 2016-08-25 09:01:20,915 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:20,915 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:20,915 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-3 ====================================== 27.4. Browse the queue on new master ====================================== ====================================== 27.5. Found ====================================== ====================================== 28.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:20,942 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) is shutting down 2016-08-25 09:01:20,944 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:20,949 | INFO | Starting broker node: node-1-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:01:20,949 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,950 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:20,950 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:20,950 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:20,950 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:20,953 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:20,954 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001d 2016-08-25 09:01:20,955 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55138 which had sessionid 0x156bec91725001d 2016-08-25 09:01:20,955 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:20,955 | INFO | Time-limited test | Session: 0x156bec91725001d closed 2016-08-25 09:01:20,955 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:21,073 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:21,074 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) uptime 1.160 seconds 2016-08-25 09:01:21,074 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) is shutdown Wait for master to start up... 2016-08-25 09:01:21,077 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:21,079 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4dca593e 2016-08-25 09:01:21,079 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:21,080 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:21,081 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:21,081 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55153 2016-08-25 09:01:21,081 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55153 2016-08-25 09:01:21,082 | INFO | SyncThread:0 | Established session 0x156bec91725001f with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55153 2016-08-25 09:01:21,082 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725001f, negotiated timeout = 10000 2016-08-25 09:01:21,090 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:21,190 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:21,190 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:21,191 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:21,192 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:21,196 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:21,199 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:21,331 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55154 2016-08-25 09:01:21,337 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:21,337 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:21,338 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55154 2016-08-25 09:01:21,338 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55154 2016-08-25 09:01:21,338 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:21,338 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:21,353 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:21,450 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:21,453 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000045.log 2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000043 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:21,457 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:21,458 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:21,459 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:21,459 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:21,459 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000043 2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000045.log 2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:21,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.95/14.61 kb and 1/17 files 2016-08-25 09:01:21,472 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.02/14.61 kb and 1/17 files 2016-08-25 09:01:21,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/14.61 kb and 2/17 files 2016-08-25 09:01:21,472 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.92/14.61 kb and 2/17 files 2016-08-25 09:01:21,473 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.10/14.61 kb and 3/17 files 2016-08-25 09:01:21,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.86/14.61 kb and 3/17 files 2016-08-25 09:01:21,473 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.00/14.61 kb and 4/17 files 2016-08-25 09:01:21,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.76/14.61 kb and 4/17 files 2016-08-25 09:01:21,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.66/14.61 kb and 5/17 files 2016-08-25 09:01:21,474 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.90/14.61 kb and 5/17 files 2016-08-25 09:01:21,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.56/14.61 kb and 6/17 files 2016-08-25 09:01:21,475 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.80/14.61 kb and 6/17 files 2016-08-25 09:01:21,476 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.70/14.61 kb and 7/17 files 2016-08-25 09:01:21,476 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.46/14.61 kb and 7/17 files 2016-08-25 09:01:21,477 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.60/14.61 kb and 8/17 files 2016-08-25 09:01:21,477 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.23/14.61 kb and 8/17 files 2016-08-25 09:01:21,477 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.13/14.61 kb and 9/17 files 2016-08-25 09:01:21,477 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.50/14.61 kb and 9/17 files 2016-08-25 09:01:21,478 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.40/14.61 kb and 10/17 files 2016-08-25 09:01:21,478 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.02/14.61 kb and 10/17 files 2016-08-25 09:01:21,479 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/14.61 kb and 11/17 files 2016-08-25 09:01:21,479 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.29/14.61 kb and 11/17 files 2016-08-25 09:01:21,479 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.82/14.61 kb and 12/17 files 2016-08-25 09:01:21,480 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.00/14.61 kb and 13/17 files 2016-08-25 09:01:21,480 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.06/14.61 kb and 12/17 files 2016-08-25 09:01:21,481 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.96/14.61 kb and 13/17 files 2016-08-25 09:01:21,481 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.90/14.61 kb and 14/17 files 2016-08-25 09:01:21,481 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.80/14.61 kb and 15/17 files 2016-08-25 09:01:21,481 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.91/14.61 kb and 14/17 files 2016-08-25 09:01:21,482 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.70/14.61 kb and 16/17 files 2016-08-25 09:01:21,482 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.81/14.61 kb and 15/17 files 2016-08-25 09:01:21,482 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.61/14.61 kb and 17/17 files 2016-08-25 09:01:21,483 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.71/14.61 kb and 16/17 files 2016-08-25 09:01:21,483 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:21,483 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.61/14.61 kb and 17/17 files 2016-08-25 09:01:21,483 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:21,486 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:21,487 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) is starting 2016-08-25 09:01:21,487 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:21,491 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55159 2016-08-25 09:01:21,491 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:21,492 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) started 2016-08-25 09:01:21,492 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:21,492 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:21,492 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-1 ====================================== 28.4. Browse the queue on new master ====================================== ====================================== 28.5. Found ====================================== ====================================== 29.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:21,511 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) is shutting down 2016-08-25 09:01:21,511 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:21,515 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:21,515 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:21,516 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001c 2016-08-25 09:01:21,516 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:21,516 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:21,517 | INFO | Time-limited test | Session: 0x156bec91725001c closed 2016-08-25 09:01:21,517 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:21,517 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55130 which had sessionid 0x156bec91725001c 2016-08-25 09:01:21,517 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:21,517 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:21,623 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:21,623 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) uptime 2.332 seconds 2016-08-25 09:01:21,624 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) is shutdown Wait for master to start up... 2016-08-25 09:01:21,626 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:21,629 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@27b68be7 2016-08-25 09:01:21,629 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:21,630 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:21,631 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:21,631 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55161 2016-08-25 09:01:21,631 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55161 2016-08-25 09:01:21,632 | INFO | SyncThread:0 | Established session 0x156bec917250020 with negotiated timeout 10000 for client /127.0.0.1:55161 2016-08-25 09:01:21,632 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250020, negotiated timeout = 10000 2016-08-25 09:01:21,640 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:21,844 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:21,845 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:21,847 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:21,849 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:21,854 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:21,860 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:21,966 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55162 2016-08-25 09:01:21,971 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:21,971 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:21,972 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55162 2016-08-25 09:01:21,972 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:21,974 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55162 2016-08-25 09:01:21,974 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:22,064 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:22,074 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:22,074 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:22,075 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000048.log 2016-08-25 09:01:22,075 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:22,075 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000046 2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:22,083 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000046 2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000048.log 2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:22,094 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/15.56 kb and 1/18 files 2016-08-25 09:01:22,095 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.09/15.56 kb and 2/18 files 2016-08-25 09:01:22,095 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.90/15.56 kb and 3/18 files 2016-08-25 09:01:22,096 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.80/15.56 kb and 4/18 files 2016-08-25 09:01:22,097 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.70/15.56 kb and 5/18 files 2016-08-25 09:01:22,097 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.60/15.56 kb and 6/18 files 2016-08-25 09:01:22,098 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.50/15.56 kb and 7/18 files 2016-08-25 09:01:22,099 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.40/15.56 kb and 8/18 files 2016-08-25 09:01:22,099 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.29/15.56 kb and 9/18 files 2016-08-25 09:01:22,100 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.31/15.56 kb and 10/18 files 2016-08-25 09:01:22,101 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.26/15.56 kb and 11/18 files 2016-08-25 09:01:22,102 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.16/15.56 kb and 12/18 files 2016-08-25 09:01:22,102 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.06/15.56 kb and 13/18 files 2016-08-25 09:01:22,103 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.96/15.56 kb and 14/18 files 2016-08-25 09:01:22,104 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.86/15.56 kb and 15/18 files 2016-08-25 09:01:22,105 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.76/15.56 kb and 16/18 files 2016-08-25 09:01:22,105 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/15.56 kb and 1/18 files 2016-08-25 09:01:22,105 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/15.56 kb and 2/18 files 2016-08-25 09:01:22,105 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.66/15.56 kb and 17/18 files 2016-08-25 09:01:22,106 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.75/15.56 kb and 3/18 files 2016-08-25 09:01:22,106 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.56/15.56 kb and 18/18 files 2016-08-25 09:01:22,106 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:22,107 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.65/15.56 kb and 4/18 files 2016-08-25 09:01:22,107 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.47/15.56 kb and 5/18 files 2016-08-25 09:01:22,108 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.37/15.56 kb and 6/18 files 2016-08-25 09:01:22,109 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.26/15.56 kb and 7/18 files 2016-08-25 09:01:22,109 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.16/15.56 kb and 8/18 files 2016-08-25 09:01:22,110 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.18/15.56 kb and 9/18 files 2016-08-25 09:01:22,110 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:22,111 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.08/15.56 kb and 10/18 files 2016-08-25 09:01:22,111 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) is starting 2016-08-25 09:01:22,112 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.97/15.56 kb and 11/18 files 2016-08-25 09:01:22,112 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.87/15.56 kb and 12/18 files 2016-08-25 09:01:22,113 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.77/15.56 kb and 13/18 files 2016-08-25 09:01:22,115 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.67/15.56 kb and 14/18 files 2016-08-25 09:01:22,116 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.57/15.56 kb and 15/18 files 2016-08-25 09:01:22,117 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.47/15.56 kb and 16/18 files 2016-08-25 09:01:22,118 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55167 2016-08-25 09:01:22,118 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.66/15.56 kb and 17/18 files 2016-08-25 09:01:22,118 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:22,118 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) started 2016-08-25 09:01:22,118 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:22,118 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:22,119 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.56/15.56 kb and 18/18 files 2016-08-25 09:01:22,119 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:22,119 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-2 ====================================== 29.4. Browse the queue on new master ====================================== 2016-08-25 09:01:22,124 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 29.5. Found ====================================== ====================================== 30.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:22,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) is shutting down 2016-08-25 09:01:22,142 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:22,144 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:22,145 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:22,145 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001e 2016-08-25 09:01:22,146 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:22,146 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:22,146 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55145 which had sessionid 0x156bec91725001e 2016-08-25 09:01:22,146 | INFO | Time-limited test | Session: 0x156bec91725001e closed 2016-08-25 09:01:22,146 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:22,151 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:22,151 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:22,262 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:22,263 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) uptime 1.884 seconds 2016-08-25 09:01:22,263 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) is shutdown Wait for master to start up... 2016-08-25 09:01:22,265 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:22,269 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4264a42e 2016-08-25 09:01:22,270 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:22,270 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:22,271 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:22,271 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55169 2016-08-25 09:01:22,271 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55169 2016-08-25 09:01:22,272 | INFO | SyncThread:0 | Established session 0x156bec917250021 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55169 2016-08-25 09:01:22,272 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250021, negotiated timeout = 10000 2016-08-25 09:01:22,282 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:22,362 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:22,366 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:22,368 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:22,369 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:22,375 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:22,384 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:22,493 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55170 2016-08-25 09:01:22,500 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:22,500 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:22,502 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55170 2016-08-25 09:01:22,502 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:22,503 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55170 2016-08-25 09:01:22,503 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:22,515 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:22,591 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:22,592 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000048.log 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000046 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000046 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000048.log 2016-08-25 09:01:22,605 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/15.56 kb and 1/18 files 2016-08-25 09:01:22,605 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/15.56 kb and 2/18 files 2016-08-25 09:01:22,606 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/15.56 kb and 3/18 files 2016-08-25 09:01:22,606 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.65/15.56 kb and 4/18 files 2016-08-25 09:01:22,607 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.55/15.56 kb and 5/18 files 2016-08-25 09:01:22,608 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.45/15.56 kb and 6/18 files 2016-08-25 09:01:22,609 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/15.56 kb and 7/18 files 2016-08-25 09:01:22,609 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.53/15.56 kb and 8/18 files 2016-08-25 09:01:22,610 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.43/15.56 kb and 9/18 files 2016-08-25 09:01:22,611 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/15.56 kb and 10/18 files 2016-08-25 09:01:22,612 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.34/15.56 kb and 11/18 files 2016-08-25 09:01:22,613 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.24/15.56 kb and 12/18 files 2016-08-25 09:01:22,614 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.14/15.56 kb and 13/18 files 2016-08-25 09:01:22,615 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.04/15.56 kb and 14/18 files 2016-08-25 09:01:22,616 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.94/15.56 kb and 15/18 files 2016-08-25 09:01:22,616 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/15.56 kb and 1/18 files 2016-08-25 09:01:22,617 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.84/15.56 kb and 16/18 files 2016-08-25 09:01:22,617 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/15.56 kb and 2/18 files 2016-08-25 09:01:22,618 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.66/15.56 kb and 17/18 files 2016-08-25 09:01:22,618 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.56/15.56 kb and 18/18 files 2016-08-25 09:01:22,618 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/15.56 kb and 3/18 files 2016-08-25 09:01:22,618 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:22,619 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/15.56 kb and 4/18 files 2016-08-25 09:01:22,620 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/15.56 kb and 5/18 files 2016-08-25 09:01:22,621 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.43/15.56 kb and 6/18 files 2016-08-25 09:01:22,621 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.33/15.56 kb and 7/18 files 2016-08-25 09:01:22,622 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.22/15.56 kb and 8/18 files 2016-08-25 09:01:22,622 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:22,623 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.12/15.56 kb and 9/18 files 2016-08-25 09:01:22,624 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.02/15.56 kb and 10/18 files 2016-08-25 09:01:22,624 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) is starting 2016-08-25 09:01:22,625 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/15.56 kb and 11/18 files 2016-08-25 09:01:22,625 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.82/15.56 kb and 12/18 files 2016-08-25 09:01:22,626 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.72/15.56 kb and 13/18 files 2016-08-25 09:01:22,627 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.62/15.56 kb and 14/18 files 2016-08-25 09:01:22,627 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.52/15.56 kb and 15/18 files 2016-08-25 09:01:22,628 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55175 2016-08-25 09:01:22,628 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.47/15.56 kb and 16/18 files 2016-08-25 09:01:22,628 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:22,628 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) started 2016-08-25 09:01:22,628 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:22,628 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:22,628 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.37/15.56 kb and 17/18 files 2016-08-25 09:01:22,628 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:22,629 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.56/15.56 kb and 18/18 files 2016-08-25 09:01:22,629 | INFO | hawtdispatch-DEFAULT-1 | Attached Master started: node-3 ====================================== 30.4. Browse the queue on new master ====================================== 2016-08-25 09:01:22,638 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d ====================================== 30.5. Found ====================================== ====================================== 31.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:22,650 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) is shutting down 2016-08-25 09:01:22,651 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:22,654 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:22,656 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:22,656 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001f 2016-08-25 09:01:22,657 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55153 which had sessionid 0x156bec91725001f 2016-08-25 09:01:22,658 | INFO | Time-limited test | Session: 0x156bec91725001f closed 2016-08-25 09:01:22,658 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:22,658 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:22,658 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:22,658 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:22,658 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:22,772 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:22,773 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) uptime 1.698 seconds 2016-08-25 09:01:22,773 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) is shutdown Wait for master to start up... 2016-08-25 09:01:22,780 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:22,782 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@21a086ed 2016-08-25 09:01:22,782 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:22,783 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:22,784 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:22,784 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55177 2016-08-25 09:01:22,784 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55177 2016-08-25 09:01:22,785 | INFO | SyncThread:0 | Established session 0x156bec917250022 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55177 2016-08-25 09:01:22,785 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250022, negotiated timeout = 10000 2016-08-25 09:01:22,799 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:22,980 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:22,982 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:23,017 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:23,017 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:23,023 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:23,030 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:23,158 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55178 2016-08-25 09:01:23,163 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:23,163 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:23,164 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55178 2016-08-25 09:01:23,165 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:23,167 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55178 2016-08-25 09:01:23,167 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:23,274 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:23,278 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:23,278 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000051.log 2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000052.sst 2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:23,280 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:23,280 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:23,280 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000049 2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:23,285 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:23,289 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000051.log 2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000052.sst 2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/16.51 kb and 1/20 files 2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000049 2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/16.51 kb and 2/20 files 2016-08-25 09:01:23,293 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/16.51 kb and 3/20 files 2016-08-25 09:01:23,293 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.98/16.51 kb and 4/20 files 2016-08-25 09:01:23,294 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.88/16.51 kb and 5/20 files 2016-08-25 09:01:23,294 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.79/16.51 kb and 6/20 files 2016-08-25 09:01:23,295 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.69/16.51 kb and 7/20 files 2016-08-25 09:01:23,296 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.58/16.51 kb and 8/20 files 2016-08-25 09:01:23,297 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.48/16.51 kb and 9/20 files 2016-08-25 09:01:23,297 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.38/16.51 kb and 10/20 files 2016-08-25 09:01:23,298 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.28/16.51 kb and 11/20 files 2016-08-25 09:01:23,298 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.18/16.51 kb and 12/20 files 2016-08-25 09:01:23,299 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.04/16.51 kb and 13/20 files 2016-08-25 09:01:23,300 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.99/16.51 kb and 14/20 files 2016-08-25 09:01:23,300 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.89/16.51 kb and 15/20 files 2016-08-25 09:01:23,301 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.79/16.51 kb and 16/20 files 2016-08-25 09:01:23,302 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.69/16.51 kb and 17/20 files 2016-08-25 09:01:23,302 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.18/16.51 kb and 1/20 files 2016-08-25 09:01:23,302 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.59/16.51 kb and 18/20 files 2016-08-25 09:01:23,303 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.08/16.51 kb and 2/20 files 2016-08-25 09:01:23,303 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.61/16.51 kb and 19/20 files 2016-08-25 09:01:23,303 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.51/16.51 kb and 20/20 files 2016-08-25 09:01:23,303 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.98/16.51 kb and 3/20 files 2016-08-25 09:01:23,304 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:23,304 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.98/16.51 kb and 4/20 files 2016-08-25 09:01:23,305 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.88/16.51 kb and 5/20 files 2016-08-25 09:01:23,306 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.79/16.51 kb and 6/20 files 2016-08-25 09:01:23,306 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.69/16.51 kb and 7/20 files 2016-08-25 09:01:23,307 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/16.51 kb and 8/20 files 2016-08-25 09:01:23,307 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.48/16.51 kb and 9/20 files 2016-08-25 09:01:23,308 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.38/16.51 kb and 10/20 files 2016-08-25 09:01:23,308 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:23,309 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.33/16.51 kb and 11/20 files 2016-08-25 09:01:23,309 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) is starting 2016-08-25 09:01:23,310 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.35/16.51 kb and 12/20 files 2016-08-25 09:01:23,310 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.25/16.51 kb and 13/20 files 2016-08-25 09:01:23,311 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.15/16.51 kb and 14/20 files 2016-08-25 09:01:23,312 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.05/16.51 kb and 15/20 files 2016-08-25 09:01:23,313 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.94/16.51 kb and 16/20 files 2016-08-25 09:01:23,313 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.84/16.51 kb and 17/20 files 2016-08-25 09:01:23,314 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.74/16.51 kb and 18/20 files 2016-08-25 09:01:23,314 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55183 2016-08-25 09:01:23,315 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:23,315 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) started 2016-08-25 09:01:23,315 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:23,315 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.64/16.51 kb and 19/20 files 2016-08-25 09:01:23,315 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:23,315 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:23,316 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.51/16.51 kb and 20/20 files 2016-08-25 09:01:23,316 | INFO | hawtdispatch-DEFAULT-1 | Attached Master started: node-1 ====================================== 31.4. Browse the queue on new master ====================================== 2016-08-25 09:01:23,322 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 31.5. Found ====================================== ====================================== 32.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:23,340 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) is shutting down 2016-08-25 09:01:23,341 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:23,346 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:23,349 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:23,350 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250020 2016-08-25 09:01:23,351 | INFO | Time-limited test | Session: 0x156bec917250020 closed 2016-08-25 09:01:23,351 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55161 which had sessionid 0x156bec917250020 2016-08-25 09:01:23,351 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:23,352 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:23,352 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:23,352 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:23,353 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:23,473 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:23,474 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) uptime 1.850 seconds 2016-08-25 09:01:23,474 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) is shutdown Wait for master to start up... 2016-08-25 09:01:23,476 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:23,479 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@54cc6586 2016-08-25 09:01:23,479 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:23,480 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:23,480 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:23,480 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55185 2016-08-25 09:01:23,481 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55185 2016-08-25 09:01:23,482 | INFO | SyncThread:0 | Established session 0x156bec917250023 with negotiated timeout 10000 for client /127.0.0.1:55185 2016-08-25 09:01:23,482 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250023, negotiated timeout = 10000 2016-08-25 09:01:23,492 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:23,588 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:23,590 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:23,595 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:23,595 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:23,599 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:23,603 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:23,728 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55186 2016-08-25 09:01:23,736 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:23,737 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55186 2016-08-25 09:01:23,737 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:23,738 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:23,739 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55186 2016-08-25 09:01:23,739 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:23,758 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:23,762 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000052.sst 2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000051.log 2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000049 2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:23,780 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/16.51 kb and 1/20 files 2016-08-25 09:01:23,781 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/16.51 kb and 2/20 files 2016-08-25 09:01:23,781 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/16.51 kb and 3/20 files 2016-08-25 09:01:23,782 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/16.51 kb and 4/20 files 2016-08-25 09:01:23,783 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/16.51 kb and 5/20 files 2016-08-25 09:01:23,784 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.79/16.51 kb and 6/20 files 2016-08-25 09:01:23,785 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.69/16.51 kb and 7/20 files 2016-08-25 09:01:23,785 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/16.51 kb and 8/20 files 2016-08-25 09:01:23,786 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.48/16.51 kb and 9/20 files 2016-08-25 09:01:23,787 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.38/16.51 kb and 10/20 files 2016-08-25 09:01:23,788 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.33/16.51 kb and 11/20 files 2016-08-25 09:01:23,788 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.23/16.51 kb and 12/20 files 2016-08-25 09:01:23,789 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.24/16.51 kb and 13/20 files 2016-08-25 09:01:23,790 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.14/16.51 kb and 14/20 files 2016-08-25 09:01:23,790 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.01/16.51 kb and 15/20 files 2016-08-25 09:01:23,791 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.91/16.51 kb and 16/20 files 2016-08-25 09:01:23,792 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.81/16.51 kb and 17/20 files 2016-08-25 09:01:23,792 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.71/16.51 kb and 18/20 files 2016-08-25 09:01:23,793 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.61/16.51 kb and 19/20 files 2016-08-25 09:01:23,794 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.51/16.51 kb and 20/20 files 2016-08-25 09:01:23,794 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:23,796 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:23,797 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) is starting 2016-08-25 09:01:23,801 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55190 2016-08-25 09:01:23,802 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:23,802 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) started 2016-08-25 09:01:23,802 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:23,803 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:23,803 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-2 ====================================== 32.4. Browse the queue on new master ====================================== ====================================== 32.5. Found ====================================== ====================================== 33.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:23,826 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) is shutting down 2016-08-25 09:01:23,826 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:23,829 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:23,829 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:23,830 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:23,830 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:23,831 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250021 2016-08-25 09:01:23,831 | INFO | Time-limited test | Session: 0x156bec917250021 closed 2016-08-25 09:01:23,831 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:23,832 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55169 which had sessionid 0x156bec917250021 2016-08-25 09:01:23,935 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:23,936 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) uptime 1.672 seconds 2016-08-25 09:01:23,936 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) is shutdown Wait for master to start up... 2016-08-25 09:01:23,938 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:23,941 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@50a4a6f0 2016-08-25 09:01:23,941 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:23,942 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:23,943 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:23,943 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55192 2016-08-25 09:01:23,944 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55192 2016-08-25 09:01:23,944 | INFO | SyncThread:0 | Established session 0x156bec917250024 with negotiated timeout 10000 for client /127.0.0.1:55192 2016-08-25 09:01:23,945 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250024, negotiated timeout = 10000 2016-08-25 09:01:23,957 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:24,007 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:24,007 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:24,008 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:24,010 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:24,159 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:24,165 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:24,170 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:24,287 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55193 2016-08-25 09:01:24,293 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:24,294 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55193 2016-08-25 09:01:24,294 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:24,295 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:24,296 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55193 2016-08-25 09:01:24,296 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:24,389 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:24,393 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:24,394 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:24,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:24,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000052 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000054.log 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000054.log 2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000052 2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:24,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:24,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/17.46 kb and 1/20 files 2016-08-25 09:01:24,409 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/17.46 kb and 2/20 files 2016-08-25 09:01:24,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/17.46 kb and 3/20 files 2016-08-25 09:01:24,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/17.46 kb and 4/20 files 2016-08-25 09:01:24,412 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/17.46 kb and 5/20 files 2016-08-25 09:01:24,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/17.46 kb and 6/20 files 2016-08-25 09:01:24,414 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.29/17.46 kb and 7/20 files 2016-08-25 09:01:24,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.19/17.46 kb and 8/20 files 2016-08-25 09:01:24,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.11/17.46 kb and 9/20 files 2016-08-25 09:01:24,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.29/17.46 kb and 10/20 files 2016-08-25 09:01:24,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.19/17.46 kb and 11/20 files 2016-08-25 09:01:24,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.09/17.46 kb and 12/20 files 2016-08-25 09:01:24,418 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.99/17.46 kb and 13/20 files 2016-08-25 09:01:24,418 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.89/17.46 kb and 14/20 files 2016-08-25 09:01:24,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.90/17.46 kb and 15/20 files 2016-08-25 09:01:24,419 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/17.46 kb and 1/20 files 2016-08-25 09:01:24,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.80/17.46 kb and 16/20 files 2016-08-25 09:01:24,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/17.46 kb and 2/20 files 2016-08-25 09:01:24,420 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.71/17.46 kb and 17/20 files 2016-08-25 09:01:24,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/17.46 kb and 3/20 files 2016-08-25 09:01:24,420 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.61/17.46 kb and 18/20 files 2016-08-25 09:01:24,421 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.51/17.46 kb and 19/20 files 2016-08-25 09:01:24,421 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/17.46 kb and 4/20 files 2016-08-25 09:01:24,422 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.46/17.46 kb and 20/20 files 2016-08-25 09:01:24,422 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/17.46 kb and 5/20 files 2016-08-25 09:01:24,422 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:24,422 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/17.46 kb and 6/20 files 2016-08-25 09:01:24,423 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.35/17.46 kb and 7/20 files 2016-08-25 09:01:24,423 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.25/17.46 kb and 8/20 files 2016-08-25 09:01:24,424 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.15/17.46 kb and 9/20 files 2016-08-25 09:01:24,425 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.05/17.46 kb and 10/20 files 2016-08-25 09:01:24,425 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.23/17.46 kb and 11/20 files 2016-08-25 09:01:24,426 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.15/17.46 kb and 12/20 files 2016-08-25 09:01:24,427 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:24,427 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.05/17.46 kb and 13/20 files 2016-08-25 09:01:24,428 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.95/17.46 kb and 14/20 files 2016-08-25 09:01:24,428 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) is starting 2016-08-25 09:01:24,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.85/17.46 kb and 15/20 files 2016-08-25 09:01:24,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.75/17.46 kb and 16/20 files 2016-08-25 09:01:24,430 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.76/17.46 kb and 17/20 files 2016-08-25 09:01:24,430 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.66/17.46 kb and 18/20 files 2016-08-25 09:01:24,431 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.56/17.46 kb and 19/20 files 2016-08-25 09:01:24,432 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.46/17.46 kb and 20/20 files 2016-08-25 09:01:24,432 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:24,433 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55198 2016-08-25 09:01:24,433 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:24,433 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) started 2016-08-25 09:01:24,433 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:24,433 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:24,433 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-1 ====================================== 33.4. Browse the queue on new master ====================================== 2016-08-25 09:01:24,438 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 33.5. Found ====================================== ====================================== 34.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:24,457 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) is shutting down 2016-08-25 09:01:24,459 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:24,463 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:01:24,463 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:24,465 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:24,466 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:24,466 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:24,466 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:24,471 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:24,473 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:24,473 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250023 2016-08-25 09:01:24,474 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55185 which had sessionid 0x156bec917250023 2016-08-25 09:01:24,475 | INFO | Time-limited test | Session: 0x156bec917250023 closed 2016-08-25 09:01:24,475 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:24,591 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:24,592 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) uptime 1.118 seconds 2016-08-25 09:01:24,592 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) is shutdown Wait for master to start up... 2016-08-25 09:01:24,594 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:24,597 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6442aec9 2016-08-25 09:01:24,597 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:24,598 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:24,598 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:24,598 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55200 2016-08-25 09:01:24,599 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55200 2016-08-25 09:01:24,600 | INFO | SyncThread:0 | Established session 0x156bec917250025 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55200 2016-08-25 09:01:24,600 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250025, negotiated timeout = 10000 2016-08-25 09:01:24,607 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:24,696 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:24,696 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:24,699 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:24,705 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:24,709 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:24,712 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:24,827 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55201 2016-08-25 09:01:24,834 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:24,834 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:24,835 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55201 2016-08-25 09:01:24,835 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:24,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55201 2016-08-25 09:01:24,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:24,944 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:24,944 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:24,947 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:24,947 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:24,947 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000052 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000052 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000054.log 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000054.log 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:24,960 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/17.46 kb and 1/20 files 2016-08-25 09:01:24,960 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/17.46 kb and 1/20 files 2016-08-25 09:01:24,960 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/17.46 kb and 2/20 files 2016-08-25 09:01:24,961 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/17.46 kb and 2/20 files 2016-08-25 09:01:24,961 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.83/17.46 kb and 3/20 files 2016-08-25 09:01:24,961 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/17.46 kb and 3/20 files 2016-08-25 09:01:24,962 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.73/17.46 kb and 4/20 files 2016-08-25 09:01:24,962 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.59/17.46 kb and 4/20 files 2016-08-25 09:01:24,962 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.63/17.46 kb and 5/20 files 2016-08-25 09:01:24,963 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/17.46 kb and 5/20 files 2016-08-25 09:01:24,963 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.53/17.46 kb and 6/20 files 2016-08-25 09:01:24,963 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.43/17.46 kb and 7/20 files 2016-08-25 09:01:24,963 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/17.46 kb and 6/20 files 2016-08-25 09:01:24,964 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/17.46 kb and 7/20 files 2016-08-25 09:01:24,964 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.38/17.46 kb and 8/20 files 2016-08-25 09:01:24,964 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.28/17.46 kb and 9/20 files 2016-08-25 09:01:24,964 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.20/17.46 kb and 8/20 files 2016-08-25 09:01:24,965 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.18/17.46 kb and 10/20 files 2016-08-25 09:01:24,965 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.14/17.46 kb and 9/20 files 2016-08-25 09:01:24,966 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.08/17.46 kb and 11/20 files 2016-08-25 09:01:24,966 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.04/17.46 kb and 10/20 files 2016-08-25 09:01:24,966 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.06/17.46 kb and 11/20 files 2016-08-25 09:01:24,966 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.98/17.46 kb and 12/20 files 2016-08-25 09:01:24,967 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/17.46 kb and 12/20 files 2016-08-25 09:01:24,967 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.87/17.46 kb and 13/20 files 2016-08-25 09:01:24,967 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/17.46 kb and 13/20 files 2016-08-25 09:01:24,967 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.77/17.46 kb and 14/20 files 2016-08-25 09:01:24,968 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.76/17.46 kb and 14/20 files 2016-08-25 09:01:24,968 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.67/17.46 kb and 15/20 files 2016-08-25 09:01:24,968 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.57/17.46 kb and 16/20 files 2016-08-25 09:01:24,968 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.67/17.46 kb and 15/20 files 2016-08-25 09:01:24,969 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.76/17.46 kb and 17/20 files 2016-08-25 09:01:24,969 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.66/17.46 kb and 18/20 files 2016-08-25 09:01:24,969 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.57/17.46 kb and 16/20 files 2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.56/17.46 kb and 19/20 files 2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.47/17.46 kb and 17/20 files 2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.46/17.46 kb and 20/20 files 2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.37/17.46 kb and 18/20 files 2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:24,971 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.27/17.46 kb and 19/20 files 2016-08-25 09:01:24,971 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.46/17.46 kb and 20/20 files 2016-08-25 09:01:24,971 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:24,972 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:24,973 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) is starting 2016-08-25 09:01:24,976 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:24,977 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55206 2016-08-25 09:01:24,978 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:24,978 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) started 2016-08-25 09:01:24,978 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:24,978 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:24,978 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-3 ====================================== 34.4. Browse the queue on new master ====================================== ====================================== 34.5. Found ====================================== ====================================== 35.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:25,002 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) is shutting down 2016-08-25 09:01:25,003 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:25,006 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:25,006 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:25,007 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250022 2016-08-25 09:01:25,008 | INFO | Time-limited test | Session: 0x156bec917250022 closed 2016-08-25 09:01:25,008 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55177 which had sessionid 0x156bec917250022 2016-08-25 09:01:25,009 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:25,012 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:25,012 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:25,013 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:25,013 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:25,112 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:25,113 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) uptime 2.339 seconds 2016-08-25 09:01:25,113 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) is shutdown Wait for master to start up... 2016-08-25 09:01:25,115 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:25,118 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7b4f3d95 2016-08-25 09:01:25,119 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:25,119 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:25,120 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:25,120 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55208 2016-08-25 09:01:25,120 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55208 2016-08-25 09:01:25,121 | INFO | SyncThread:0 | Established session 0x156bec917250026 with negotiated timeout 10000 for client /127.0.0.1:55208 2016-08-25 09:01:25,121 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250026, negotiated timeout = 10000 2016-08-25 09:01:25,133 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:25,374 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:25,377 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:25,379 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:25,379 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:25,383 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:25,386 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:25,508 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55209 2016-08-25 09:01:25,514 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:25,515 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:25,516 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55209 2016-08-25 09:01:25,516 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:25,517 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55209 2016-08-25 09:01:25,518 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:25,643 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:25,645 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:25,650 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000057.log 2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000057.log 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000055 2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000055 2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:25,666 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/18.40 kb and 1/21 files 2016-08-25 09:01:25,666 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/18.40 kb and 1/21 files 2016-08-25 09:01:25,667 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/18.40 kb and 2/21 files 2016-08-25 09:01:25,667 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/18.40 kb and 2/21 files 2016-08-25 09:01:25,668 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.98/18.40 kb and 3/21 files 2016-08-25 09:01:25,668 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/18.40 kb and 3/21 files 2016-08-25 09:01:25,668 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.60/18.40 kb and 4/21 files 2016-08-25 09:01:25,669 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.88/18.40 kb and 4/21 files 2016-08-25 09:01:25,669 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.50/18.40 kb and 5/21 files 2016-08-25 09:01:25,669 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.83/18.40 kb and 5/21 files 2016-08-25 09:01:25,669 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.40/18.40 kb and 6/21 files 2016-08-25 09:01:25,670 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.73/18.40 kb and 6/21 files 2016-08-25 09:01:25,670 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.58/18.40 kb and 7/21 files 2016-08-25 09:01:25,671 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.55/18.40 kb and 8/21 files 2016-08-25 09:01:25,671 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.62/18.40 kb and 7/21 files 2016-08-25 09:01:25,671 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.52/18.40 kb and 8/21 files 2016-08-25 09:01:25,671 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.45/18.40 kb and 9/21 files 2016-08-25 09:01:25,672 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.42/18.40 kb and 9/21 files 2016-08-25 09:01:25,672 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.35/18.40 kb and 10/21 files 2016-08-25 09:01:25,672 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.32/18.40 kb and 10/21 files 2016-08-25 09:01:25,672 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.25/18.40 kb and 11/21 files 2016-08-25 09:01:25,673 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.34/18.40 kb and 11/21 files 2016-08-25 09:01:25,673 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.20/18.40 kb and 12/21 files 2016-08-25 09:01:25,674 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.24/18.40 kb and 12/21 files 2016-08-25 09:01:25,674 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.10/18.40 kb and 13/21 files 2016-08-25 09:01:25,674 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.14/18.40 kb and 13/21 files 2016-08-25 09:01:25,675 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.99/18.40 kb and 14/21 files 2016-08-25 09:01:25,675 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.04/18.40 kb and 14/21 files 2016-08-25 09:01:25,675 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.89/18.40 kb and 15/21 files 2016-08-25 09:01:25,676 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.94/18.40 kb and 15/21 files 2016-08-25 09:01:25,676 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.79/18.40 kb and 16/21 files 2016-08-25 09:01:25,676 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.80/18.40 kb and 17/21 files 2016-08-25 09:01:25,676 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.84/18.40 kb and 16/21 files 2016-08-25 09:01:25,677 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.74/18.40 kb and 17/21 files 2016-08-25 09:01:25,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.70/18.40 kb and 18/21 files 2016-08-25 09:01:25,677 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.64/18.40 kb and 18/21 files 2016-08-25 09:01:25,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.60/18.40 kb and 19/21 files 2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.60/18.40 kb and 19/21 files 2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.50/18.40 kb and 20/21 files 2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.50/18.40 kb and 20/21 files 2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.40/18.40 kb and 21/21 files 2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:25,679 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.40/18.40 kb and 21/21 files 2016-08-25 09:01:25,679 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:25,683 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:25,685 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) is starting 2016-08-25 09:01:25,686 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:25,690 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55214 2016-08-25 09:01:25,691 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:25,691 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) started 2016-08-25 09:01:25,691 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:25,691 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:25,691 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-2 ====================================== 35.4. Browse the queue on new master ====================================== ====================================== 35.5. Found ====================================== ====================================== 36.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:25,712 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) is shutting down 2016-08-25 09:01:25,713 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:25,716 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:25,719 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:25,719 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:25,719 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:25,719 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:25,721 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:25,721 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250024 2016-08-25 09:01:25,722 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55192 which had sessionid 0x156bec917250024 2016-08-25 09:01:25,722 | INFO | Time-limited test | Session: 0x156bec917250024 closed 2016-08-25 09:01:25,722 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:25,844 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:25,845 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) uptime 1.909 seconds 2016-08-25 09:01:25,845 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) is shutdown Wait for master to start up... 2016-08-25 09:01:25,849 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:25,853 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1210a97 2016-08-25 09:01:25,853 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:25,854 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:25,855 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:25,855 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55216 2016-08-25 09:01:25,855 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55216 2016-08-25 09:01:25,859 | INFO | SyncThread:0 | Established session 0x156bec917250027 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55216 2016-08-25 09:01:25,859 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250027, negotiated timeout = 10000 2016-08-25 09:01:25,874 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:25,983 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:25,983 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:25,983 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:25,986 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:25,989 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:25,994 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:26,121 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55217 2016-08-25 09:01:26,126 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:26,126 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:26,127 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55217 2016-08-25 09:01:26,128 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:26,129 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55217 2016-08-25 09:01:26,129 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:26,141 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:26,230 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:26,231 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000055 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000057.log 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:26,234 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:26,234 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:26,234 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:26,234 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000057.log 2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:26,237 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000055 2016-08-25 09:01:26,237 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:26,237 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:26,237 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:26,243 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/18.40 kb and 1/21 files 2016-08-25 09:01:26,243 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/18.40 kb and 2/21 files 2016-08-25 09:01:26,244 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/18.40 kb and 3/21 files 2016-08-25 09:01:26,244 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/18.40 kb and 4/21 files 2016-08-25 09:01:26,245 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/18.40 kb and 5/21 files 2016-08-25 09:01:26,246 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.39/18.40 kb and 6/21 files 2016-08-25 09:01:26,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/18.40 kb and 1/21 files 2016-08-25 09:01:26,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/18.40 kb and 2/21 files 2016-08-25 09:01:26,246 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.41/18.40 kb and 7/21 files 2016-08-25 09:01:26,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/18.40 kb and 3/21 files 2016-08-25 09:01:26,247 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.31/18.40 kb and 8/21 files 2016-08-25 09:01:26,247 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.21/18.40 kb and 9/21 files 2016-08-25 09:01:26,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/18.40 kb and 4/21 files 2016-08-25 09:01:26,248 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.17/18.40 kb and 10/21 files 2016-08-25 09:01:26,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.90/18.40 kb and 5/21 files 2016-08-25 09:01:26,248 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.07/18.40 kb and 11/21 files 2016-08-25 09:01:26,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.80/18.40 kb and 6/21 files 2016-08-25 09:01:26,249 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.26/18.40 kb and 12/21 files 2016-08-25 09:01:26,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.70/18.40 kb and 7/21 files 2016-08-25 09:01:26,249 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.15/18.40 kb and 13/21 files 2016-08-25 09:01:26,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.60/18.40 kb and 8/21 files 2016-08-25 09:01:26,250 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.05/18.40 kb and 14/21 files 2016-08-25 09:01:26,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.50/18.40 kb and 9/21 files 2016-08-25 09:01:26,250 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.96/18.40 kb and 15/21 files 2016-08-25 09:01:26,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.40/18.40 kb and 10/21 files 2016-08-25 09:01:26,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.30/18.40 kb and 11/21 files 2016-08-25 09:01:26,251 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.86/18.40 kb and 16/21 files 2016-08-25 09:01:26,251 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.75/18.40 kb and 17/21 files 2016-08-25 09:01:26,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.20/18.40 kb and 12/21 files 2016-08-25 09:01:26,252 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.70/18.40 kb and 18/21 files 2016-08-25 09:01:26,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.10/18.40 kb and 13/21 files 2016-08-25 09:01:26,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.05/18.40 kb and 14/21 files 2016-08-25 09:01:26,252 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.60/18.40 kb and 19/21 files 2016-08-25 09:01:26,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.94/18.40 kb and 15/21 files 2016-08-25 09:01:26,253 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.50/18.40 kb and 20/21 files 2016-08-25 09:01:26,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.84/18.40 kb and 16/21 files 2016-08-25 09:01:26,253 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.40/18.40 kb and 21/21 files 2016-08-25 09:01:26,254 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:26,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.74/18.40 kb and 17/21 files 2016-08-25 09:01:26,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.71/18.40 kb and 18/21 files 2016-08-25 09:01:26,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.61/18.40 kb and 19/21 files 2016-08-25 09:01:26,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.51/18.40 kb and 20/21 files 2016-08-25 09:01:26,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.40/18.40 kb and 21/21 files 2016-08-25 09:01:26,256 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:26,257 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:26,258 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) is starting 2016-08-25 09:01:26,263 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:26,263 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55222 2016-08-25 09:01:26,263 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:26,264 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) started 2016-08-25 09:01:26,264 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:26,264 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:26,264 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-1 ====================================== 36.4. Browse the queue on new master ====================================== ====================================== 36.5. Found ====================================== ====================================== 37.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:26,283 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) is shutting down 2016-08-25 09:01:26,284 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:26,288 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:26,291 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:26,292 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250025 2016-08-25 09:01:26,293 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:26,293 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:26,293 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:26,293 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:26,294 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55200 which had sessionid 0x156bec917250025 2016-08-25 09:01:26,294 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:26,294 | INFO | Time-limited test | Session: 0x156bec917250025 closed 2016-08-25 09:01:26,398 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:26,398 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) uptime 1.806 seconds 2016-08-25 09:01:26,398 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) is shutdown Wait for master to start up... 2016-08-25 09:01:26,401 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:26,404 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@60bc56c1 2016-08-25 09:01:26,404 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:26,405 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:26,405 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:26,406 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55224 2016-08-25 09:01:26,406 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55224 2016-08-25 09:01:26,407 | INFO | SyncThread:0 | Established session 0x156bec917250028 with negotiated timeout 10000 for client /127.0.0.1:55224 2016-08-25 09:01:26,407 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250028, negotiated timeout = 10000 2016-08-25 09:01:26,426 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:26,681 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:26,684 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:26,687 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:26,687 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:26,691 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:26,694 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:26,825 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55225 2016-08-25 09:01:26,830 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:26,830 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:26,831 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55225 2016-08-25 09:01:26,831 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:26,833 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55225 2016-08-25 09:01:26,833 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:26,951 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:26,961 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:26,961 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:26,962 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:26,962 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000060.log 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000058 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:26,965 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:26,965 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000058 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000060.log 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:26,974 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/19.35 kb and 1/22 files 2016-08-25 09:01:26,974 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/19.35 kb and 2/22 files 2016-08-25 09:01:26,975 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/19.35 kb and 3/22 files 2016-08-25 09:01:26,976 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.59/19.35 kb and 4/22 files 2016-08-25 09:01:26,977 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.49/19.35 kb and 5/22 files 2016-08-25 09:01:26,978 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.68/19.35 kb and 6/22 files 2016-08-25 09:01:26,979 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.58/19.35 kb and 7/22 files 2016-08-25 09:01:26,979 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/19.35 kb and 8/22 files 2016-08-25 09:01:26,980 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.38/19.35 kb and 9/22 files 2016-08-25 09:01:26,981 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/19.35 kb and 10/22 files 2016-08-25 09:01:26,981 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.22/19.35 kb and 11/22 files 2016-08-25 09:01:26,982 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.12/19.35 kb and 12/22 files 2016-08-25 09:01:26,982 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.02/19.35 kb and 13/22 files 2016-08-25 09:01:26,983 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.92/19.35 kb and 14/22 files 2016-08-25 09:01:26,983 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.94/19.35 kb and 15/22 files 2016-08-25 09:01:26,984 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.84/19.35 kb and 16/22 files 2016-08-25 09:01:26,984 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.74/19.35 kb and 17/22 files 2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.64/19.35 kb and 18/22 files 2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/19.35 kb and 1/22 files 2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/19.35 kb and 2/22 files 2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.54/19.35 kb and 19/22 files 2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.69/19.35 kb and 3/22 files 2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.55/19.35 kb and 20/22 files 2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.45/19.35 kb and 21/22 files 2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/19.35 kb and 4/22 files 2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/19.35 kb and 5/22 files 2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.35/19.35 kb and 22/22 files 2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:26,987 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/19.35 kb and 6/22 files 2016-08-25 09:01:26,987 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/19.35 kb and 7/22 files 2016-08-25 09:01:26,988 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.31/19.35 kb and 8/22 files 2016-08-25 09:01:26,988 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.21/19.35 kb and 9/22 files 2016-08-25 09:01:26,989 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.10/19.35 kb and 10/22 files 2016-08-25 09:01:26,989 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.00/19.35 kb and 11/22 files 2016-08-25 09:01:26,989 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.90/19.35 kb and 12/22 files 2016-08-25 09:01:26,990 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.80/19.35 kb and 13/22 files 2016-08-25 09:01:26,990 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:26,991 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.70/19.35 kb and 14/22 files 2016-08-25 09:01:26,991 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) is starting 2016-08-25 09:01:26,991 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.60/19.35 kb and 15/22 files 2016-08-25 09:01:26,992 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.55/19.35 kb and 16/22 files 2016-08-25 09:01:26,992 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.45/19.35 kb and 17/22 files 2016-08-25 09:01:26,993 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.35/19.35 kb and 18/22 files 2016-08-25 09:01:26,993 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.53/19.35 kb and 19/22 files 2016-08-25 09:01:26,994 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.55/19.35 kb and 20/22 files 2016-08-25 09:01:26,995 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.45/19.35 kb and 21/22 files 2016-08-25 09:01:26,995 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.35/19.35 kb and 22/22 files 2016-08-25 09:01:26,995 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:26,996 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55230 2016-08-25 09:01:26,996 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:26,996 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) started 2016-08-25 09:01:26,996 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:26,996 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:26,996 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-3 ====================================== 37.4. Browse the queue on new master ====================================== 2016-08-25 09:01:27,000 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 37.5. Found ====================================== ====================================== 38.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:27,015 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) is shutting down 2016-08-25 09:01:27,016 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:27,019 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:27,020 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:27,020 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:27,021 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:27,021 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:27,021 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:27,021 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250026 2016-08-25 09:01:27,022 | INFO | Time-limited test | Session: 0x156bec917250026 closed 2016-08-25 09:01:27,022 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:27,022 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55208 which had sessionid 0x156bec917250026 2016-08-25 09:01:27,141 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:27,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) uptime 2.028 seconds 2016-08-25 09:01:27,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) is shutdown Wait for master to start up... 2016-08-25 09:01:27,145 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:27,147 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@2bf7a4f8 2016-08-25 09:01:27,148 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:27,149 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:27,149 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:27,150 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55232 2016-08-25 09:01:27,150 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55232 2016-08-25 09:01:27,151 | INFO | SyncThread:0 | Established session 0x156bec917250029 with negotiated timeout 10000 for client /127.0.0.1:55232 2016-08-25 09:01:27,151 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250029, negotiated timeout = 10000 2016-08-25 09:01:27,167 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:27,286 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:27,289 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:27,292 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:27,292 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:27,296 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:27,300 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:27,431 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55233 2016-08-25 09:01:27,437 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:27,437 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:27,438 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55233 2016-08-25 09:01:27,438 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:27,441 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55233 2016-08-25 09:01:27,441 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:27,452 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:27,557 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:27,557 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:27,558 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:27,558 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000060.log 2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000058 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:27,561 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:27,561 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:27,561 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:27,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000060.log 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000058 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:27,564 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:27,564 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:27,571 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/19.35 kb and 1/22 files 2016-08-25 09:01:27,571 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/19.35 kb and 2/22 files 2016-08-25 09:01:27,572 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/19.35 kb and 3/22 files 2016-08-25 09:01:27,573 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.64/19.35 kb and 4/22 files 2016-08-25 09:01:27,573 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.54/19.35 kb and 5/22 files 2016-08-25 09:01:27,574 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/19.35 kb and 6/22 files 2016-08-25 09:01:27,575 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.74/19.35 kb and 7/22 files 2016-08-25 09:01:27,575 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.64/19.35 kb and 8/22 files 2016-08-25 09:01:27,576 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.54/19.35 kb and 9/22 files 2016-08-25 09:01:27,577 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.44/19.35 kb and 10/22 files 2016-08-25 09:01:27,577 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.34/19.35 kb and 11/22 files 2016-08-25 09:01:27,578 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.24/19.35 kb and 12/22 files 2016-08-25 09:01:27,579 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.26/19.35 kb and 13/22 files 2016-08-25 09:01:27,579 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.16/19.35 kb and 14/22 files 2016-08-25 09:01:27,580 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.06/19.35 kb and 15/22 files 2016-08-25 09:01:27,581 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.95/19.35 kb and 16/22 files 2016-08-25 09:01:27,581 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.85/19.35 kb and 17/22 files 2016-08-25 09:01:27,582 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.75/19.35 kb and 18/22 files 2016-08-25 09:01:27,582 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.65/19.35 kb and 19/22 files 2016-08-25 09:01:27,583 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.55/19.35 kb and 20/22 files 2016-08-25 09:01:27,583 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.45/19.35 kb and 21/22 files 2016-08-25 09:01:27,584 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.35/19.35 kb and 22/22 files 2016-08-25 09:01:27,584 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:27,587 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:27,588 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) is starting 2016-08-25 09:01:27,589 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/19.35 kb and 1/22 files 2016-08-25 09:01:27,590 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/19.35 kb and 2/22 files 2016-08-25 09:01:27,591 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.69/19.35 kb and 3/22 files 2016-08-25 09:01:27,592 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.64/19.35 kb and 4/22 files 2016-08-25 09:01:27,593 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/19.35 kb and 5/22 files 2016-08-25 09:01:27,594 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55238 2016-08-25 09:01:27,594 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.44/19.35 kb and 6/22 files 2016-08-25 09:01:27,594 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:27,594 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) started 2016-08-25 09:01:27,594 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:27,594 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:27,594 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.34/19.35 kb and 7/22 files 2016-08-25 09:01:27,594 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:27,595 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.24/19.35 kb and 8/22 files 2016-08-25 09:01:27,596 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.14/19.35 kb and 9/22 files 2016-08-25 09:01:27,596 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.04/19.35 kb and 10/22 files Master started: node-2 ====================================== 38.4. Browse the queue on new master ====================================== 2016-08-25 09:01:27,597 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.94/19.35 kb and 11/22 files 2016-08-25 09:01:27,598 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.95/19.35 kb and 12/22 files 2016-08-25 09:01:27,598 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.85/19.35 kb and 13/22 files 2016-08-25 09:01:27,599 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/19.35 kb and 14/22 files 2016-08-25 09:01:27,601 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.94/19.35 kb and 15/22 files 2016-08-25 09:01:27,602 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.84/19.35 kb and 16/22 files 2016-08-25 09:01:27,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.74/19.35 kb and 17/22 files 2016-08-25 09:01:27,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.75/19.35 kb and 18/22 files 2016-08-25 09:01:27,606 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.65/19.35 kb and 19/22 files 2016-08-25 09:01:27,607 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.55/19.35 kb and 20/22 files 2016-08-25 09:01:27,609 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.45/19.35 kb and 21/22 files 2016-08-25 09:01:27,615 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.35/19.35 kb and 22/22 files 2016-08-25 09:01:27,615 | INFO | hawtdispatch-DEFAULT-1 | Attached ====================================== 38.5. Found ====================================== ====================================== 39.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:27,618 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) is shutting down 2016-08-25 09:01:27,619 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:27,623 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:27,625 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:27,625 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250027 2016-08-25 09:01:27,626 | INFO | Time-limited test | Session: 0x156bec917250027 closed 2016-08-25 09:01:27,626 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55216 which had sessionid 0x156bec917250027 2016-08-25 09:01:27,627 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:27,632 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:27,632 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:27,637 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:27,637 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:27,732 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:27,733 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) uptime 1.887 seconds 2016-08-25 09:01:27,733 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) is shutdown Wait for master to start up... 2016-08-25 09:01:27,735 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:27,738 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@289c0574 2016-08-25 09:01:27,739 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:27,740 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:27,740 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:27,741 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55240 2016-08-25 09:01:27,741 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55240 2016-08-25 09:01:27,742 | INFO | SyncThread:0 | Established session 0x156bec91725002a with negotiated timeout 10000 for client /127.0.0.1:55240 2016-08-25 09:01:27,742 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002a, negotiated timeout = 10000 2016-08-25 09:01:27,756 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:28,075 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:28,078 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:28,097 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:28,097 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:28,101 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:28,107 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:28,279 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55241 2016-08-25 09:01:28,285 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:28,285 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:28,287 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55241 2016-08-25 09:01:28,287 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:28,288 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55241 2016-08-25 09:01:28,289 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:28,441 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:28,455 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:28,455 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:28,455 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:28,455 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000061 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000063.log 2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:28,458 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000061 2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000063.log 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:28,461 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:28,461 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:28,461 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:28,461 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:28,474 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/20.30 kb and 1/23 files 2016-08-25 09:01:28,475 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/20.30 kb and 2/23 files 2016-08-25 09:01:28,476 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/20.30 kb and 3/23 files 2016-08-25 09:01:28,476 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/20.30 kb and 4/23 files 2016-08-25 09:01:28,477 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/20.30 kb and 5/23 files 2016-08-25 09:01:28,477 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/20.30 kb and 6/23 files 2016-08-25 09:01:28,478 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.41/20.30 kb and 7/23 files 2016-08-25 09:01:28,479 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.31/20.30 kb and 8/23 files 2016-08-25 09:01:28,479 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.21/20.30 kb and 9/23 files 2016-08-25 09:01:28,480 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.11/20.30 kb and 10/23 files 2016-08-25 09:01:28,480 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.01/20.30 kb and 11/23 files 2016-08-25 09:01:28,481 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/20.30 kb and 12/23 files 2016-08-25 09:01:28,482 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/20.30 kb and 13/23 files 2016-08-25 09:01:28,482 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.76/20.30 kb and 14/23 files 2016-08-25 09:01:28,483 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.66/20.30 kb and 15/23 files 2016-08-25 09:01:28,484 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.56/20.30 kb and 16/23 files 2016-08-25 09:01:28,484 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.46/20.30 kb and 17/23 files 2016-08-25 09:01:28,485 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.35/20.30 kb and 18/23 files 2016-08-25 09:01:28,485 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.25/20.30 kb and 19/23 files 2016-08-25 09:01:28,486 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.15/20.30 kb and 20/23 files 2016-08-25 09:01:28,487 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.22/20.30 kb and 21/23 files 2016-08-25 09:01:28,487 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/20.30 kb and 1/23 files 2016-08-25 09:01:28,487 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.40/20.30 kb and 22/23 files 2016-08-25 09:01:28,488 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/20.30 kb and 2/23 files 2016-08-25 09:01:28,488 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.30/20.30 kb and 23/23 files 2016-08-25 09:01:28,488 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:28,488 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.87/20.30 kb and 3/23 files 2016-08-25 09:01:28,489 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.77/20.30 kb and 4/23 files 2016-08-25 09:01:28,490 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.67/20.30 kb and 5/23 files 2016-08-25 09:01:28,490 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.56/20.30 kb and 6/23 files 2016-08-25 09:01:28,491 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.46/20.30 kb and 7/23 files 2016-08-25 09:01:28,491 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.36/20.30 kb and 8/23 files 2016-08-25 09:01:28,492 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.38/20.30 kb and 9/23 files 2016-08-25 09:01:28,493 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.56/20.30 kb and 10/23 files 2016-08-25 09:01:28,494 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.46/20.30 kb and 11/23 files 2016-08-25 09:01:28,494 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:28,494 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.36/20.30 kb and 12/23 files 2016-08-25 09:01:28,495 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) is starting 2016-08-25 09:01:28,495 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.30/20.30 kb and 13/23 files 2016-08-25 09:01:28,496 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.20/20.30 kb and 14/23 files 2016-08-25 09:01:28,496 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.10/20.30 kb and 15/23 files 2016-08-25 09:01:28,497 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.00/20.30 kb and 16/23 files 2016-08-25 09:01:28,497 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.90/20.30 kb and 17/23 files 2016-08-25 09:01:28,498 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.80/20.30 kb and 18/23 files 2016-08-25 09:01:28,499 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.71/20.30 kb and 19/23 files 2016-08-25 09:01:28,499 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55246 2016-08-25 09:01:28,499 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:28,499 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) started 2016-08-25 09:01:28,499 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:28,499 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.60/20.30 kb and 20/23 files 2016-08-25 09:01:28,500 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:28,500 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:28,500 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.50/20.30 kb and 21/23 files 2016-08-25 09:01:28,501 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.40/20.30 kb and 22/23 files Master started: node-1 ====================================== 39.4. Browse the queue on new master ====================================== 2016-08-25 09:01:28,501 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.30/20.30 kb and 23/23 files 2016-08-25 09:01:28,502 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:28,508 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 39.5. Found ====================================== ====================================== 40.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:28,517 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) is shutting down 2016-08-25 09:01:28,518 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:28,520 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:28,521 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:28,521 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250028 2016-08-25 09:01:28,522 | INFO | Time-limited test | Session: 0x156bec917250028 closed 2016-08-25 09:01:28,522 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:28,522 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55224 which had sessionid 0x156bec917250028 2016-08-25 09:01:28,523 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:28,523 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:28,524 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:28,524 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:28,639 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:28,640 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) uptime 2.240 seconds 2016-08-25 09:01:28,640 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) is shutdown Wait for master to start up... 2016-08-25 09:01:28,642 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:28,645 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1c7d7259 2016-08-25 09:01:28,645 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:28,646 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:28,647 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:28,647 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55248 2016-08-25 09:01:28,647 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55248 2016-08-25 09:01:28,648 | INFO | SyncThread:0 | Established session 0x156bec91725002b with negotiated timeout 10000 for client /127.0.0.1:55248 2016-08-25 09:01:28,648 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002b, negotiated timeout = 10000 2016-08-25 09:01:28,655 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:28,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:28,839 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:28,858 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:28,858 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:28,865 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:28,869 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:29,031 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55249 2016-08-25 09:01:29,036 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:29,037 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:29,038 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55249 2016-08-25 09:01:29,038 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:29,040 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55249 2016-08-25 09:01:29,040 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:29,053 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:29,056 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:29,057 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:29,057 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:29,057 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000063.log 2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000061 2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:29,074 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/20.30 kb and 1/23 files 2016-08-25 09:01:29,074 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/20.30 kb and 2/23 files 2016-08-25 09:01:29,075 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/20.30 kb and 3/23 files 2016-08-25 09:01:29,076 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.89/20.30 kb and 4/23 files 2016-08-25 09:01:29,077 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.96/20.30 kb and 5/23 files 2016-08-25 09:01:29,077 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.85/20.30 kb and 6/23 files 2016-08-25 09:01:29,078 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.75/20.30 kb and 7/23 files 2016-08-25 09:01:29,079 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.65/20.30 kb and 8/23 files 2016-08-25 09:01:29,079 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.55/20.30 kb and 9/23 files 2016-08-25 09:01:29,080 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.45/20.30 kb and 10/23 files 2016-08-25 09:01:29,081 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.35/20.30 kb and 11/23 files 2016-08-25 09:01:29,081 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.25/20.30 kb and 12/23 files 2016-08-25 09:01:29,082 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.15/20.30 kb and 13/23 files 2016-08-25 09:01:29,083 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.04/20.30 kb and 14/23 files 2016-08-25 09:01:29,083 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.94/20.30 kb and 15/23 files 2016-08-25 09:01:29,084 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.84/20.30 kb and 16/23 files 2016-08-25 09:01:29,085 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.86/20.30 kb and 17/23 files 2016-08-25 09:01:29,086 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.76/20.30 kb and 18/23 files 2016-08-25 09:01:29,086 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.66/20.30 kb and 19/23 files 2016-08-25 09:01:29,087 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.56/20.30 kb and 20/23 files 2016-08-25 09:01:29,088 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.45/20.30 kb and 21/23 files 2016-08-25 09:01:29,088 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.35/20.30 kb and 22/23 files 2016-08-25 09:01:29,089 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.30/20.30 kb and 23/23 files 2016-08-25 09:01:29,089 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:29,092 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:29,093 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) is starting 2016-08-25 09:01:29,097 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55253 2016-08-25 09:01:29,097 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:29,098 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) started 2016-08-25 09:01:29,098 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:29,098 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:29,098 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-3 ====================================== 40.4. Browse the queue on new master ====================================== ====================================== 40.5. Found ====================================== ====================================== 41.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:29,113 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) is shutting down 2016-08-25 09:01:29,113 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:29,115 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:29,116 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250029 2016-08-25 09:01:29,116 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:29,117 | INFO | Time-limited test | Session: 0x156bec917250029 closed 2016-08-25 09:01:29,117 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:29,117 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55232 which had sessionid 0x156bec917250029 2016-08-25 09:01:29,121 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:29,121 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:29,220 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:29,220 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) uptime 2.077 seconds 2016-08-25 09:01:29,220 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) is shutdown Wait for master to start up... 2016-08-25 09:01:29,222 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:29,225 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1afd5978 2016-08-25 09:01:29,225 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:29,225 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:29,226 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:29,226 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55255 2016-08-25 09:01:29,226 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55255 2016-08-25 09:01:29,227 | INFO | SyncThread:0 | Established session 0x156bec91725002c with negotiated timeout 10000 for client /127.0.0.1:55255 2016-08-25 09:01:29,227 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002c, negotiated timeout = 10000 2016-08-25 09:01:29,233 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:29,363 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:29,363 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:29,364 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:29,367 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:29,580 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:29,584 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:29,587 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:29,735 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55256 2016-08-25 09:01:29,743 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:29,743 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:29,743 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55256 2016-08-25 09:01:29,743 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:29,745 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55256 2016-08-25 09:01:29,745 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:29,908 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:29,915 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:29,915 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:29,916 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:29,916 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:29,916 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000064 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000066.log 2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:29,920 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:29,920 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:29,920 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000066.log 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000064 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:29,923 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:29,923 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:29,923 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:29,945 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/21.25 kb and 1/24 files 2016-08-25 09:01:29,945 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/21.25 kb and 1/24 files 2016-08-25 09:01:29,945 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/21.25 kb and 2/24 files 2016-08-25 09:01:29,945 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/21.25 kb and 2/24 files 2016-08-25 09:01:29,946 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/21.25 kb and 3/24 files 2016-08-25 09:01:29,946 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/21.25 kb and 3/24 files 2016-08-25 09:01:29,947 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/21.25 kb and 4/24 files 2016-08-25 09:01:29,947 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.60/21.25 kb and 4/24 files 2016-08-25 09:01:29,947 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/21.25 kb and 5/24 files 2016-08-25 09:01:29,947 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.50/21.25 kb and 5/24 files 2016-08-25 09:01:29,948 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.51/21.25 kb and 6/24 files 2016-08-25 09:01:29,948 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/21.25 kb and 6/24 files 2016-08-25 09:01:29,949 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/21.25 kb and 7/24 files 2016-08-25 09:01:29,949 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.41/21.25 kb and 7/24 files 2016-08-25 09:01:29,949 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.31/21.25 kb and 8/24 files 2016-08-25 09:01:29,949 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.24/21.25 kb and 8/24 files 2016-08-25 09:01:29,950 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.21/21.25 kb and 9/24 files 2016-08-25 09:01:29,950 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.43/21.25 kb and 9/24 files 2016-08-25 09:01:29,951 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.11/21.25 kb and 10/24 files 2016-08-25 09:01:29,951 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.33/21.25 kb and 10/24 files 2016-08-25 09:01:29,952 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.45/21.25 kb and 11/24 files 2016-08-25 09:01:29,952 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.23/21.25 kb and 11/24 files 2016-08-25 09:01:29,952 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.13/21.25 kb and 12/24 files 2016-08-25 09:01:29,952 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.35/21.25 kb and 12/24 files 2016-08-25 09:01:29,954 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.03/21.25 kb and 13/24 files 2016-08-25 09:01:29,954 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.25/21.25 kb and 13/24 files 2016-08-25 09:01:29,954 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.15/21.25 kb and 14/24 files 2016-08-25 09:01:29,954 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.93/21.25 kb and 14/24 files 2016-08-25 09:01:29,955 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.04/21.25 kb and 15/24 files 2016-08-25 09:01:29,955 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.83/21.25 kb and 15/24 files 2016-08-25 09:01:29,955 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.78/21.25 kb and 16/24 files 2016-08-25 09:01:29,955 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.94/21.25 kb and 16/24 files 2016-08-25 09:01:29,956 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.68/21.25 kb and 17/24 files 2016-08-25 09:01:29,956 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.96/21.25 kb and 17/24 files 2016-08-25 09:01:29,957 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.86/21.25 kb and 18/24 files 2016-08-25 09:01:29,957 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.58/21.25 kb and 18/24 files 2016-08-25 09:01:29,957 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.76/21.25 kb and 19/24 files 2016-08-25 09:01:29,957 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.47/21.25 kb and 19/24 files 2016-08-25 09:01:29,958 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.37/21.25 kb and 20/24 files 2016-08-25 09:01:29,958 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.66/21.25 kb and 20/24 files 2016-08-25 09:01:29,959 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.56/21.25 kb and 21/24 files 2016-08-25 09:01:29,959 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.27/21.25 kb and 21/24 files 2016-08-25 09:01:29,960 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.17/21.25 kb and 22/24 files 2016-08-25 09:01:29,960 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.46/21.25 kb and 22/24 files 2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.35/21.25 kb and 23/24 files 2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.35/21.25 kb and 23/24 files 2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.25/21.25 kb and 24/24 files 2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.25/21.25 kb and 24/24 files 2016-08-25 09:01:29,962 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:29,967 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:29,968 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) is starting 2016-08-25 09:01:29,970 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:29,972 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55261 2016-08-25 09:01:29,973 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:29,973 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) started 2016-08-25 09:01:29,973 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:29,973 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:29,973 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-1 ====================================== 41.4. Browse the queue on new master ====================================== ====================================== 41.5. Found ====================================== ====================================== 42.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:29,994 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) is shutting down 2016-08-25 09:01:29,995 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:29,997 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:29,998 | INFO | Starting broker node: node-2-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:01:29,998 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:29,998 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002b 2016-08-25 09:01:29,999 | INFO | Time-limited test | Session: 0x156bec91725002b closed 2016-08-25 09:01:29,999 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:29,999 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55248 which had sessionid 0x156bec91725002b 2016-08-25 09:01:30,001 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:30,005 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:30,005 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:30,005 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:30,005 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:30,120 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:30,121 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) uptime 1.481 seconds 2016-08-25 09:01:30,121 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) is shutdown Wait for master to start up... 2016-08-25 09:01:30,123 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:30,126 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5a8234a5 2016-08-25 09:01:30,126 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:30,127 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:30,128 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:30,128 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55263 2016-08-25 09:01:30,129 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55263 2016-08-25 09:01:30,129 | INFO | SyncThread:0 | Established session 0x156bec91725002d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55263 2016-08-25 09:01:30,130 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725002d, negotiated timeout = 10000 2016-08-25 09:01:30,138 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:30,328 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:30,331 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:30,333 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:30,333 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:30,337 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:30,340 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:30,525 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55264 2016-08-25 09:01:30,531 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:30,531 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:30,532 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55264 2016-08-25 09:01:30,532 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:30,534 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55264 2016-08-25 09:01:30,535 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:30,546 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:30,679 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:30,679 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:30,680 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:30,680 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:30,680 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000066.log 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000064 2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:30,696 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/21.25 kb and 1/24 files 2016-08-25 09:01:30,697 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:30,697 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.85/21.25 kb and 2/24 files 2016-08-25 09:01:30,697 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.75/21.25 kb and 3/24 files 2016-08-25 09:01:30,697 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.94/21.25 kb and 4/24 files 2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.84/21.25 kb and 5/24 files 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.74/21.25 kb and 6/24 files 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000064 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/21.25 kb and 7/24 files 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.53/21.25 kb and 8/24 files 2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000066.log 2016-08-25 09:01:30,701 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.43/21.25 kb and 9/24 files 2016-08-25 09:01:30,702 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/21.25 kb and 10/24 files 2016-08-25 09:01:30,702 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.23/21.25 kb and 11/24 files 2016-08-25 09:01:30,703 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.13/21.25 kb and 12/24 files 2016-08-25 09:01:30,704 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.25/21.25 kb and 13/24 files 2016-08-25 09:01:30,704 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.15/21.25 kb and 14/24 files 2016-08-25 09:01:30,705 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.05/21.25 kb and 15/24 files 2016-08-25 09:01:30,706 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.95/21.25 kb and 16/24 files 2016-08-25 09:01:30,707 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.85/21.25 kb and 17/24 files 2016-08-25 09:01:30,707 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.75/21.25 kb and 18/24 files 2016-08-25 09:01:30,708 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.65/21.25 kb and 19/24 files 2016-08-25 09:01:30,709 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.55/21.25 kb and 20/24 files 2016-08-25 09:01:30,709 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.45/21.25 kb and 21/24 files 2016-08-25 09:01:30,710 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.46/21.25 kb and 22/24 files 2016-08-25 09:01:30,711 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.36/21.25 kb and 23/24 files 2016-08-25 09:01:30,711 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.25/21.25 kb and 24/24 files 2016-08-25 09:01:30,711 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:30,713 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:30,714 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/21.25 kb and 1/24 files 2016-08-25 09:01:30,715 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) is starting 2016-08-25 09:01:30,715 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/21.25 kb and 2/24 files 2016-08-25 09:01:30,716 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/21.25 kb and 3/24 files 2016-08-25 09:01:30,716 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/21.25 kb and 4/24 files 2016-08-25 09:01:30,717 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/21.25 kb and 5/24 files 2016-08-25 09:01:30,717 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/21.25 kb and 6/24 files 2016-08-25 09:01:30,718 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/21.25 kb and 7/24 files 2016-08-25 09:01:30,719 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/21.25 kb and 8/24 files 2016-08-25 09:01:30,719 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.09/21.25 kb and 9/24 files 2016-08-25 09:01:30,720 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55269 2016-08-25 09:01:30,720 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.99/21.25 kb and 10/24 files 2016-08-25 09:01:30,720 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:30,720 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) started 2016-08-25 09:01:30,720 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:30,720 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:30,720 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:30,720 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.90/21.25 kb and 11/24 files 2016-08-25 09:01:30,721 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.80/21.25 kb and 12/24 files 2016-08-25 09:01:30,722 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/21.25 kb and 13/24 files Master started: node-2 ====================================== 42.4. Browse the queue on new master ====================================== 2016-08-25 09:01:30,723 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.64/21.25 kb and 14/24 files 2016-08-25 09:01:30,724 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.76/21.25 kb and 15/24 files 2016-08-25 09:01:30,724 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.78/21.25 kb and 16/24 files 2016-08-25 09:01:30,725 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.68/21.25 kb and 17/24 files 2016-08-25 09:01:30,726 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.58/21.25 kb and 18/24 files 2016-08-25 09:01:30,726 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.48/21.25 kb and 19/24 files 2016-08-25 09:01:30,727 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.38/21.25 kb and 20/24 files 2016-08-25 09:01:30,728 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.27/21.25 kb and 21/24 files 2016-08-25 09:01:30,728 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.17/21.25 kb and 22/24 files 2016-08-25 09:01:30,729 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.07/21.25 kb and 23/24 files 2016-08-25 09:01:30,730 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.25/21.25 kb and 24/24 files 2016-08-25 09:01:30,730 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:30,736 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 42.5. Found ====================================== ====================================== 43.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:30,739 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) is shutting down 2016-08-25 09:01:30,740 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:30,743 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:30,744 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:30,745 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002a 2016-08-25 09:01:30,745 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55240 which had sessionid 0x156bec91725002a 2016-08-25 09:01:30,745 | INFO | Time-limited test | Session: 0x156bec91725002a closed 2016-08-25 09:01:30,746 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:30,748 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:30,748 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:30,748 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:30,748 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:30,850 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:30,850 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) uptime 3.116 seconds 2016-08-25 09:01:30,851 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) is shutdown Wait for master to start up... 2016-08-25 09:01:30,853 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:30,856 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4b0f573b 2016-08-25 09:01:30,856 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:30,857 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:30,857 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55271 2016-08-25 09:01:30,857 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:30,858 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55271 2016-08-25 09:01:30,859 | INFO | SyncThread:0 | Established session 0x156bec91725002e with negotiated timeout 10000 for client /127.0.0.1:55271 2016-08-25 09:01:30,859 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002e, negotiated timeout = 10000 2016-08-25 09:01:30,868 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:31,230 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:31,232 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:31,253 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:31,253 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:31,257 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:31,262 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:31,450 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55272 2016-08-25 09:01:31,454 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:31,454 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:31,456 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55272 2016-08-25 09:01:31,456 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:31,458 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55272 2016-08-25 09:01:31,458 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:31,617 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:31,624 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:31,624 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000069.log 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000067 2016-08-25 09:01:31,628 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:31,628 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000067 2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000069.log 2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:31,636 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:31,636 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:31,636 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:31,649 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/22.20 kb and 1/25 files 2016-08-25 09:01:31,650 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/22.20 kb and 2/25 files 2016-08-25 09:01:31,650 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.81/22.20 kb and 3/25 files 2016-08-25 09:01:31,651 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.71/22.20 kb and 4/25 files 2016-08-25 09:01:31,651 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.61/22.20 kb and 5/25 files 2016-08-25 09:01:31,652 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.51/22.20 kb and 6/25 files 2016-08-25 09:01:31,652 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.95/22.20 kb and 1/25 files 2016-08-25 09:01:31,652 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.41/22.20 kb and 7/25 files 2016-08-25 09:01:31,653 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.85/22.20 kb and 2/25 files 2016-08-25 09:01:31,653 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.31/22.20 kb and 8/25 files 2016-08-25 09:01:31,654 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.75/22.20 kb and 3/25 files 2016-08-25 09:01:31,654 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.21/22.20 kb and 9/25 files 2016-08-25 09:01:31,654 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.92/22.20 kb and 4/25 files 2016-08-25 09:01:31,654 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.39/22.20 kb and 10/25 files 2016-08-25 09:01:31,655 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.82/22.20 kb and 5/25 files 2016-08-25 09:01:31,655 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.29/22.20 kb and 11/25 files 2016-08-25 09:01:31,655 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.72/22.20 kb and 6/25 files 2016-08-25 09:01:31,656 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.19/22.20 kb and 12/25 files 2016-08-25 09:01:31,656 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.62/22.20 kb and 7/25 files 2016-08-25 09:01:31,656 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.09/22.20 kb and 13/25 files 2016-08-25 09:01:31,657 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.52/22.20 kb and 8/25 files 2016-08-25 09:01:31,657 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.99/22.20 kb and 14/25 files 2016-08-25 09:01:31,657 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.42/22.20 kb and 9/25 files 2016-08-25 09:01:31,657 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.94/22.20 kb and 15/25 files 2016-08-25 09:01:31,658 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.84/22.20 kb and 16/25 files 2016-08-25 09:01:31,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.31/22.20 kb and 10/25 files 2016-08-25 09:01:31,658 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.73/22.20 kb and 17/25 files 2016-08-25 09:01:31,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.50/22.20 kb and 11/25 files 2016-08-25 09:01:31,659 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.40/22.20 kb and 12/25 files 2016-08-25 09:01:31,659 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.64/22.20 kb and 18/25 files 2016-08-25 09:01:31,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.30/22.20 kb and 13/25 files 2016-08-25 09:01:31,660 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.54/22.20 kb and 19/25 files 2016-08-25 09:01:31,660 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.44/22.20 kb and 20/25 files 2016-08-25 09:01:31,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.20/22.20 kb and 14/25 files 2016-08-25 09:01:31,661 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.10/22.20 kb and 15/25 files 2016-08-25 09:01:31,661 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.33/22.20 kb and 21/25 files 2016-08-25 09:01:31,661 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.99/22.20 kb and 16/25 files 2016-08-25 09:01:31,661 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.23/22.20 kb and 22/25 files 2016-08-25 09:01:31,662 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.89/22.20 kb and 17/25 files 2016-08-25 09:01:31,662 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.13/22.20 kb and 23/25 files 2016-08-25 09:01:31,662 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.79/22.20 kb and 18/25 files 2016-08-25 09:01:31,662 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.03/22.20 kb and 24/25 files 2016-08-25 09:01:31,663 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.20/22.20 kb and 25/25 files 2016-08-25 09:01:31,663 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.69/22.20 kb and 19/25 files 2016-08-25 09:01:31,663 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:31,663 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.70/22.20 kb and 20/25 files 2016-08-25 09:01:31,664 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.60/22.20 kb and 21/25 files 2016-08-25 09:01:31,665 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.50/22.20 kb and 22/25 files 2016-08-25 09:01:31,665 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.41/22.20 kb and 23/25 files 2016-08-25 09:01:31,666 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.30/22.20 kb and 24/25 files 2016-08-25 09:01:31,666 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 22.20/22.20 kb and 25/25 files 2016-08-25 09:01:31,666 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:31,670 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:31,670 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:31,672 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) is starting 2016-08-25 09:01:31,676 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55277 2016-08-25 09:01:31,676 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:31,676 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) started 2016-08-25 09:01:31,676 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:31,676 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:31,676 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-3 ====================================== 43.4. Browse the queue on new master ====================================== ====================================== 43.5. Found ====================================== ====================================== 44.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:31,693 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) is shutting down 2016-08-25 09:01:31,694 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:31,696 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:31,696 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002c 2016-08-25 09:01:31,696 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:31,697 | INFO | Time-limited test | Session: 0x156bec91725002c closed 2016-08-25 09:01:31,697 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:31,697 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55255 which had sessionid 0x156bec91725002c 2016-08-25 09:01:31,701 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:31,701 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:31,701 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:31,701 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:31,811 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:31,812 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) uptime 2.591 seconds 2016-08-25 09:01:31,812 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) is shutdown Wait for master to start up... 2016-08-25 09:01:31,815 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:31,817 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4b8e51d2 2016-08-25 09:01:31,817 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:31,818 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:31,818 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:31,818 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55279 2016-08-25 09:01:31,819 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55279 2016-08-25 09:01:31,819 | INFO | SyncThread:0 | Established session 0x156bec91725002f with negotiated timeout 10000 for client /127.0.0.1:55279 2016-08-25 09:01:31,819 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002f, negotiated timeout = 10000 2016-08-25 09:01:31,827 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:32,015 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:32,017 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:32,020 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:32,020 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:32,025 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:32,029 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:32,217 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55280 2016-08-25 09:01:32,222 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:32,222 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:32,224 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55280 2016-08-25 09:01:32,224 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:32,226 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55280 2016-08-25 09:01:32,226 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:32,239 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:32,392 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:32,393 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:32,393 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:32,393 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:32,393 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000069.log 2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000067 2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:32,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:32,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:32,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:32,397 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:32,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:32,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:32,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000069.log 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000067 2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:32,401 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:32,401 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:32,401 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:32,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/22.20 kb and 1/25 files 2016-08-25 09:01:32,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.92/22.20 kb and 2/25 files 2016-08-25 09:01:32,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.10/22.20 kb and 3/25 files 2016-08-25 09:01:32,412 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.00/22.20 kb and 4/25 files 2016-08-25 09:01:32,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.90/22.20 kb and 5/25 files 2016-08-25 09:01:32,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.79/22.20 kb and 6/25 files 2016-08-25 09:01:32,414 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.69/22.20 kb and 7/25 files 2016-08-25 09:01:32,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.59/22.20 kb and 8/25 files 2016-08-25 09:01:32,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.76/22.20 kb and 9/25 files 2016-08-25 09:01:32,416 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/22.20 kb and 1/25 files 2016-08-25 09:01:32,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.66/22.20 kb and 10/25 files 2016-08-25 09:01:32,417 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/22.20 kb and 2/25 files 2016-08-25 09:01:32,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.56/22.20 kb and 11/25 files 2016-08-25 09:01:32,418 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/22.20 kb and 3/25 files 2016-08-25 09:01:32,418 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.46/22.20 kb and 12/25 files 2016-08-25 09:01:32,418 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/22.20 kb and 4/25 files 2016-08-25 09:01:32,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.41/22.20 kb and 13/25 files 2016-08-25 09:01:32,419 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/22.20 kb and 5/25 files 2016-08-25 09:01:32,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.31/22.20 kb and 14/25 files 2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/22.20 kb and 6/25 files 2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.21/22.20 kb and 15/25 files 2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/22.20 kb and 7/25 files 2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.11/22.20 kb and 16/25 files 2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/22.20 kb and 8/25 files 2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.01/22.20 kb and 17/25 files 2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.09/22.20 kb and 9/25 files 2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.91/22.20 kb and 18/25 files 2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.99/22.20 kb and 10/25 files 2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.81/22.20 kb and 19/25 files 2016-08-25 09:01:32,422 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.71/22.20 kb and 20/25 files 2016-08-25 09:01:32,422 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.94/22.20 kb and 11/25 files 2016-08-25 09:01:32,422 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.61/22.20 kb and 21/25 files 2016-08-25 09:01:32,422 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.12/22.20 kb and 12/25 files 2016-08-25 09:01:32,423 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.51/22.20 kb and 22/25 files 2016-08-25 09:01:32,423 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.02/22.20 kb and 13/25 files 2016-08-25 09:01:32,423 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.92/22.20 kb and 14/25 files 2016-08-25 09:01:32,423 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.41/22.20 kb and 23/25 files 2016-08-25 09:01:32,424 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.30/22.20 kb and 24/25 files 2016-08-25 09:01:32,424 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.82/22.20 kb and 15/25 files 2016-08-25 09:01:32,425 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.72/22.20 kb and 16/25 files 2016-08-25 09:01:32,425 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.20/22.20 kb and 25/25 files 2016-08-25 09:01:32,425 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:32,425 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.62/22.20 kb and 17/25 files 2016-08-25 09:01:32,426 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.52/22.20 kb and 18/25 files 2016-08-25 09:01:32,426 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.42/22.20 kb and 19/25 files 2016-08-25 09:01:32,427 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.32/22.20 kb and 20/25 files 2016-08-25 09:01:32,427 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:32,428 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.49/22.20 kb and 21/25 files 2016-08-25 09:01:32,428 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) is starting 2016-08-25 09:01:32,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.51/22.20 kb and 22/25 files 2016-08-25 09:01:32,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.41/22.20 kb and 23/25 files 2016-08-25 09:01:32,430 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.31/22.20 kb and 24/25 files 2016-08-25 09:01:32,431 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.20/22.20 kb and 25/25 files 2016-08-25 09:01:32,431 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:32,433 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55285 2016-08-25 09:01:32,433 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:32,433 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) started 2016-08-25 09:01:32,433 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:32,433 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:32,433 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-1 ====================================== 44.4. Browse the queue on new master ====================================== 2016-08-25 09:01:32,437 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 44.5. Found ====================================== ====================================== 45.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:32,448 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) is shutting down 2016-08-25 09:01:32,448 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:32,451 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:32,452 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:32,452 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:32,452 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:32,452 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002d 2016-08-25 09:01:32,452 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:32,453 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:32,453 | INFO | Time-limited test | Session: 0x156bec91725002d closed 2016-08-25 09:01:32,453 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55263 which had sessionid 0x156bec91725002d 2016-08-25 09:01:32,453 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:32,558 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:32,559 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) uptime 2.437 seconds 2016-08-25 09:01:32,559 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) is shutdown Wait for master to start up... 2016-08-25 09:01:32,562 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:32,564 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1886ff2 2016-08-25 09:01:32,564 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:32,565 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:32,565 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:32,565 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55287 2016-08-25 09:01:32,565 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55287 2016-08-25 09:01:32,566 | INFO | SyncThread:0 | Established session 0x156bec917250030 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55287 2016-08-25 09:01:32,566 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250030, negotiated timeout = 10000 2016-08-25 09:01:32,572 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:32,981 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:32,983 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:32,985 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:32,986 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:32,990 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:32,997 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:33,177 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55288 2016-08-25 09:01:33,181 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:33,181 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:33,183 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55288 2016-08-25 09:01:33,183 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:33,185 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55288 2016-08-25 09:01:33,185 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:33,365 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:33,381 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:33,381 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:33,382 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:33,382 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:33,382 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:33,382 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000070 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000072.log 2016-08-25 09:01:33,386 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000072.log 2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000070 2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:33,399 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/23.15 kb and 1/26 files 2016-08-25 09:01:33,400 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.85/23.15 kb and 2/26 files 2016-08-25 09:01:33,400 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.86/23.15 kb and 3/26 files 2016-08-25 09:01:33,401 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.76/23.15 kb and 4/26 files 2016-08-25 09:01:33,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.66/23.15 kb and 5/26 files 2016-08-25 09:01:33,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.56/23.15 kb and 6/26 files 2016-08-25 09:01:33,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.46/23.15 kb and 7/26 files 2016-08-25 09:01:33,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.36/23.15 kb and 8/26 files 2016-08-25 09:01:33,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.26/23.15 kb and 9/26 files 2016-08-25 09:01:33,405 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/23.15 kb and 1/26 files 2016-08-25 09:01:33,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.16/23.15 kb and 10/26 files 2016-08-25 09:01:33,405 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/23.15 kb and 2/26 files 2016-08-25 09:01:33,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.06/23.15 kb and 11/26 files 2016-08-25 09:01:33,406 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/23.15 kb and 3/26 files 2016-08-25 09:01:33,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/23.15 kb and 12/26 files 2016-08-25 09:01:33,406 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/23.15 kb and 4/26 files 2016-08-25 09:01:33,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.85/23.15 kb and 13/26 files 2016-08-25 09:01:33,407 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/23.15 kb and 5/26 files 2016-08-25 09:01:33,407 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.75/23.15 kb and 14/26 files 2016-08-25 09:01:33,407 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.65/23.15 kb and 15/26 files 2016-08-25 09:01:33,407 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/23.15 kb and 6/26 files 2016-08-25 09:01:33,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.55/23.15 kb and 16/26 files 2016-08-25 09:01:33,408 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/23.15 kb and 7/26 files 2016-08-25 09:01:33,408 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/23.15 kb and 8/26 files 2016-08-25 09:01:33,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.45/23.15 kb and 17/26 files 2016-08-25 09:01:33,409 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.35/23.15 kb and 18/26 files 2016-08-25 09:01:33,409 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/23.15 kb and 9/26 files 2016-08-25 09:01:33,410 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/23.15 kb and 10/26 files 2016-08-25 09:01:33,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.25/23.15 kb and 19/26 files 2016-08-25 09:01:33,410 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.89/23.15 kb and 11/26 files 2016-08-25 09:01:33,411 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.79/23.15 kb and 12/26 files 2016-08-25 09:01:33,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.15/23.15 kb and 20/26 files 2016-08-25 09:01:33,411 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.69/23.15 kb and 13/26 files 2016-08-25 09:01:33,412 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.59/23.15 kb and 14/26 files 2016-08-25 09:01:33,412 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.49/23.15 kb and 15/26 files 2016-08-25 09:01:33,412 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.05/23.15 kb and 21/26 files 2016-08-25 09:01:33,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.27/23.15 kb and 22/26 files 2016-08-25 09:01:33,413 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.44/23.15 kb and 16/26 files 2016-08-25 09:01:33,414 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.63/23.15 kb and 17/26 files 2016-08-25 09:01:33,414 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.17/23.15 kb and 23/26 files 2016-08-25 09:01:33,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.07/23.15 kb and 24/26 files 2016-08-25 09:01:33,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.97/23.15 kb and 25/26 files 2016-08-25 09:01:33,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.15/23.15 kb and 26/26 files 2016-08-25 09:01:33,416 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:33,417 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.52/23.15 kb and 18/26 files 2016-08-25 09:01:33,417 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.54/23.15 kb and 19/26 files 2016-08-25 09:01:33,417 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.44/23.15 kb and 20/26 files 2016-08-25 09:01:33,418 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.33/23.15 kb and 21/26 files 2016-08-25 09:01:33,418 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.23/23.15 kb and 22/26 files 2016-08-25 09:01:33,419 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.46/23.15 kb and 23/26 files 2016-08-25 09:01:33,419 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.36/23.15 kb and 24/26 files 2016-08-25 09:01:33,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.25/23.15 kb and 25/26 files 2016-08-25 09:01:33,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.15/23.15 kb and 26/26 files 2016-08-25 09:01:33,420 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:33,421 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:33,422 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) is starting 2016-08-25 09:01:33,426 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:33,426 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55293 2016-08-25 09:01:33,426 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:33,427 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) started 2016-08-25 09:01:33,427 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:33,427 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb 2016-08-25 09:01:33,427 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb Master started: node-2 ====================================== 45.4. Browse the queue on new master ====================================== ====================================== 45.5. Found ====================================== ====================================== 46.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:33,445 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) is shutting down 2016-08-25 09:01:33,446 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:33,448 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:33,448 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002e 2016-08-25 09:01:33,448 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:33,449 | INFO | Time-limited test | Session: 0x156bec91725002e closed 2016-08-25 09:01:33,449 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:33,449 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55271 which had sessionid 0x156bec91725002e 2016-08-25 09:01:33,451 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:33,451 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:33,451 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:33,451 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:33,565 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:33,566 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) uptime 2.714 seconds 2016-08-25 09:01:33,566 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) is shutdown Wait for master to start up... 2016-08-25 09:01:33,568 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:33,570 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7f8574aa 2016-08-25 09:01:33,570 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:33,570 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:33,571 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:33,571 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55295 2016-08-25 09:01:33,571 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55295 2016-08-25 09:01:33,572 | INFO | SyncThread:0 | Established session 0x156bec917250031 with negotiated timeout 10000 for client /127.0.0.1:55295 2016-08-25 09:01:33,572 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250031, negotiated timeout = 10000 2016-08-25 09:01:33,580 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:33,824 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:33,826 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:33,829 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:33,829 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:33,834 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:33,837 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:34,046 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55296 2016-08-25 09:01:34,050 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:34,051 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:34,052 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55296 2016-08-25 09:01:34,052 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:34,054 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55296 2016-08-25 09:01:34,054 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:34,067 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:34,237 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000072.log 2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000070 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:34,241 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000072.log 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000070 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:34,255 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/23.15 kb and 1/26 files 2016-08-25 09:01:34,256 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/23.15 kb and 2/26 files 2016-08-25 09:01:34,257 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.75/23.15 kb and 3/26 files 2016-08-25 09:01:34,258 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.64/23.15 kb and 4/26 files 2016-08-25 09:01:34,259 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.54/23.15 kb and 5/26 files 2016-08-25 09:01:34,259 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.44/23.15 kb and 6/26 files 2016-08-25 09:01:34,260 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.35/23.15 kb and 7/26 files 2016-08-25 09:01:34,261 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.25/23.15 kb and 8/26 files 2016-08-25 09:01:34,261 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.15/23.15 kb and 9/26 files 2016-08-25 09:01:34,262 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.16/23.15 kb and 10/26 files 2016-08-25 09:01:34,263 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.06/23.15 kb and 11/26 files 2016-08-25 09:01:34,263 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/23.15 kb and 12/26 files 2016-08-25 09:01:34,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/23.15 kb and 13/26 files 2016-08-25 09:01:34,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.76/23.15 kb and 14/26 files 2016-08-25 09:01:34,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.66/23.15 kb and 15/26 files 2016-08-25 09:01:34,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.84/23.15 kb and 16/26 files 2016-08-25 09:01:34,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.74/23.15 kb and 17/26 files 2016-08-25 09:01:34,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.64/23.15 kb and 18/26 files 2016-08-25 09:01:34,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.54/23.15 kb and 19/26 files 2016-08-25 09:01:34,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.76/23.15 kb and 20/26 files 2016-08-25 09:01:34,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.66/23.15 kb and 21/26 files 2016-08-25 09:01:34,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.56/23.15 kb and 22/26 files 2016-08-25 09:01:34,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.46/23.15 kb and 23/26 files 2016-08-25 09:01:34,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.36/23.15 kb and 24/26 files 2016-08-25 09:01:34,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.25/23.15 kb and 25/26 files 2016-08-25 09:01:34,272 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/23.15 kb and 1/26 files 2016-08-25 09:01:34,272 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.15/23.15 kb and 26/26 files 2016-08-25 09:01:34,272 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:34,272 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/23.15 kb and 2/26 files 2016-08-25 09:01:34,273 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/23.15 kb and 3/26 files 2016-08-25 09:01:34,274 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/23.15 kb and 4/26 files 2016-08-25 09:01:34,275 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/23.15 kb and 5/26 files 2016-08-25 09:01:34,275 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:34,276 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/23.15 kb and 6/26 files 2016-08-25 09:01:34,276 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) is starting 2016-08-25 09:01:34,276 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/23.15 kb and 7/26 files 2016-08-25 09:01:34,277 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/23.15 kb and 8/26 files 2016-08-25 09:01:34,277 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.09/23.15 kb and 9/26 files 2016-08-25 09:01:34,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.99/23.15 kb and 10/26 files 2016-08-25 09:01:34,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.94/23.15 kb and 11/26 files 2016-08-25 09:01:34,279 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.84/23.15 kb and 12/26 files 2016-08-25 09:01:34,280 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.02/23.15 kb and 13/26 files 2016-08-25 09:01:34,280 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.92/23.15 kb and 14/26 files 2016-08-25 09:01:34,281 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.82/23.15 kb and 15/26 files 2016-08-25 09:01:34,281 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55301 2016-08-25 09:01:34,281 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:34,281 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) started 2016-08-25 09:01:34,281 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:34,281 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.72/23.15 kb and 16/26 files 2016-08-25 09:01:34,281 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:34,281 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:34,282 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.62/23.15 kb and 17/26 files 2016-08-25 09:01:34,283 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.52/23.15 kb and 18/26 files Master started: node-3 ====================================== 46.4. Browse the queue on new master ====================================== 2016-08-25 09:01:34,283 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.42/23.15 kb and 19/26 files 2016-08-25 09:01:34,284 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.32/23.15 kb and 20/26 files 2016-08-25 09:01:34,285 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.54/23.15 kb and 21/26 files 2016-08-25 09:01:34,286 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.44/23.15 kb and 22/26 files 2016-08-25 09:01:34,286 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.34/23.15 kb and 23/26 files 2016-08-25 09:01:34,287 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.24/23.15 kb and 24/26 files 2016-08-25 09:01:34,288 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.25/23.15 kb and 25/26 files 2016-08-25 09:01:34,288 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.15/23.15 kb and 26/26 files 2016-08-25 09:01:34,288 | INFO | hawtdispatch-DEFAULT-1 | Attached ====================================== 46.5. Found ====================================== ====================================== 47.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:34,295 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) is shutting down 2016-08-25 09:01:34,296 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:34,296 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:34,298 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:34,300 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:34,300 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:34,300 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002f 2016-08-25 09:01:34,300 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:34,300 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:34,300 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:34,301 | INFO | Time-limited test | Session: 0x156bec91725002f closed 2016-08-25 09:01:34,301 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:34,301 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55279 which had sessionid 0x156bec91725002f 2016-08-25 09:01:34,404 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:34,405 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) uptime 2.592 seconds 2016-08-25 09:01:34,405 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) is shutdown Wait for master to start up... 2016-08-25 09:01:34,407 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:34,409 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@100a494b 2016-08-25 09:01:34,409 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:34,410 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:34,410 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:34,410 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55303 2016-08-25 09:01:34,410 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55303 2016-08-25 09:01:34,411 | INFO | SyncThread:0 | Established session 0x156bec917250032 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55303 2016-08-25 09:01:34,411 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250032, negotiated timeout = 10000 2016-08-25 09:01:34,418 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:34,852 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:34,854 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:34,875 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:34,876 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:34,880 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:34,883 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:35,079 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55304 2016-08-25 09:01:35,084 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:35,085 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:35,087 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55304 2016-08-25 09:01:35,087 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:35,088 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55304 2016-08-25 09:01:35,088 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:35,269 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:35,277 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:35,278 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000073 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000075.log 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000073 2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000075.log 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:35,297 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/24.10 kb and 1/27 files 2016-08-25 09:01:35,298 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/24.10 kb and 2/27 files 2016-08-25 09:01:35,298 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/24.10 kb and 3/27 files 2016-08-25 09:01:35,299 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/24.10 kb and 4/27 files 2016-08-25 09:01:35,300 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.61/24.10 kb and 5/27 files 2016-08-25 09:01:35,300 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.51/24.10 kb and 6/27 files 2016-08-25 09:01:35,300 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.41/24.10 kb and 7/27 files 2016-08-25 09:01:35,301 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.68/24.10 kb and 8/27 files 2016-08-25 09:01:35,301 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.58/24.10 kb and 9/27 files 2016-08-25 09:01:35,302 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.48/24.10 kb and 10/27 files 2016-08-25 09:01:35,303 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.43/24.10 kb and 11/27 files 2016-08-25 09:01:35,303 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.33/24.10 kb and 12/27 files 2016-08-25 09:01:35,304 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.52/24.10 kb and 13/27 files 2016-08-25 09:01:35,305 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.42/24.10 kb and 14/27 files 2016-08-25 09:01:35,305 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.32/24.10 kb and 15/27 files 2016-08-25 09:01:35,306 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.21/24.10 kb and 16/27 files 2016-08-25 09:01:35,306 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.12/24.10 kb and 17/27 files 2016-08-25 09:01:35,307 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.02/24.10 kb and 18/27 files 2016-08-25 09:01:35,308 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.92/24.10 kb and 19/27 files 2016-08-25 09:01:35,308 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.82/24.10 kb and 20/27 files 2016-08-25 09:01:35,309 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.71/24.10 kb and 21/27 files 2016-08-25 09:01:35,309 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.61/24.10 kb and 22/27 files 2016-08-25 09:01:35,310 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.51/24.10 kb and 23/27 files 2016-08-25 09:01:35,310 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.41/24.10 kb and 24/27 files 2016-08-25 09:01:35,311 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.31/24.10 kb and 25/27 files 2016-08-25 09:01:35,311 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.21/24.10 kb and 26/27 files 2016-08-25 09:01:35,312 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.10/24.10 kb and 27/27 files 2016-08-25 09:01:35,312 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:35,317 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.27/24.10 kb and 1/27 files 2016-08-25 09:01:35,318 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:35,318 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.17/24.10 kb and 2/27 files 2016-08-25 09:01:35,320 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.07/24.10 kb and 3/27 files 2016-08-25 09:01:35,321 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) is starting 2016-08-25 09:01:35,322 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.97/24.10 kb and 4/27 files 2016-08-25 09:01:35,323 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.87/24.10 kb and 5/27 files 2016-08-25 09:01:35,323 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.77/24.10 kb and 6/27 files 2016-08-25 09:01:35,324 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.67/24.10 kb and 7/27 files 2016-08-25 09:01:35,324 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.62/24.10 kb and 8/27 files 2016-08-25 09:01:35,325 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.52/24.10 kb and 9/27 files 2016-08-25 09:01:35,326 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.42/24.10 kb and 10/27 files 2016-08-25 09:01:35,326 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55309 2016-08-25 09:01:35,326 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:35,326 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.60/24.10 kb and 11/27 files 2016-08-25 09:01:35,326 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) started 2016-08-25 09:01:35,326 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:35,326 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:35,327 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:35,327 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.50/24.10 kb and 12/27 files 2016-08-25 09:01:35,327 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.40/24.10 kb and 13/27 files 2016-08-25 09:01:35,328 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.30/24.10 kb and 14/27 files Master started: node-1 ====================================== 47.4. Browse the queue on new master ====================================== 2016-08-25 09:01:35,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.20/24.10 kb and 15/27 files 2016-08-25 09:01:35,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.09/24.10 kb and 16/27 files 2016-08-25 09:01:35,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.99/24.10 kb and 17/27 files 2016-08-25 09:01:35,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.89/24.10 kb and 18/27 files 2016-08-25 09:01:35,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.79/24.10 kb and 19/27 files 2016-08-25 09:01:35,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.69/24.10 kb and 20/27 files 2016-08-25 09:01:35,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.59/24.10 kb and 21/27 files 2016-08-25 09:01:35,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.49/24.10 kb and 22/27 files 2016-08-25 09:01:35,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.39/24.10 kb and 23/27 files 2016-08-25 09:01:35,334 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.29/24.10 kb and 24/27 files 2016-08-25 09:01:35,336 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.31/24.10 kb and 25/27 files 2016-08-25 09:01:35,337 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.21/24.10 kb and 26/27 files 2016-08-25 09:01:35,338 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.10/24.10 kb and 27/27 files 2016-08-25 09:01:35,338 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:35,345 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 47.5. Found ====================================== ====================================== 48.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:35,351 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) is shutting down 2016-08-25 09:01:35,351 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:35,355 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:35,357 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:35,357 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:35,357 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:35,357 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:35,360 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:35,361 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250030 2016-08-25 09:01:35,362 | INFO | Time-limited test | Session: 0x156bec917250030 closed 2016-08-25 09:01:35,362 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:35,362 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55287 which had sessionid 0x156bec917250030 2016-08-25 09:01:35,482 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:35,482 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) uptime 2.922 seconds 2016-08-25 09:01:35,482 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) is shutdown Wait for master to start up... 2016-08-25 09:01:35,484 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:35,486 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7e333bfc 2016-08-25 09:01:35,487 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:35,487 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:35,488 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:35,488 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55311 2016-08-25 09:01:35,488 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55311 2016-08-25 09:01:35,489 | INFO | SyncThread:0 | Established session 0x156bec917250033 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55311 2016-08-25 09:01:35,489 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250033, negotiated timeout = 10000 2016-08-25 09:01:35,496 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:35,745 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:35,749 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:35,772 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:35,772 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:35,776 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:35,780 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:35,974 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55312 2016-08-25 09:01:35,979 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:35,979 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:35,980 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55312 2016-08-25 09:01:35,981 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:35,983 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55312 2016-08-25 09:01:35,983 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:35,997 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:36,151 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:36,152 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:36,153 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:36,153 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000073 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000075.log 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:36,156 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:36,157 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:36,157 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:36,157 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:36,157 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000075.log 2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000073 2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:36,160 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:36,160 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:36,160 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:36,160 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:36,173 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/24.10 kb and 1/27 files 2016-08-25 09:01:36,174 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/24.10 kb and 2/27 files 2016-08-25 09:01:36,175 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/24.10 kb and 3/27 files 2016-08-25 09:01:36,175 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.65/24.10 kb and 4/27 files 2016-08-25 09:01:36,176 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.55/24.10 kb and 5/27 files 2016-08-25 09:01:36,177 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.45/24.10 kb and 6/27 files 2016-08-25 09:01:36,178 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.35/24.10 kb and 7/27 files 2016-08-25 09:01:36,178 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.25/24.10 kb and 8/27 files 2016-08-25 09:01:36,179 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.15/24.10 kb and 9/27 files 2016-08-25 09:01:36,179 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.42/24.10 kb and 10/27 files 2016-08-25 09:01:36,180 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.32/24.10 kb and 11/27 files 2016-08-25 09:01:36,181 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.22/24.10 kb and 12/27 files 2016-08-25 09:01:36,181 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.12/24.10 kb and 13/27 files 2016-08-25 09:01:36,182 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.01/24.10 kb and 14/27 files 2016-08-25 09:01:36,183 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.91/24.10 kb and 15/27 files 2016-08-25 09:01:36,184 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.81/24.10 kb and 16/27 files 2016-08-25 09:01:36,184 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.71/24.10 kb and 17/27 files 2016-08-25 09:01:36,185 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.73/24.10 kb and 18/27 files 2016-08-25 09:01:36,185 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.63/24.10 kb and 19/27 files 2016-08-25 09:01:36,186 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.52/24.10 kb and 20/27 files 2016-08-25 09:01:36,187 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.42/24.10 kb and 21/27 files 2016-08-25 09:01:36,188 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.61/24.10 kb and 22/27 files 2016-08-25 09:01:36,188 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.51/24.10 kb and 23/27 files 2016-08-25 09:01:36,189 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.41/24.10 kb and 24/27 files 2016-08-25 09:01:36,189 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.31/24.10 kb and 25/27 files 2016-08-25 09:01:36,190 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 23.21/24.10 kb and 26/27 files 2016-08-25 09:01:36,191 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 24.10/24.10 kb and 27/27 files 2016-08-25 09:01:36,191 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:36,194 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:36,195 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) is starting 2016-08-25 09:01:36,199 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/24.10 kb and 1/27 files 2016-08-25 09:01:36,199 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55317 2016-08-25 09:01:36,199 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:36,200 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) started 2016-08-25 09:01:36,200 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/24.10 kb and 2/27 files 2016-08-25 09:01:36,200 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:36,200 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:36,200 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:36,200 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/24.10 kb and 3/27 files 2016-08-25 09:01:36,201 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/24.10 kb and 4/27 files 2016-08-25 09:01:36,201 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/24.10 kb and 5/27 files Master started: node-2 ====================================== 48.4. Browse the queue on new master ====================================== 2016-08-25 09:01:36,202 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.51/24.10 kb and 6/27 files 2016-08-25 09:01:36,203 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.41/24.10 kb and 7/27 files 2016-08-25 09:01:36,203 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.60/24.10 kb and 8/27 files 2016-08-25 09:01:36,204 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.87/24.10 kb and 9/27 files 2016-08-25 09:01:36,205 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.77/24.10 kb and 10/27 files 2016-08-25 09:01:36,205 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.67/24.10 kb and 11/27 files 2016-08-25 09:01:36,206 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.57/24.10 kb and 12/27 files 2016-08-25 09:01:36,207 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.47/24.10 kb and 13/27 files 2016-08-25 09:01:36,208 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.37/24.10 kb and 14/27 files 2016-08-25 09:01:36,209 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.26/24.10 kb and 15/27 files 2016-08-25 09:01:36,210 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.16/24.10 kb and 16/27 files 2016-08-25 09:01:36,211 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.06/24.10 kb and 17/27 files 2016-08-25 09:01:36,212 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.01/24.10 kb and 18/27 files 2016-08-25 09:01:36,213 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.91/24.10 kb and 19/27 files 2016-08-25 09:01:36,213 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.80/24.10 kb and 20/27 files 2016-08-25 09:01:36,214 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.70/24.10 kb and 21/27 files 2016-08-25 09:01:36,215 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.60/24.10 kb and 22/27 files 2016-08-25 09:01:36,216 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.50/24.10 kb and 23/27 files 2016-08-25 09:01:36,216 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.40/24.10 kb and 24/27 files 2016-08-25 09:01:36,217 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.31/24.10 kb and 25/27 files 2016-08-25 09:01:36,218 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.21/24.10 kb and 26/27 files 2016-08-25 09:01:36,219 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.10/24.10 kb and 27/27 files 2016-08-25 09:01:36,219 | INFO | hawtdispatch-DEFAULT-1 | Attached ====================================== 48.5. Found ====================================== ====================================== 49.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:36,222 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) is shutting down 2016-08-25 09:01:36,223 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:36,229 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:36,230 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:36,230 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250031 2016-08-25 09:01:36,232 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55295 which had sessionid 0x156bec917250031 2016-08-25 09:01:36,233 | INFO | Time-limited test | Session: 0x156bec917250031 closed 2016-08-25 09:01:36,234 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:36,238 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:36,238 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:36,238 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:36,239 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:36,338 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:36,338 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) uptime 2.772 seconds 2016-08-25 09:01:36,338 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) is shutdown Wait for master to start up... 2016-08-25 09:01:36,340 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:36,343 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@39d47789 2016-08-25 09:01:36,343 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:36,344 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:36,344 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:36,344 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55319 2016-08-25 09:01:36,345 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55319 2016-08-25 09:01:36,345 | INFO | SyncThread:0 | Established session 0x156bec917250034 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55319 2016-08-25 09:01:36,345 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250034, negotiated timeout = 10000 2016-08-25 09:01:36,352 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:36,841 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:36,843 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:36,856 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:36,857 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:36,860 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:36,872 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:37,082 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55320 2016-08-25 09:01:37,087 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:37,088 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55320 2016-08-25 09:01:37,088 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:37,087 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:37,092 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55320 2016-08-25 09:01:37,092 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:37,293 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:37,306 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:37,306 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000076 2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000078.log 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000076 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:37,312 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:37,312 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:37,312 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000078.log 2016-08-25 09:01:37,327 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/25.06 kb and 1/28 files 2016-08-25 09:01:37,328 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/25.06 kb and 2/28 files 2016-08-25 09:01:37,328 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/25.06 kb and 1/28 files 2016-08-25 09:01:37,328 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.12/25.06 kb and 3/28 files 2016-08-25 09:01:37,329 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.02/25.06 kb and 4/28 files 2016-08-25 09:01:37,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/25.06 kb and 2/28 files 2016-08-25 09:01:37,329 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.92/25.06 kb and 5/28 files 2016-08-25 09:01:37,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.69/25.06 kb and 3/28 files 2016-08-25 09:01:37,330 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.82/25.06 kb and 6/28 files 2016-08-25 09:01:37,330 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.72/25.06 kb and 7/28 files 2016-08-25 09:01:37,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/25.06 kb and 4/28 files 2016-08-25 09:01:37,331 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.62/25.06 kb and 8/28 files 2016-08-25 09:01:37,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/25.06 kb and 5/28 files 2016-08-25 09:01:37,331 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.51/25.06 kb and 9/28 files 2016-08-25 09:01:37,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/25.06 kb and 6/28 files 2016-08-25 09:01:37,332 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.41/25.06 kb and 10/28 files 2016-08-25 09:01:37,333 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.31/25.06 kb and 11/28 files 2016-08-25 09:01:37,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/25.06 kb and 7/28 files 2016-08-25 09:01:37,333 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.21/25.06 kb and 12/28 files 2016-08-25 09:01:37,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/25.06 kb and 8/28 files 2016-08-25 09:01:37,334 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.16/25.06 kb and 13/28 files 2016-08-25 09:01:37,335 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.06/25.06 kb and 14/28 files 2016-08-25 09:01:37,335 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.09/25.06 kb and 9/28 files 2016-08-25 09:01:37,335 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.96/25.06 kb and 15/28 files 2016-08-25 09:01:37,335 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/25.06 kb and 10/28 files 2016-08-25 09:01:37,336 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.15/25.06 kb and 16/28 files 2016-08-25 09:01:37,336 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/25.06 kb and 11/28 files 2016-08-25 09:01:37,336 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.04/25.06 kb and 17/28 files 2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.90/25.06 kb and 12/28 files 2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.06/25.06 kb and 18/28 files 2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.80/25.06 kb and 13/28 files 2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.70/25.06 kb and 14/28 files 2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.96/25.06 kb and 19/28 files 2016-08-25 09:01:37,338 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.60/25.06 kb and 15/28 files 2016-08-25 09:01:37,338 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.86/25.06 kb and 20/28 files 2016-08-25 09:01:37,338 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.76/25.06 kb and 21/28 files 2016-08-25 09:01:37,338 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.50/25.06 kb and 16/28 files 2016-08-25 09:01:37,339 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.66/25.06 kb and 22/28 files 2016-08-25 09:01:37,339 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.40/25.06 kb and 17/28 files 2016-08-25 09:01:37,339 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.56/25.06 kb and 23/28 files 2016-08-25 09:01:37,340 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.30/25.06 kb and 18/28 files 2016-08-25 09:01:37,340 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.46/25.06 kb and 24/28 files 2016-08-25 09:01:37,341 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.20/25.06 kb and 19/28 files 2016-08-25 09:01:37,341 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.36/25.06 kb and 25/28 files 2016-08-25 09:01:37,341 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.10/25.06 kb and 20/28 files 2016-08-25 09:01:37,341 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.25/25.06 kb and 26/28 files 2016-08-25 09:01:37,342 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.16/25.06 kb and 27/28 files 2016-08-25 09:01:37,342 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.00/25.06 kb and 21/28 files 2016-08-25 09:01:37,342 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.90/25.06 kb and 22/28 files 2016-08-25 09:01:37,342 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.06/25.06 kb and 28/28 files 2016-08-25 09:01:37,343 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:37,343 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.22/25.06 kb and 23/28 files 2016-08-25 09:01:37,344 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.17/25.06 kb and 24/28 files 2016-08-25 09:01:37,346 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.07/25.06 kb and 25/28 files 2016-08-25 09:01:37,347 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.97/25.06 kb and 26/28 files 2016-08-25 09:01:37,347 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.87/25.06 kb and 27/28 files 2016-08-25 09:01:37,348 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.06/25.06 kb and 28/28 files 2016-08-25 09:01:37,348 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:37,351 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:37,352 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) is starting 2016-08-25 09:01:37,356 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55325 2016-08-25 09:01:37,356 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:37,356 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) started 2016-08-25 09:01:37,356 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:37,356 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:37,357 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-3 ====================================== 2016-08-25 09:01:37,358 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 49.4. Browse the queue on new master ====================================== ====================================== 49.5. Found ====================================== ====================================== 50.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:37,373 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) is shutting down 2016-08-25 09:01:37,373 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:37,375 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:37,376 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:37,376 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250032 2016-08-25 09:01:37,377 | INFO | Time-limited test | Session: 0x156bec917250032 closed 2016-08-25 09:01:37,377 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:37,377 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55303 which had sessionid 0x156bec917250032 2016-08-25 09:01:37,379 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:37,379 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:37,379 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:37,379 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:37,496 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:37,497 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) uptime 3.091 seconds 2016-08-25 09:01:37,497 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) is shutdown Wait for master to start up... 2016-08-25 09:01:37,499 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:37,501 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@566822b0 2016-08-25 09:01:37,501 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:37,502 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:37,502 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:37,503 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55327 2016-08-25 09:01:37,503 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55327 2016-08-25 09:01:37,503 | INFO | SyncThread:0 | Established session 0x156bec917250035 with negotiated timeout 10000 for client /127.0.0.1:55327 2016-08-25 09:01:37,504 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250035, negotiated timeout = 10000 2016-08-25 09:01:37,510 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:37,784 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:37,787 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:37,794 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:37,794 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:37,798 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:37,804 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:38,000 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55328 2016-08-25 09:01:38,006 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:38,007 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:38,008 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55328 2016-08-25 09:01:38,008 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:38,010 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55328 2016-08-25 09:01:38,010 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:38,024 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:38,166 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:38,167 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:38,167 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000076 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000078.log 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000076 2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:38,172 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:38,172 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:38,172 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:38,172 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000078.log 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:38,174 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:38,174 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:38,184 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/25.06 kb and 1/28 files 2016-08-25 09:01:38,185 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/25.06 kb and 2/28 files 2016-08-25 09:01:38,187 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.69/25.06 kb and 3/28 files 2016-08-25 09:01:38,187 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.59/25.06 kb and 4/28 files 2016-08-25 09:01:38,187 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/25.06 kb and 1/28 files 2016-08-25 09:01:38,188 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.92/25.06 kb and 2/28 files 2016-08-25 09:01:38,188 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.49/25.06 kb and 5/28 files 2016-08-25 09:01:38,188 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.39/25.06 kb and 6/28 files 2016-08-25 09:01:38,188 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/25.06 kb and 3/28 files 2016-08-25 09:01:38,189 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/25.06 kb and 4/28 files 2016-08-25 09:01:38,189 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.29/25.06 kb and 7/28 files 2016-08-25 09:01:38,189 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/25.06 kb and 5/28 files 2016-08-25 09:01:38,189 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.19/25.06 kb and 8/28 files 2016-08-25 09:01:38,190 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.56/25.06 kb and 6/28 files 2016-08-25 09:01:38,190 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.09/25.06 kb and 9/28 files 2016-08-25 09:01:38,190 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.89/25.06 kb and 7/28 files 2016-08-25 09:01:38,190 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.99/25.06 kb and 10/28 files 2016-08-25 09:01:38,191 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.79/25.06 kb and 8/28 files 2016-08-25 09:01:38,191 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.31/25.06 kb and 11/28 files 2016-08-25 09:01:38,192 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.68/25.06 kb and 9/28 files 2016-08-25 09:01:38,192 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.21/25.06 kb and 12/28 files 2016-08-25 09:01:38,192 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.11/25.06 kb and 13/28 files 2016-08-25 09:01:38,192 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.58/25.06 kb and 10/28 files 2016-08-25 09:01:38,193 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.48/25.06 kb and 11/28 files 2016-08-25 09:01:38,193 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.01/25.06 kb and 14/28 files 2016-08-25 09:01:38,193 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.38/25.06 kb and 12/28 files 2016-08-25 09:01:38,193 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.91/25.06 kb and 15/28 files 2016-08-25 09:01:38,194 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.28/25.06 kb and 13/28 files 2016-08-25 09:01:38,194 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.81/25.06 kb and 16/28 files 2016-08-25 09:01:38,195 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.18/25.06 kb and 14/28 files 2016-08-25 09:01:38,195 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.71/25.06 kb and 17/28 files 2016-08-25 09:01:38,195 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.36/25.06 kb and 15/28 files 2016-08-25 09:01:38,195 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.61/25.06 kb and 18/28 files 2016-08-25 09:01:38,196 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.62/25.06 kb and 19/28 files 2016-08-25 09:01:38,196 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.26/25.06 kb and 16/28 files 2016-08-25 09:01:38,197 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.81/25.06 kb and 20/28 files 2016-08-25 09:01:38,197 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.16/25.06 kb and 17/28 files 2016-08-25 09:01:38,198 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.71/25.06 kb and 21/28 files 2016-08-25 09:01:38,198 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.06/25.06 kb and 18/28 files 2016-08-25 09:01:38,198 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.96/25.06 kb and 19/28 files 2016-08-25 09:01:38,198 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.60/25.06 kb and 22/28 files 2016-08-25 09:01:38,199 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.86/25.06 kb and 20/28 files 2016-08-25 09:01:38,199 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.50/25.06 kb and 23/28 files 2016-08-25 09:01:38,199 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.40/25.06 kb and 24/28 files 2016-08-25 09:01:38,199 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.76/25.06 kb and 21/28 files 2016-08-25 09:01:38,200 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.66/25.06 kb and 22/28 files 2016-08-25 09:01:38,200 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.31/25.06 kb and 25/28 files 2016-08-25 09:01:38,200 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 23.26/25.06 kb and 26/28 files 2016-08-25 09:01:38,200 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.56/25.06 kb and 23/28 files 2016-08-25 09:01:38,201 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.46/25.06 kb and 24/28 files 2016-08-25 09:01:38,201 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 24.16/25.06 kb and 27/28 files 2016-08-25 09:01:38,202 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.36/25.06 kb and 25/28 files 2016-08-25 09:01:38,202 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 25.06/25.06 kb and 28/28 files 2016-08-25 09:01:38,202 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:38,203 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.26/25.06 kb and 26/28 files 2016-08-25 09:01:38,204 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.16/25.06 kb and 27/28 files 2016-08-25 09:01:38,205 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.06/25.06 kb and 28/28 files 2016-08-25 09:01:38,205 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:38,206 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:38,208 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) is starting 2016-08-25 09:01:38,211 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:38,212 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55333 2016-08-25 09:01:38,212 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:38,212 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) started 2016-08-25 09:01:38,212 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:38,212 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:38,212 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-1 ====================================== 50.4. Browse the queue on new master ====================================== ====================================== 50.5. Found ====================================== ====================================== 51.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:38,229 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) is shutting down 2016-08-25 09:01:38,229 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:38,232 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:38,234 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:38,234 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250033 2016-08-25 09:01:38,235 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55311 which had sessionid 0x156bec917250033 2016-08-25 09:01:38,235 | INFO | Time-limited test | Session: 0x156bec917250033 closed 2016-08-25 09:01:38,235 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:38,237 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:38,237 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:38,238 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:38,238 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:38,339 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:38,340 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) uptime 2.857 seconds 2016-08-25 09:01:38,340 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) is shutdown Wait for master to start up... 2016-08-25 09:01:38,343 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:38,346 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6fac659 2016-08-25 09:01:38,346 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:38,347 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:38,347 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:38,348 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55335 2016-08-25 09:01:38,348 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55335 2016-08-25 09:01:38,349 | INFO | SyncThread:0 | Established session 0x156bec917250036 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55335 2016-08-25 09:01:38,349 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250036, negotiated timeout = 10000 2016-08-25 09:01:38,361 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:38,796 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:38,798 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:38,798 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:38,799 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:38,802 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:38,806 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:39,025 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55336 2016-08-25 09:01:39,030 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:39,030 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:39,032 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55336 2016-08-25 09:01:39,032 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:39,034 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55336 2016-08-25 09:01:39,034 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:39,219 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:39,236 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:39,236 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000081.log 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000079 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:39,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:39,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:39,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:39,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:39,240 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:39,240 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000079 2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:39,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000081.log 2016-08-25 09:01:39,260 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/26.01 kb and 1/29 files 2016-08-25 09:01:39,260 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/26.01 kb and 2/29 files 2016-08-25 09:01:39,261 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.75/26.01 kb and 3/29 files 2016-08-25 09:01:39,262 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.65/26.01 kb and 4/29 files 2016-08-25 09:01:39,262 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.55/26.01 kb and 5/29 files 2016-08-25 09:01:39,263 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.45/26.01 kb and 6/29 files 2016-08-25 09:01:39,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/26.01 kb and 7/29 files 2016-08-25 09:01:39,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.53/26.01 kb and 8/29 files 2016-08-25 09:01:39,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.43/26.01 kb and 9/29 files 2016-08-25 09:01:39,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/26.01 kb and 10/29 files 2016-08-25 09:01:39,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.23/26.01 kb and 11/29 files 2016-08-25 09:01:39,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.13/26.01 kb and 12/29 files 2016-08-25 09:01:39,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.03/26.01 kb and 13/29 files 2016-08-25 09:01:39,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.92/26.01 kb and 14/29 files 2016-08-25 09:01:39,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.82/26.01 kb and 15/29 files 2016-08-25 09:01:39,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.72/26.01 kb and 16/29 files 2016-08-25 09:01:39,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.62/26.01 kb and 17/29 files 2016-08-25 09:01:39,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.99/26.01 kb and 18/29 files 2016-08-25 09:01:39,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.89/26.01 kb and 19/29 files 2016-08-25 09:01:39,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.79/26.01 kb and 20/29 files 2016-08-25 09:01:39,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.69/26.01 kb and 21/29 files 2016-08-25 09:01:39,272 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.59/26.01 kb and 22/29 files 2016-08-25 09:01:39,273 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.49/26.01 kb and 23/29 files 2016-08-25 09:01:39,273 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.39/26.01 kb and 24/29 files 2016-08-25 09:01:39,274 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.29/26.01 kb and 25/29 files 2016-08-25 09:01:39,275 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.19/26.01 kb and 26/29 files 2016-08-25 09:01:39,275 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.09/26.01 kb and 27/29 files 2016-08-25 09:01:39,276 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.11/26.01 kb and 28/29 files 2016-08-25 09:01:39,276 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.01/26.01 kb and 29/29 files 2016-08-25 09:01:39,276 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:39,277 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/26.01 kb and 1/29 files 2016-08-25 09:01:39,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/26.01 kb and 2/29 files 2016-08-25 09:01:39,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/26.01 kb and 3/29 files 2016-08-25 09:01:39,279 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.07/26.01 kb and 4/29 files 2016-08-25 09:01:39,279 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.97/26.01 kb and 5/29 files 2016-08-25 09:01:39,280 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.87/26.01 kb and 6/29 files 2016-08-25 09:01:39,281 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.77/26.01 kb and 7/29 files 2016-08-25 09:01:39,281 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.67/26.01 kb and 8/29 files 2016-08-25 09:01:39,282 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:39,282 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.57/26.01 kb and 9/29 files 2016-08-25 09:01:39,282 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.47/26.01 kb and 10/29 files 2016-08-25 09:01:39,282 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) is starting 2016-08-25 09:01:39,283 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.37/26.01 kb and 11/29 files 2016-08-25 09:01:39,283 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.27/26.01 kb and 12/29 files 2016-08-25 09:01:39,284 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.17/26.01 kb and 13/29 files 2016-08-25 09:01:39,284 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.07/26.01 kb and 14/29 files 2016-08-25 09:01:39,284 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.02/26.01 kb and 15/29 files 2016-08-25 09:01:39,285 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.92/26.01 kb and 16/29 files 2016-08-25 09:01:39,286 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55341 2016-08-25 09:01:39,286 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:39,287 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) started 2016-08-25 09:01:39,287 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:39,287 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:39,287 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:39,287 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.82/26.01 kb and 17/29 files 2016-08-25 09:01:39,288 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.72/26.01 kb and 18/29 files 2016-08-25 09:01:39,290 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.61/26.01 kb and 19/29 files Master started: node-2 ====================================== 51.4. Browse the queue on new master ====================================== 2016-08-25 09:01:39,290 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.51/26.01 kb and 20/29 files 2016-08-25 09:01:39,291 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.41/26.01 kb and 21/29 files 2016-08-25 09:01:39,292 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.31/26.01 kb and 22/29 files 2016-08-25 09:01:39,293 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.21/26.01 kb and 23/29 files 2016-08-25 09:01:39,293 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.11/26.01 kb and 24/29 files 2016-08-25 09:01:39,294 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.01/26.01 kb and 25/29 files 2016-08-25 09:01:39,295 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.02/26.01 kb and 26/29 files 2016-08-25 09:01:39,295 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.92/26.01 kb and 27/29 files 2016-08-25 09:01:39,296 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.82/26.01 kb and 28/29 files 2016-08-25 09:01:39,296 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.01/26.01 kb and 29/29 files 2016-08-25 09:01:39,296 | INFO | hawtdispatch-DEFAULT-1 | Attached ====================================== 51.5. Found ====================================== ====================================== 52.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:39,306 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) is shutting down 2016-08-25 09:01:39,306 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:39,306 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:39,308 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:39,308 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250034 2016-08-25 09:01:39,309 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:39,309 | INFO | Time-limited test | Session: 0x156bec917250034 closed 2016-08-25 09:01:39,309 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:39,309 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55319 which had sessionid 0x156bec917250034 2016-08-25 09:01:39,312 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:39,312 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:39,312 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:39,312 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:39,437 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:39,438 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) uptime 3.099 seconds 2016-08-25 09:01:39,438 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) is shutdown Wait for master to start up... 2016-08-25 09:01:39,440 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:39,445 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4ceb4f17 2016-08-25 09:01:39,445 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:39,446 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:39,446 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:39,446 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55343 2016-08-25 09:01:39,446 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55343 2016-08-25 09:01:39,447 | INFO | SyncThread:0 | Established session 0x156bec917250037 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55343 2016-08-25 09:01:39,447 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250037, negotiated timeout = 10000 2016-08-25 09:01:39,454 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:39,760 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:39,762 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:39,767 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:39,767 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:39,771 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:39,776 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:39,998 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55344 2016-08-25 09:01:40,005 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:40,005 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:40,007 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55344 2016-08-25 09:01:40,007 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:40,010 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55344 2016-08-25 09:01:40,010 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:40,021 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:40,206 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:40,207 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:40,207 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:40,207 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000081.log 2016-08-25 09:01:40,207 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000079 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:40,210 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:40,210 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:40,210 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:40,211 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:40,211 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000081.log 2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000079 2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:40,215 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:40,215 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:40,215 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:40,223 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/26.01 kb and 1/29 files 2016-08-25 09:01:40,224 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.08/26.01 kb and 2/29 files 2016-08-25 09:01:40,225 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/26.01 kb and 3/29 files 2016-08-25 09:01:40,225 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/26.01 kb and 4/29 files 2016-08-25 09:01:40,226 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.78/26.01 kb and 5/29 files 2016-08-25 09:01:40,227 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/26.01 kb and 6/29 files 2016-08-25 09:01:40,228 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/26.01 kb and 7/29 files 2016-08-25 09:01:40,229 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.01/26.01 kb and 8/29 files 2016-08-25 09:01:40,230 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.91/26.01 kb and 9/29 files 2016-08-25 09:01:40,230 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.80/26.01 kb and 10/29 files 2016-08-25 09:01:40,231 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.70/26.01 kb and 11/29 files 2016-08-25 09:01:40,231 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.60/26.01 kb and 12/29 files 2016-08-25 09:01:40,232 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.50/26.01 kb and 13/29 files 2016-08-25 09:01:40,233 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.40/26.01 kb and 14/29 files 2016-08-25 09:01:40,233 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.30/26.01 kb and 15/29 files 2016-08-25 09:01:40,234 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.20/26.01 kb and 16/29 files 2016-08-25 09:01:40,234 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.22/26.01 kb and 17/29 files 2016-08-25 09:01:40,235 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.12/26.01 kb and 18/29 files 2016-08-25 09:01:40,236 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.02/26.01 kb and 19/29 files 2016-08-25 09:01:40,236 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.92/26.01 kb and 20/29 files 2016-08-25 09:01:40,237 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.81/26.01 kb and 21/29 files 2016-08-25 09:01:40,237 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.71/26.01 kb and 22/29 files 2016-08-25 09:01:40,238 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.61/26.01 kb and 23/29 files 2016-08-25 09:01:40,239 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.51/26.01 kb and 24/29 files 2016-08-25 09:01:40,239 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.41/26.01 kb and 25/29 files 2016-08-25 09:01:40,240 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.31/26.01 kb and 26/29 files 2016-08-25 09:01:40,240 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.21/26.01 kb and 27/29 files 2016-08-25 09:01:40,241 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/26.01 kb and 1/29 files 2016-08-25 09:01:40,241 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.11/26.01 kb and 28/29 files 2016-08-25 09:01:40,241 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/26.01 kb and 2/29 files 2016-08-25 09:01:40,241 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.01/26.01 kb and 29/29 files 2016-08-25 09:01:40,242 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:40,242 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.99/26.01 kb and 3/29 files 2016-08-25 09:01:40,242 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.89/26.01 kb and 4/29 files 2016-08-25 09:01:40,243 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.79/26.01 kb and 5/29 files 2016-08-25 09:01:40,250 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:40,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.16/26.01 kb and 6/29 files 2016-08-25 09:01:40,251 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) is starting 2016-08-25 09:01:40,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.06/26.01 kb and 7/29 files 2016-08-25 09:01:40,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.96/26.01 kb and 8/29 files 2016-08-25 09:01:40,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.85/26.01 kb and 9/29 files 2016-08-25 09:01:40,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.80/26.01 kb and 10/29 files 2016-08-25 09:01:40,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.70/26.01 kb and 11/29 files 2016-08-25 09:01:40,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.60/26.01 kb and 12/29 files 2016-08-25 09:01:40,255 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55349 2016-08-25 09:01:40,255 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:40,255 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) started 2016-08-25 09:01:40,255 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:40,255 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:40,255 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:40,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.62/26.01 kb and 13/29 files 2016-08-25 09:01:40,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.52/26.01 kb and 14/29 files Master started: node-3 ====================================== 52.4. Browse the queue on new master ====================================== 2016-08-25 09:01:40,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.42/26.01 kb and 15/29 files 2016-08-25 09:01:40,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.32/26.01 kb and 16/29 files 2016-08-25 09:01:40,260 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.21/26.01 kb and 17/29 files 2016-08-25 09:01:40,261 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.11/26.01 kb and 18/29 files 2016-08-25 09:01:40,262 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.01/26.01 kb and 19/29 files 2016-08-25 09:01:40,263 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.91/26.01 kb and 20/29 files 2016-08-25 09:01:40,264 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.81/26.01 kb and 21/29 files 2016-08-25 09:01:40,265 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.71/26.01 kb and 22/29 files 2016-08-25 09:01:40,266 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.61/26.01 kb and 23/29 files 2016-08-25 09:01:40,266 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.51/26.01 kb and 24/29 files ====================================== 52.5. Found ====================================== ====================================== 53.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:40,269 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.41/26.01 kb and 25/29 files 2016-08-25 09:01:40,271 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) is shutting down 2016-08-25 09:01:40,272 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:40,274 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:40,275 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:40,275 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250035 2016-08-25 09:01:40,275 | INFO | Time-limited test | Session: 0x156bec917250035 closed 2016-08-25 09:01:40,276 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55327 which had sessionid 0x156bec917250035 2016-08-25 09:01:40,276 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:40,276 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:40,276 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:40,276 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.31/26.01 kb and 26/29 files 2016-08-25 09:01:40,277 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.21/26.01 kb and 27/29 files 2016-08-25 09:01:40,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.11/26.01 kb and 28/29 files 2016-08-25 09:01:40,279 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.01/26.01 kb and 29/29 files 2016-08-25 09:01:40,279 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:40,289 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:40,289 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:40,379 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:40,380 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) uptime 2.883 seconds 2016-08-25 09:01:40,380 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) is shutdown Wait for master to start up... 2016-08-25 09:01:40,382 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:40,385 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4e326f04 2016-08-25 09:01:40,385 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:40,386 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:40,386 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:40,386 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55351 2016-08-25 09:01:40,387 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55351 2016-08-25 09:01:40,387 | INFO | SyncThread:0 | Established session 0x156bec917250038 with negotiated timeout 10000 for client /127.0.0.1:55351 2016-08-25 09:01:40,387 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250038, negotiated timeout = 10000 2016-08-25 09:01:40,395 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:40,884 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:40,887 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:40,917 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:40,917 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:40,921 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:40,925 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:41,154 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55352 2016-08-25 09:01:41,158 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:41,158 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:41,160 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55352 2016-08-25 09:01:41,160 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:41,162 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55352 2016-08-25 09:01:41,162 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:41,339 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:41,350 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000084.log 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000082 2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:41,354 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:41,355 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:41,355 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:41,355 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000084.log 2016-08-25 09:01:41,355 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000082 2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:41,367 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/26.96 kb and 1/30 files 2016-08-25 09:01:41,367 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/26.96 kb and 2/30 files 2016-08-25 09:01:41,368 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/26.96 kb and 3/30 files 2016-08-25 09:01:41,368 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/26.96 kb and 4/30 files 2016-08-25 09:01:41,369 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/26.96 kb and 5/30 files 2016-08-25 09:01:41,369 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/26.96 kb and 6/30 files 2016-08-25 09:01:41,370 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/26.96 kb and 7/30 files 2016-08-25 09:01:41,370 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.19/26.96 kb and 8/30 files 2016-08-25 09:01:41,371 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.09/26.96 kb and 9/30 files 2016-08-25 09:01:41,371 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.99/26.96 kb and 10/30 files 2016-08-25 09:01:41,372 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.89/26.96 kb and 11/30 files 2016-08-25 09:01:41,373 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.79/26.96 kb and 12/30 files 2016-08-25 09:01:41,373 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.69/26.96 kb and 13/30 files 2016-08-25 09:01:41,374 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.59/26.96 kb and 14/30 files 2016-08-25 09:01:41,374 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.49/26.96 kb and 15/30 files 2016-08-25 09:01:41,375 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.39/26.96 kb and 16/30 files 2016-08-25 09:01:41,375 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.29/26.96 kb and 17/30 files 2016-08-25 09:01:41,376 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.19/26.96 kb and 18/30 files 2016-08-25 09:01:41,377 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.37/26.96 kb and 19/30 files 2016-08-25 09:01:41,377 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.27/26.96 kb and 20/30 files 2016-08-25 09:01:41,378 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.17/26.96 kb and 21/30 files 2016-08-25 09:01:41,378 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.07/26.96 kb and 22/30 files 2016-08-25 09:01:41,379 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.97/26.96 kb and 23/30 files 2016-08-25 09:01:41,379 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.87/26.96 kb and 24/30 files 2016-08-25 09:01:41,380 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.29/26.96 kb and 25/30 files 2016-08-25 09:01:41,381 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.31/26.96 kb and 26/30 files 2016-08-25 09:01:41,381 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.26/26.96 kb and 27/30 files 2016-08-25 09:01:41,382 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.16/26.96 kb and 28/30 files 2016-08-25 09:01:41,382 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/26.96 kb and 1/30 files 2016-08-25 09:01:41,382 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.06/26.96 kb and 29/30 files 2016-08-25 09:01:41,383 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/26.96 kb and 2/30 files 2016-08-25 09:01:41,383 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.96/26.96 kb and 30/30 files 2016-08-25 09:01:41,383 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:41,383 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.98/26.96 kb and 3/30 files 2016-08-25 09:01:41,384 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.88/26.96 kb and 4/30 files 2016-08-25 09:01:41,384 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.78/26.96 kb and 5/30 files 2016-08-25 09:01:41,385 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.68/26.96 kb and 6/30 files 2016-08-25 09:01:41,385 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/26.96 kb and 7/30 files 2016-08-25 09:01:41,386 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.48/26.96 kb and 8/30 files 2016-08-25 09:01:41,387 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.38/26.96 kb and 9/30 files 2016-08-25 09:01:41,387 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.28/26.96 kb and 10/30 files 2016-08-25 09:01:41,388 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.18/26.96 kb and 11/30 files 2016-08-25 09:01:41,388 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.08/26.96 kb and 12/30 files 2016-08-25 09:01:41,389 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:41,389 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.98/26.96 kb and 13/30 files 2016-08-25 09:01:41,389 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.88/26.96 kb and 14/30 files 2016-08-25 09:01:41,390 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) is starting 2016-08-25 09:01:41,390 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.78/26.96 kb and 15/30 files 2016-08-25 09:01:41,391 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.68/26.96 kb and 16/30 files 2016-08-25 09:01:41,391 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.58/26.96 kb and 17/30 files 2016-08-25 09:01:41,392 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.00/26.96 kb and 18/30 files 2016-08-25 09:01:41,392 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.90/26.96 kb and 19/30 files 2016-08-25 09:01:41,393 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.80/26.96 kb and 20/30 files 2016-08-25 09:01:41,394 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.70/26.96 kb and 21/30 files 2016-08-25 09:01:41,395 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55357 2016-08-25 09:01:41,395 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.59/26.96 kb and 22/30 files 2016-08-25 09:01:41,395 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:41,395 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) started 2016-08-25 09:01:41,395 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:41,395 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:41,395 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:41,396 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.61/26.96 kb and 23/30 files 2016-08-25 09:01:41,396 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.51/26.96 kb and 24/30 files 2016-08-25 09:01:41,397 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.41/26.96 kb and 25/30 files Master started: node-1 ====================================== 53.4. Browse the queue on new master ====================================== 2016-08-25 09:01:41,398 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.31/26.96 kb and 26/30 files 2016-08-25 09:01:41,398 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.21/26.96 kb and 27/30 files 2016-08-25 09:01:41,399 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.11/26.96 kb and 28/30 files 2016-08-25 09:01:41,400 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.06/26.96 kb and 29/30 files 2016-08-25 09:01:41,400 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.96/26.96 kb and 30/30 files 2016-08-25 09:01:41,401 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:41,407 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 53.5. Found ====================================== ====================================== 54.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:41,416 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) is shutting down 2016-08-25 09:01:41,416 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:41,420 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:41,421 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:41,421 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:41,421 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:41,421 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:41,422 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:41,422 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250036 2016-08-25 09:01:41,423 | INFO | Time-limited test | Session: 0x156bec917250036 closed 2016-08-25 09:01:41,423 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55335 which had sessionid 0x156bec917250036 2016-08-25 09:01:41,423 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:41,546 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:41,546 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) uptime 3.205 seconds 2016-08-25 09:01:41,547 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) is shutdown Wait for master to start up... 2016-08-25 09:01:41,549 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:41,551 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@25d73bfd 2016-08-25 09:01:41,551 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:41,552 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:41,553 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:41,553 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55359 2016-08-25 09:01:41,553 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55359 2016-08-25 09:01:41,554 | INFO | SyncThread:0 | Established session 0x156bec917250039 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55359 2016-08-25 09:01:41,554 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250039, negotiated timeout = 10000 2016-08-25 09:01:41,561 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:41,843 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:41,845 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:41,856 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:41,857 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:41,860 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:41,868 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:42,097 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55360 2016-08-25 09:01:42,103 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:42,103 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:42,105 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55360 2016-08-25 09:01:42,105 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:42,108 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55360 2016-08-25 09:01:42,108 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:42,122 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:42,125 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000084.log 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000082 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:42,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:42,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:42,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:42,145 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/26.96 kb and 1/30 files 2016-08-25 09:01:42,145 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/26.96 kb and 2/30 files 2016-08-25 09:01:42,146 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/26.96 kb and 3/30 files 2016-08-25 09:01:42,146 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/26.96 kb and 4/30 files 2016-08-25 09:01:42,147 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/26.96 kb and 5/30 files 2016-08-25 09:01:42,147 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/26.96 kb and 6/30 files 2016-08-25 09:01:42,148 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/26.96 kb and 7/30 files 2016-08-25 09:01:42,149 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.32/26.96 kb and 8/30 files 2016-08-25 09:01:42,149 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.27/26.96 kb and 9/30 files 2016-08-25 09:01:42,150 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.17/26.96 kb and 10/30 files 2016-08-25 09:01:42,150 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.06/26.96 kb and 11/30 files 2016-08-25 09:01:42,151 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.96/26.96 kb and 12/30 files 2016-08-25 09:01:42,152 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.86/26.96 kb and 13/30 files 2016-08-25 09:01:42,152 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.76/26.96 kb and 14/30 files 2016-08-25 09:01:42,153 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.66/26.96 kb and 15/30 files 2016-08-25 09:01:42,153 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.56/26.96 kb and 16/30 files 2016-08-25 09:01:42,154 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.75/26.96 kb and 17/30 files 2016-08-25 09:01:42,155 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.64/26.96 kb and 18/30 files 2016-08-25 09:01:42,155 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.54/26.96 kb and 19/30 files 2016-08-25 09:01:42,156 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.44/26.96 kb and 20/30 files 2016-08-25 09:01:42,157 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.34/26.96 kb and 21/30 files 2016-08-25 09:01:42,157 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.24/26.96 kb and 22/30 files 2016-08-25 09:01:42,159 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.66/26.96 kb and 23/30 files 2016-08-25 09:01:42,159 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.56/26.96 kb and 24/30 files 2016-08-25 09:01:42,160 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.46/26.96 kb and 25/30 files 2016-08-25 09:01:42,161 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.36/26.96 kb and 26/30 files 2016-08-25 09:01:42,161 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.26/26.96 kb and 27/30 files 2016-08-25 09:01:42,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.16/26.96 kb and 28/30 files 2016-08-25 09:01:42,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.06/26.96 kb and 29/30 files 2016-08-25 09:01:42,163 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.96/26.96 kb and 30/30 files 2016-08-25 09:01:42,163 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:42,165 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:42,166 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) is starting 2016-08-25 09:01:42,170 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55364 2016-08-25 09:01:42,170 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:42,170 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) started 2016-08-25 09:01:42,170 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:42,170 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:42,171 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-2 ====================================== 54.4. Browse the queue on new master ====================================== ====================================== 54.5. Found ====================================== ====================================== 55.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:42,188 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) is shutting down 2016-08-25 09:01:42,188 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:42,190 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:42,191 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:42,191 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:42,192 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:42,192 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250037 2016-08-25 09:01:42,192 | INFO | Time-limited test | Session: 0x156bec917250037 closed 2016-08-25 09:01:42,193 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55343 which had sessionid 0x156bec917250037 2016-08-25 09:01:42,193 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:42,298 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:42,298 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) uptime 2.859 seconds 2016-08-25 09:01:42,299 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) is shutdown Wait for master to start up... 2016-08-25 09:01:42,301 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:42,303 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@77bac550 2016-08-25 09:01:42,303 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:42,304 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:42,305 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:42,305 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55366 2016-08-25 09:01:42,305 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55366 2016-08-25 09:01:42,307 | INFO | SyncThread:0 | Established session 0x156bec91725003a with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55366 2016-08-25 09:01:42,307 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003a, negotiated timeout = 10000 2016-08-25 09:01:42,313 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:42,532 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:42,532 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:42,532 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:42,534 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:42,784 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:42,788 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:42,792 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:42,999 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55367 2016-08-25 09:01:43,004 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:43,004 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:43,004 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55367 2016-08-25 09:01:43,004 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:43,005 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55367 2016-08-25 09:01:43,005 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:43,224 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:43,227 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000085 2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000087.log 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:43,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/27.91 kb and 1/31 files 2016-08-25 09:01:43,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/27.91 kb and 2/31 files 2016-08-25 09:01:43,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.75/27.91 kb and 3/31 files 2016-08-25 09:01:43,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.64/27.91 kb and 4/31 files 2016-08-25 09:01:43,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/27.91 kb and 5/31 files 2016-08-25 09:01:43,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.02/27.91 kb and 6/31 files 2016-08-25 09:01:43,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.92/27.91 kb and 7/31 files 2016-08-25 09:01:43,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.82/27.91 kb and 8/31 files 2016-08-25 09:01:43,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.72/27.91 kb and 9/31 files 2016-08-25 09:01:43,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.90/27.91 kb and 10/31 files 2016-08-25 09:01:43,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.80/27.91 kb and 11/31 files 2016-08-25 09:01:43,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.70/27.91 kb and 12/31 files 2016-08-25 09:01:43,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.60/27.91 kb and 13/31 files 2016-08-25 09:01:43,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.50/27.91 kb and 14/31 files 2016-08-25 09:01:43,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.40/27.91 kb and 15/31 files 2016-08-25 09:01:43,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.30/27.91 kb and 16/31 files 2016-08-25 09:01:43,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.20/27.91 kb and 17/31 files 2016-08-25 09:01:43,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.10/27.91 kb and 18/31 files 2016-08-25 09:01:43,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.00/27.91 kb and 19/31 files 2016-08-25 09:01:43,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.90/27.91 kb and 20/31 files 2016-08-25 09:01:43,258 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.80/27.91 kb and 21/31 files 2016-08-25 09:01:43,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.70/27.91 kb and 22/31 files 2016-08-25 09:01:43,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.59/27.91 kb and 23/31 files 2016-08-25 09:01:43,260 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.49/27.91 kb and 24/31 files 2016-08-25 09:01:43,261 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.39/27.91 kb and 25/31 files 2016-08-25 09:01:43,261 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.29/27.91 kb and 26/31 files 2016-08-25 09:01:43,261 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.19/27.91 kb and 27/31 files 2016-08-25 09:01:43,262 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.09/27.91 kb and 28/31 files 2016-08-25 09:01:43,262 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.99/27.91 kb and 29/31 files 2016-08-25 09:01:43,263 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.01/27.91 kb and 30/31 files 2016-08-25 09:01:43,263 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.91/27.91 kb and 31/31 files 2016-08-25 09:01:43,263 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:43,267 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:43,268 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:43,268 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) is starting 2016-08-25 09:01:43,269 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:43,269 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:43,269 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000087.log 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000085 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:43,272 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55372 2016-08-25 09:01:43,273 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:43,273 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) started 2016-08-25 09:01:43,273 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:43,273 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:43,273 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-1 ====================================== 55.4. Browse the queue on new master ====================================== ====================================== 55.5. Found ====================================== ====================================== 56.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:43,289 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) is shutting down 2016-08-25 09:01:43,290 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:43,294 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:01:43,294 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:43,294 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:43,295 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250039 2016-08-25 09:01:43,296 | INFO | Time-limited test | Session: 0x156bec917250039 closed 2016-08-25 09:01:43,296 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55359 which had sessionid 0x156bec917250039 2016-08-25 09:01:43,296 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:43,298 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:43,298 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:43,299 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:43,299 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:43,299 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:43,420 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:43,420 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) uptime 1.873 seconds 2016-08-25 09:01:43,420 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) is shutdown Wait for master to start up... 2016-08-25 09:01:43,423 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:43,425 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6fdbce7c 2016-08-25 09:01:43,425 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:43,426 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:43,427 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:43,428 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55374 2016-08-25 09:01:43,428 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55374 2016-08-25 09:01:43,429 | INFO | SyncThread:0 | Established session 0x156bec91725003b with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55374 2016-08-25 09:01:43,429 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003b, negotiated timeout = 10000 2016-08-25 09:01:43,437 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:43,522 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:43,525 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:43,727 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:43,727 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:43,731 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:43,734 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:43,928 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55375 2016-08-25 09:01:43,934 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:43,934 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:43,934 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55375 2016-08-25 09:01:43,934 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:43,936 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55375 2016-08-25 09:01:43,936 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:43,952 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:43,956 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:43,957 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000085 2016-08-25 09:01:43,957 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:43,957 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:43,957 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000087.log 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:43,981 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.48/27.91 kb and 1/31 files 2016-08-25 09:01:43,982 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.38/27.91 kb and 2/31 files 2016-08-25 09:01:43,982 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.27/27.91 kb and 3/31 files 2016-08-25 09:01:43,983 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.17/27.91 kb and 4/31 files 2016-08-25 09:01:43,984 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.07/27.91 kb and 5/31 files 2016-08-25 09:01:43,984 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.97/27.91 kb and 6/31 files 2016-08-25 09:01:43,985 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.87/27.91 kb and 7/31 files 2016-08-25 09:01:43,985 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.77/27.91 kb and 8/31 files 2016-08-25 09:01:43,986 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.67/27.91 kb and 9/31 files 2016-08-25 09:01:43,986 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.57/27.91 kb and 10/31 files 2016-08-25 09:01:43,987 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.47/27.91 kb and 11/31 files 2016-08-25 09:01:43,987 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.65/27.91 kb and 12/31 files 2016-08-25 09:01:43,988 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.56/27.91 kb and 13/31 files 2016-08-25 09:01:43,988 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.57/27.91 kb and 14/31 files 2016-08-25 09:01:43,989 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.47/27.91 kb and 15/31 files 2016-08-25 09:01:43,989 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.37/27.91 kb and 16/31 files 2016-08-25 09:01:43,990 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.27/27.91 kb and 17/31 files 2016-08-25 09:01:43,991 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.17/27.91 kb and 18/31 files 2016-08-25 09:01:43,991 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.07/27.91 kb and 19/31 files 2016-08-25 09:01:43,992 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.96/27.91 kb and 20/31 files 2016-08-25 09:01:43,993 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.86/27.91 kb and 21/31 files 2016-08-25 09:01:43,993 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.76/27.91 kb and 22/31 files 2016-08-25 09:01:43,994 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.66/27.91 kb and 23/31 files 2016-08-25 09:01:43,995 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.56/27.91 kb and 24/31 files 2016-08-25 09:01:43,995 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 22.46/27.91 kb and 25/31 files 2016-08-25 09:01:43,996 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 23.36/27.91 kb and 26/31 files 2016-08-25 09:01:43,996 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 24.26/27.91 kb and 27/31 files 2016-08-25 09:01:43,997 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 25.16/27.91 kb and 28/31 files 2016-08-25 09:01:43,997 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 26.06/27.91 kb and 29/31 files 2016-08-25 09:01:43,998 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 27.01/27.91 kb and 30/31 files 2016-08-25 09:01:43,999 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 27.91/27.91 kb and 31/31 files 2016-08-25 09:01:43,999 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:44,001 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:44,002 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) is starting 2016-08-25 09:01:44,006 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55379 2016-08-25 09:01:44,006 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:44,006 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) started 2016-08-25 09:01:44,006 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:44,006 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:44,006 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-3 ====================================== 56.4. Browse the queue on new master ====================================== ====================================== 56.5. Found ====================================== ====================================== 57.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:44,026 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) is shutting down 2016-08-25 09:01:44,027 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:44,030 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:44,031 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:44,032 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:44,032 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:44,032 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250038 2016-08-25 09:01:44,033 | INFO | Time-limited test | Session: 0x156bec917250038 closed 2016-08-25 09:01:44,033 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:44,033 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55351 which had sessionid 0x156bec917250038 2016-08-25 09:01:44,138 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:44,139 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) uptime 3.759 seconds 2016-08-25 09:01:44,139 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) is shutdown Wait for master to start up... 2016-08-25 09:01:44,141 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:44,144 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3d2bbb19 2016-08-25 09:01:44,144 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:44,145 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:44,145 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:44,145 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55381 2016-08-25 09:01:44,145 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55381 2016-08-25 09:01:44,146 | INFO | SyncThread:0 | Established session 0x156bec91725003c with negotiated timeout 10000 for client /127.0.0.1:55381 2016-08-25 09:01:44,146 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725003c, negotiated timeout = 10000 2016-08-25 09:01:44,155 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:44,373 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:44,373 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:44,373 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:44,375 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:44,669 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:44,673 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:44,677 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:44,881 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55382 2016-08-25 09:01:44,886 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:44,887 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55382 2016-08-25 09:01:44,887 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:44,887 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:44,888 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55382 2016-08-25 09:01:44,888 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:45,096 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:45,110 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:45,110 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:45,112 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:45,112 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:45,112 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:45,112 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:45,113 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:45,113 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:45,113 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:45,113 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000088 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000090.log 2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000088 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000090.log 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:45,128 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/28.86 kb and 1/32 files 2016-08-25 09:01:45,128 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/28.86 kb and 2/32 files 2016-08-25 09:01:45,129 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/28.86 kb and 3/32 files 2016-08-25 09:01:45,129 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/28.86 kb and 4/32 files 2016-08-25 09:01:45,130 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/28.86 kb and 5/32 files 2016-08-25 09:01:45,130 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/28.86 kb and 6/32 files 2016-08-25 09:01:45,131 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/28.86 kb and 7/32 files 2016-08-25 09:01:45,132 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.20/28.86 kb and 8/32 files 2016-08-25 09:01:45,132 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.09/28.86 kb and 9/32 files 2016-08-25 09:01:45,133 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.00/28.86 kb and 10/32 files 2016-08-25 09:01:45,133 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.01/28.86 kb and 11/32 files 2016-08-25 09:01:45,134 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.91/28.86 kb and 12/32 files 2016-08-25 09:01:45,135 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/28.86 kb and 13/32 files 2016-08-25 09:01:45,135 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.76/28.86 kb and 14/32 files 2016-08-25 09:01:45,136 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.66/28.86 kb and 15/32 files 2016-08-25 09:01:45,136 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.56/28.86 kb and 16/32 files 2016-08-25 09:01:45,137 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.46/28.86 kb and 17/32 files 2016-08-25 09:01:45,137 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.36/28.86 kb and 18/32 files 2016-08-25 09:01:45,138 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.26/28.86 kb and 19/32 files 2016-08-25 09:01:45,139 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.16/28.86 kb and 20/32 files 2016-08-25 09:01:45,139 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.05/28.86 kb and 21/32 files 2016-08-25 09:01:45,140 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.95/28.86 kb and 22/32 files 2016-08-25 09:01:45,140 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.85/28.86 kb and 23/32 files 2016-08-25 09:01:45,141 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.75/28.86 kb and 24/32 files 2016-08-25 09:01:45,141 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.65/28.86 kb and 25/32 files 2016-08-25 09:01:45,142 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.55/28.86 kb and 26/32 files 2016-08-25 09:01:45,142 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.45/28.86 kb and 27/32 files 2016-08-25 09:01:45,143 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.35/28.86 kb and 28/32 files 2016-08-25 09:01:45,144 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.25/28.86 kb and 29/32 files 2016-08-25 09:01:45,145 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.77/28.86 kb and 30/32 files 2016-08-25 09:01:45,145 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/28.86 kb and 1/32 files 2016-08-25 09:01:45,145 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/28.86 kb and 2/32 files 2016-08-25 09:01:45,145 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.67/28.86 kb and 31/32 files 2016-08-25 09:01:45,146 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.86/28.86 kb and 32/32 files 2016-08-25 09:01:45,146 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/28.86 kb and 3/32 files 2016-08-25 09:01:45,146 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:45,147 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/28.86 kb and 4/32 files 2016-08-25 09:01:45,148 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/28.86 kb and 5/32 files 2016-08-25 09:01:45,148 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/28.86 kb and 6/32 files 2016-08-25 09:01:45,149 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/28.86 kb and 7/32 files 2016-08-25 09:01:45,150 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.20/28.86 kb and 8/32 files 2016-08-25 09:01:45,150 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/28.86 kb and 9/32 files 2016-08-25 09:01:45,151 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/28.86 kb and 10/32 files 2016-08-25 09:01:45,152 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.90/28.86 kb and 11/32 files 2016-08-25 09:01:45,152 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.79/28.86 kb and 12/32 files 2016-08-25 09:01:45,153 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.74/28.86 kb and 13/32 files 2016-08-25 09:01:45,153 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.64/28.86 kb and 14/32 files 2016-08-25 09:01:45,154 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.54/28.86 kb and 15/32 files 2016-08-25 09:01:45,154 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:45,154 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.44/28.86 kb and 16/32 files 2016-08-25 09:01:45,155 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.46/28.86 kb and 17/32 files 2016-08-25 09:01:45,155 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) is starting 2016-08-25 09:01:45,155 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.36/28.86 kb and 18/32 files 2016-08-25 09:01:45,156 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.88/28.86 kb and 19/32 files 2016-08-25 09:01:45,156 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.78/28.86 kb and 20/32 files 2016-08-25 09:01:45,157 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.68/28.86 kb and 21/32 files 2016-08-25 09:01:45,158 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.58/28.86 kb and 22/32 files 2016-08-25 09:01:45,158 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.48/28.86 kb and 23/32 files 2016-08-25 09:01:45,159 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.66/28.86 kb and 24/32 files 2016-08-25 09:01:45,159 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55387 2016-08-25 09:01:45,159 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.56/28.86 kb and 25/32 files 2016-08-25 09:01:45,159 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:45,160 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) started 2016-08-25 09:01:45,160 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:45,160 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:45,160 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:45,160 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.46/28.86 kb and 26/32 files 2016-08-25 09:01:45,161 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.36/28.86 kb and 27/32 files 2016-08-25 09:01:45,161 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.26/28.86 kb and 28/32 files Master started: node-1 ====================================== 57.4. Browse the queue on new master ====================================== 2016-08-25 09:01:45,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.16/28.86 kb and 29/32 files 2016-08-25 09:01:45,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.06/28.86 kb and 30/32 files 2016-08-25 09:01:45,163 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.96/28.86 kb and 31/32 files 2016-08-25 09:01:45,164 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.86/28.86 kb and 32/32 files 2016-08-25 09:01:45,164 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:45,171 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 57.5. Found ====================================== ====================================== 58.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:45,179 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) is shutting down 2016-08-25 09:01:45,180 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:45,182 | INFO | Starting broker node: node-2-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:01:45,182 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:45,184 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:45,185 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003b 2016-08-25 09:01:45,185 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:45,185 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55374 which had sessionid 0x156bec91725003b 2016-08-25 09:01:45,185 | INFO | Time-limited test | Session: 0x156bec91725003b closed 2016-08-25 09:01:45,186 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:45,189 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:45,189 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:45,189 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:45,189 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:45,304 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:45,305 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) uptime 1.884 seconds 2016-08-25 09:01:45,305 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) is shutdown Wait for master to start up... 2016-08-25 09:01:45,307 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:45,309 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@dabaed5 2016-08-25 09:01:45,310 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:45,310 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:45,311 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:45,311 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55389 2016-08-25 09:01:45,311 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55389 2016-08-25 09:01:45,311 | INFO | SyncThread:0 | Established session 0x156bec91725003d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55389 2016-08-25 09:01:45,312 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003d, negotiated timeout = 10000 2016-08-25 09:01:45,321 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:45,634 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:45,634 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:45,636 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:45,644 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:45,649 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:45,653 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:45,898 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55390 2016-08-25 09:01:45,903 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:45,904 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:45,906 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55390 2016-08-25 09:01:45,906 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:45,907 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55390 2016-08-25 09:01:45,907 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:45,917 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000090.log 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000088 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:46,124 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000090.log 2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000088 2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:46,130 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:46,130 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:46,130 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:46,130 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:46,131 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:46,144 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/28.86 kb and 1/32 files 2016-08-25 09:01:46,145 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/28.86 kb and 2/32 files 2016-08-25 09:01:46,145 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/28.86 kb and 3/32 files 2016-08-25 09:01:46,146 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/28.86 kb and 4/32 files 2016-08-25 09:01:46,147 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.55/28.86 kb and 5/32 files 2016-08-25 09:01:46,147 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.45/28.86 kb and 6/32 files 2016-08-25 09:01:46,148 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.35/28.86 kb and 7/32 files 2016-08-25 09:01:46,148 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.25/28.86 kb and 8/32 files 2016-08-25 09:01:46,149 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.14/28.86 kb and 9/32 files 2016-08-25 09:01:46,149 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.05/28.86 kb and 10/32 files 2016-08-25 09:01:46,150 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.95/28.86 kb and 11/32 files 2016-08-25 09:01:46,151 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.85/28.86 kb and 12/32 files 2016-08-25 09:01:46,151 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.75/28.86 kb and 13/32 files 2016-08-25 09:01:46,152 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.65/28.86 kb and 14/32 files 2016-08-25 09:01:46,152 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.66/28.86 kb and 15/32 files 2016-08-25 09:01:46,153 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.56/28.86 kb and 16/32 files 2016-08-25 09:01:46,153 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.46/28.86 kb and 17/32 files 2016-08-25 09:01:46,154 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.35/28.86 kb and 18/32 files 2016-08-25 09:01:46,155 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.25/28.86 kb and 19/32 files 2016-08-25 09:01:46,155 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.15/28.86 kb and 20/32 files 2016-08-25 09:01:46,156 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.05/28.86 kb and 21/32 files 2016-08-25 09:01:46,156 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.95/28.86 kb and 22/32 files 2016-08-25 09:01:46,157 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.13/28.86 kb and 23/32 files 2016-08-25 09:01:46,157 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.03/28.86 kb and 24/32 files 2016-08-25 09:01:46,158 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.56/28.86 kb and 25/32 files 2016-08-25 09:01:46,158 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.46/28.86 kb and 26/32 files 2016-08-25 09:01:46,159 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.36/28.86 kb and 27/32 files 2016-08-25 09:01:46,160 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.26/28.86 kb and 28/32 files 2016-08-25 09:01:46,160 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.16/28.86 kb and 29/32 files 2016-08-25 09:01:46,161 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.06/28.86 kb and 30/32 files 2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.96/28.86 kb and 31/32 files 2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.86/28.86 kb and 32/32 files 2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/28.86 kb and 1/32 files 2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/28.86 kb and 2/32 files 2016-08-25 09:01:46,163 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/28.86 kb and 3/32 files 2016-08-25 09:01:46,163 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/28.86 kb and 4/32 files 2016-08-25 09:01:46,164 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/28.86 kb and 5/32 files 2016-08-25 09:01:46,164 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:46,165 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/28.86 kb and 6/32 files 2016-08-25 09:01:46,165 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/28.86 kb and 7/32 files 2016-08-25 09:01:46,165 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) is starting 2016-08-25 09:01:46,166 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.20/28.86 kb and 8/32 files 2016-08-25 09:01:46,166 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/28.86 kb and 9/32 files 2016-08-25 09:01:46,167 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.05/28.86 kb and 10/32 files 2016-08-25 09:01:46,168 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.95/28.86 kb and 11/32 files 2016-08-25 09:01:46,169 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.85/28.86 kb and 12/32 files 2016-08-25 09:01:46,169 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/28.86 kb and 13/32 files 2016-08-25 09:01:46,169 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55395 2016-08-25 09:01:46,170 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:46,170 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) started 2016-08-25 09:01:46,170 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.64/28.86 kb and 14/32 files 2016-08-25 09:01:46,171 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.54/28.86 kb and 15/32 files 2016-08-25 09:01:46,171 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.44/28.86 kb and 16/32 files 2016-08-25 09:01:46,172 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.34/28.86 kb and 17/32 files 2016-08-25 09:01:46,170 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:46,173 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.24/28.86 kb and 18/32 files 2016-08-25 09:01:46,173 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:46,173 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:46,173 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.14/28.86 kb and 19/32 files 2016-08-25 09:01:46,174 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.15/28.86 kb and 20/32 files 2016-08-25 09:01:46,174 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.05/28.86 kb and 21/32 files Master started: node-2 ====================================== 58.4. Browse the queue on new master ====================================== 2016-08-25 09:01:46,175 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.24/28.86 kb and 22/32 files 2016-08-25 09:01:46,176 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.14/28.86 kb and 23/32 files 2016-08-25 09:01:46,177 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.04/28.86 kb and 24/32 files 2016-08-25 09:01:46,177 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.94/28.86 kb and 25/32 files 2016-08-25 09:01:46,178 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.46/28.86 kb and 26/32 files 2016-08-25 09:01:46,179 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.36/28.86 kb and 27/32 files 2016-08-25 09:01:46,179 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.26/28.86 kb and 28/32 files 2016-08-25 09:01:46,180 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.16/28.86 kb and 29/32 files 2016-08-25 09:01:46,180 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.06/28.86 kb and 30/32 files 2016-08-25 09:01:46,180 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.96/28.86 kb and 31/32 files 2016-08-25 09:01:46,181 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.86/28.86 kb and 32/32 files 2016-08-25 09:01:46,181 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:46,187 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 58.5. Found ====================================== ====================================== 59.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:46,194 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) is shutting down 2016-08-25 09:01:46,194 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:46,197 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:46,198 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:46,199 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:46,199 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:46,199 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:46,199 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:46,200 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003a 2016-08-25 09:01:46,201 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55366 which had sessionid 0x156bec91725003a 2016-08-25 09:01:46,202 | INFO | Time-limited test | Session: 0x156bec91725003a closed 2016-08-25 09:01:46,202 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:46,307 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:46,307 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) uptime 4.008 seconds 2016-08-25 09:01:46,307 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) is shutdown Wait for master to start up... 2016-08-25 09:01:46,310 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:46,312 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5f238a57 2016-08-25 09:01:46,312 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:46,313 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:46,313 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:46,313 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55397 2016-08-25 09:01:46,314 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55397 2016-08-25 09:01:46,314 | INFO | SyncThread:0 | Established session 0x156bec91725003e with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55397 2016-08-25 09:01:46,315 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003e, negotiated timeout = 10000 2016-08-25 09:01:46,322 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:46,901 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:46,904 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:46,914 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:46,914 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:46,917 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:46,920 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:47,179 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55398 2016-08-25 09:01:47,186 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:47,186 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:47,187 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55398 2016-08-25 09:01:47,187 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:47,190 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55398 2016-08-25 09:01:47,191 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:47,418 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:47,426 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000091 2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000093.log 2016-08-25 09:01:47,430 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000091 2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000093.log 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:47,447 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/29.81 kb and 1/33 files 2016-08-25 09:01:47,448 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/29.81 kb and 2/33 files 2016-08-25 09:01:47,448 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/29.81 kb and 3/33 files 2016-08-25 09:01:47,449 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/29.81 kb and 4/33 files 2016-08-25 09:01:47,449 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/29.81 kb and 5/33 files 2016-08-25 09:01:47,450 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/29.81 kb and 6/33 files 2016-08-25 09:01:47,451 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/29.81 kb and 7/33 files 2016-08-25 09:01:47,451 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.20/29.81 kb and 8/33 files 2016-08-25 09:01:47,452 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.10/29.81 kb and 9/33 files 2016-08-25 09:01:47,452 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.00/29.81 kb and 10/33 files 2016-08-25 09:01:47,453 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.90/29.81 kb and 11/33 files 2016-08-25 09:01:47,453 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.80/29.81 kb and 12/33 files 2016-08-25 09:01:47,454 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.70/29.81 kb and 13/33 files 2016-08-25 09:01:47,454 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.60/29.81 kb and 14/33 files 2016-08-25 09:01:47,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.49/29.81 kb and 15/33 files 2016-08-25 09:01:47,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.39/29.81 kb and 16/33 files 2016-08-25 09:01:47,456 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.29/29.81 kb and 17/33 files 2016-08-25 09:01:47,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.24/29.81 kb and 18/33 files 2016-08-25 09:01:47,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.14/29.81 kb and 19/33 files 2016-08-25 09:01:47,458 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.04/29.81 kb and 20/33 files 2016-08-25 09:01:47,458 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.62/29.81 kb and 21/33 files 2016-08-25 09:01:47,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.52/29.81 kb and 22/33 files 2016-08-25 09:01:47,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.42/29.81 kb and 23/33 files 2016-08-25 09:01:47,460 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.31/29.81 kb and 24/33 files 2016-08-25 09:01:47,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.21/29.81 kb and 25/33 files 2016-08-25 09:01:47,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.11/29.81 kb and 26/33 files 2016-08-25 09:01:47,462 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.13/29.81 kb and 27/33 files 2016-08-25 09:01:47,462 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.03/29.81 kb and 28/33 files 2016-08-25 09:01:47,463 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.93/29.81 kb and 29/33 files 2016-08-25 09:01:47,463 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.83/29.81 kb and 30/33 files 2016-08-25 09:01:47,464 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.73/29.81 kb and 31/33 files 2016-08-25 09:01:47,464 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.62/29.81 kb and 32/33 files 2016-08-25 09:01:47,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.95/29.81 kb and 1/33 files 2016-08-25 09:01:47,465 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.81/29.81 kb and 33/33 files 2016-08-25 09:01:47,465 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:47,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/29.81 kb and 2/33 files 2016-08-25 09:01:47,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.75/29.81 kb and 3/33 files 2016-08-25 09:01:47,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.64/29.81 kb and 4/33 files 2016-08-25 09:01:47,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/29.81 kb and 5/33 files 2016-08-25 09:01:47,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.44/29.81 kb and 6/33 files 2016-08-25 09:01:47,468 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.34/29.81 kb and 7/33 files 2016-08-25 09:01:47,468 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.24/29.81 kb and 8/33 files 2016-08-25 09:01:47,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.14/29.81 kb and 9/33 files 2016-08-25 09:01:47,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.04/29.81 kb and 10/33 files 2016-08-25 09:01:47,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.94/29.81 kb and 11/33 files 2016-08-25 09:01:47,470 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:47,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.83/29.81 kb and 12/33 files 2016-08-25 09:01:47,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.73/29.81 kb and 13/33 files 2016-08-25 09:01:47,471 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) is starting 2016-08-25 09:01:47,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.64/29.81 kb and 14/33 files 2016-08-25 09:01:47,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.54/29.81 kb and 15/33 files 2016-08-25 09:01:47,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.44/29.81 kb and 16/33 files 2016-08-25 09:01:47,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.34/29.81 kb and 17/33 files 2016-08-25 09:01:47,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.24/29.81 kb and 18/33 files 2016-08-25 09:01:47,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.14/29.81 kb and 19/33 files 2016-08-25 09:01:47,476 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.04/29.81 kb and 20/33 files 2016-08-25 09:01:47,476 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55403 2016-08-25 09:01:47,477 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:47,477 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.93/29.81 kb and 21/33 files 2016-08-25 09:01:47,477 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) started 2016-08-25 09:01:47,477 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:47,477 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:47,477 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:47,478 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.83/29.81 kb and 22/33 files 2016-08-25 09:01:47,478 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.41/29.81 kb and 23/33 files 2016-08-25 09:01:47,478 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.31/29.81 kb and 24/33 files Master started: node-3 ====================================== 59.4. Browse the queue on new master ====================================== 2016-08-25 09:01:47,479 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.21/29.81 kb and 25/33 files 2016-08-25 09:01:47,480 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.11/29.81 kb and 26/33 files 2016-08-25 09:01:47,480 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.01/29.81 kb and 27/33 files 2016-08-25 09:01:47,481 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.91/29.81 kb and 28/33 files 2016-08-25 09:01:47,482 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.81/29.81 kb and 29/33 files 2016-08-25 09:01:47,482 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.99/29.81 kb and 30/33 files 2016-08-25 09:01:47,483 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.89/29.81 kb and 31/33 files 2016-08-25 09:01:47,484 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.79/29.81 kb and 32/33 files 2016-08-25 09:01:47,484 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.81/29.81 kb and 33/33 files 2016-08-25 09:01:47,484 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:47,492 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d ====================================== 59.5. Found ====================================== ====================================== 60.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:47,497 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) is shutting down 2016-08-25 09:01:47,498 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:47,501 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:47,501 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:47,502 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003c 2016-08-25 09:01:47,502 | INFO | Time-limited test | Session: 0x156bec91725003c closed 2016-08-25 09:01:47,502 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:47,502 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55381 which had sessionid 0x156bec91725003c 2016-08-25 09:01:47,505 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:47,505 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:47,505 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:47,506 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:47,622 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:47,623 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) uptime 3.484 seconds 2016-08-25 09:01:47,623 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) is shutdown Wait for master to start up... 2016-08-25 09:01:47,625 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:47,628 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@369fb2ed 2016-08-25 09:01:47,628 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:47,629 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:47,630 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:47,630 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55405 2016-08-25 09:01:47,630 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55405 2016-08-25 09:01:47,631 | INFO | SyncThread:0 | Established session 0x156bec91725003f with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55405 2016-08-25 09:01:47,631 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003f, negotiated timeout = 10000 2016-08-25 09:01:47,638 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:47,947 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:47,950 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:47,969 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:47,970 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:47,973 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:47,976 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:48,204 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55406 2016-08-25 09:01:48,209 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:48,209 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:48,210 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55406 2016-08-25 09:01:48,210 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:48,213 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55406 2016-08-25 09:01:48,213 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:48,223 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:48,384 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:48,384 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000093.log 2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000091 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000091 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000093.log 2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:48,401 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/29.81 kb and 1/33 files 2016-08-25 09:01:48,401 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/29.81 kb and 2/33 files 2016-08-25 09:01:48,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/29.81 kb and 3/33 files 2016-08-25 09:01:48,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/29.81 kb and 4/33 files 2016-08-25 09:01:48,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/29.81 kb and 5/33 files 2016-08-25 09:01:48,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/29.81 kb and 6/33 files 2016-08-25 09:01:48,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.29/29.81 kb and 7/33 files 2016-08-25 09:01:48,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.19/29.81 kb and 8/33 files 2016-08-25 09:01:48,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.09/29.81 kb and 9/33 files 2016-08-25 09:01:48,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.28/29.81 kb and 10/33 files 2016-08-25 09:01:48,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.18/29.81 kb and 11/33 files 2016-08-25 09:01:48,407 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.08/29.81 kb and 12/33 files 2016-08-25 09:01:48,407 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.09/29.81 kb and 13/33 files 2016-08-25 09:01:48,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.99/29.81 kb and 14/33 files 2016-08-25 09:01:48,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.89/29.81 kb and 15/33 files 2016-08-25 09:01:48,409 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.79/29.81 kb and 16/33 files 2016-08-25 09:01:48,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.69/29.81 kb and 17/33 files 2016-08-25 09:01:48,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.59/29.81 kb and 18/33 files 2016-08-25 09:01:48,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.54/29.81 kb and 19/33 files 2016-08-25 09:01:48,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.44/29.81 kb and 20/33 files 2016-08-25 09:01:48,412 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.02/29.81 kb and 21/33 files 2016-08-25 09:01:48,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.92/29.81 kb and 22/33 files 2016-08-25 09:01:48,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.82/29.81 kb and 23/33 files 2016-08-25 09:01:48,414 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.71/29.81 kb and 24/33 files 2016-08-25 09:01:48,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.61/29.81 kb and 25/33 files 2016-08-25 09:01:48,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.51/29.81 kb and 26/33 files 2016-08-25 09:01:48,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.41/29.81 kb and 27/33 files 2016-08-25 09:01:48,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.31/29.81 kb and 28/33 files 2016-08-25 09:01:48,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.21/29.81 kb and 29/33 files 2016-08-25 09:01:48,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.11/29.81 kb and 30/33 files 2016-08-25 09:01:48,418 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.01/29.81 kb and 31/33 files 2016-08-25 09:01:48,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.91/29.81 kb and 32/33 files 2016-08-25 09:01:48,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.81/29.81 kb and 33/33 files 2016-08-25 09:01:48,419 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:48,422 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:48,424 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) is starting 2016-08-25 09:01:48,428 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55411 2016-08-25 09:01:48,428 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:48,428 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) started 2016-08-25 09:01:48,428 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:48,428 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:48,428 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:48,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/29.81 kb and 1/33 files 2016-08-25 09:01:48,434 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/29.81 kb and 2/33 files 2016-08-25 09:01:48,434 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/29.81 kb and 3/33 files Master started: node-1 ====================================== 60.4. Browse the queue on new master ====================================== 2016-08-25 09:01:48,435 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/29.81 kb and 4/33 files 2016-08-25 09:01:48,436 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/29.81 kb and 5/33 files 2016-08-25 09:01:48,436 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/29.81 kb and 6/33 files 2016-08-25 09:01:48,437 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.97/29.81 kb and 7/33 files 2016-08-25 09:01:48,437 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.87/29.81 kb and 8/33 files 2016-08-25 09:01:48,438 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.77/29.81 kb and 9/33 files 2016-08-25 09:01:48,439 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.67/29.81 kb and 10/33 files 2016-08-25 09:01:48,440 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.57/29.81 kb and 11/33 files 2016-08-25 09:01:48,441 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.47/29.81 kb and 12/33 files 2016-08-25 09:01:48,442 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.37/29.81 kb and 13/33 files 2016-08-25 09:01:48,442 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.27/29.81 kb and 14/33 files 2016-08-25 09:01:48,443 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.17/29.81 kb and 15/33 files 2016-08-25 09:01:48,444 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.18/29.81 kb and 16/33 files 2016-08-25 09:01:48,445 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.08/29.81 kb and 17/33 files 2016-08-25 09:01:48,445 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.98/29.81 kb and 18/33 files 2016-08-25 09:01:48,446 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.88/29.81 kb and 19/33 files 2016-08-25 09:01:48,446 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.78/29.81 kb and 20/33 files 2016-08-25 09:01:48,446 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.68/29.81 kb and 21/33 files 2016-08-25 09:01:48,447 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.58/29.81 kb and 22/33 files 2016-08-25 09:01:48,448 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.48/29.81 kb and 23/33 files 2016-08-25 09:01:48,448 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.38/29.81 kb and 24/33 files 2016-08-25 09:01:48,449 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.28/29.81 kb and 25/33 files 2016-08-25 09:01:48,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.18/29.81 kb and 26/33 files 2016-08-25 09:01:48,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.13/29.81 kb and 27/33 files 2016-08-25 09:01:48,451 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.31/29.81 kb and 28/33 files 2016-08-25 09:01:48,452 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.21/29.81 kb and 29/33 files 2016-08-25 09:01:48,453 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.11/29.81 kb and 30/33 files 2016-08-25 09:01:48,454 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.01/29.81 kb and 31/33 files 2016-08-25 09:01:48,455 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.91/29.81 kb and 32/33 files ====================================== 2016-08-25 09:01:48,456 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.81/29.81 kb and 33/33 files 60.5. Found ====================================== ====================================== 61.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:48,456 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:48,456 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) is shutting down 2016-08-25 09:01:48,457 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:48,461 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:48,461 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:48,462 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:48,463 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:48,463 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003d 2016-08-25 09:01:48,464 | INFO | Time-limited test | Session: 0x156bec91725003d closed 2016-08-25 09:01:48,464 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55389 which had sessionid 0x156bec91725003d 2016-08-25 09:01:48,464 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:48,468 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:48,469 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:48,568 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:48,569 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) uptime 3.264 seconds 2016-08-25 09:01:48,569 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) is shutdown Wait for master to start up... 2016-08-25 09:01:48,572 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:48,577 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3ec6fd33 2016-08-25 09:01:48,577 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:48,578 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:48,579 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:48,579 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55413 2016-08-25 09:01:48,579 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55413 2016-08-25 09:01:48,580 | INFO | SyncThread:0 | Established session 0x156bec917250040 with negotiated timeout 10000 for client /127.0.0.1:55413 2016-08-25 09:01:48,580 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250040, negotiated timeout = 10000 2016-08-25 09:01:48,588 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:49,116 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:49,118 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:49,126 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:49,126 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:49,129 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:49,131 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:49,341 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55414 2016-08-25 09:01:49,347 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:49,348 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:49,349 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55414 2016-08-25 09:01:49,349 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:49,351 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55414 2016-08-25 09:01:49,351 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:49,582 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:49,586 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:49,586 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:49,586 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000096.log 2016-08-25 09:01:49,586 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000095.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000094 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:49,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:49,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:49,592 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:49,606 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/30.76 kb and 1/34 files 2016-08-25 09:01:49,607 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.09/30.76 kb and 2/34 files 2016-08-25 09:01:49,607 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.99/30.76 kb and 3/34 files 2016-08-25 09:01:49,608 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.89/30.76 kb and 4/34 files 2016-08-25 09:01:49,608 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.79/30.76 kb and 5/34 files 2016-08-25 09:01:49,609 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.80/30.76 kb and 6/34 files 2016-08-25 09:01:49,609 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.70/30.76 kb and 7/34 files 2016-08-25 09:01:49,610 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.60/30.76 kb and 8/34 files 2016-08-25 09:01:49,610 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.50/30.76 kb and 9/34 files 2016-08-25 09:01:49,611 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.40/30.76 kb and 10/34 files 2016-08-25 09:01:49,611 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.30/30.76 kb and 11/34 files 2016-08-25 09:01:49,611 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.20/30.76 kb and 12/34 files 2016-08-25 09:01:49,611 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.10/30.76 kb and 13/34 files 2016-08-25 09:01:49,612 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.00/30.76 kb and 14/34 files 2016-08-25 09:01:49,612 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.90/30.76 kb and 15/34 files 2016-08-25 09:01:49,612 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.80/30.76 kb and 16/34 files 2016-08-25 09:01:49,613 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.69/30.76 kb and 17/34 files 2016-08-25 09:01:49,613 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.59/30.76 kb and 18/34 files 2016-08-25 09:01:49,613 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.49/30.76 kb and 19/34 files 2016-08-25 09:01:49,613 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.39/30.76 kb and 20/34 files 2016-08-25 09:01:49,614 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.29/30.76 kb and 21/34 files 2016-08-25 09:01:49,614 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.19/30.76 kb and 22/34 files 2016-08-25 09:01:49,615 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.09/30.76 kb and 23/34 files 2016-08-25 09:01:49,615 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.99/30.76 kb and 24/34 files 2016-08-25 09:01:49,615 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.62/30.76 kb and 25/34 files 2016-08-25 09:01:49,616 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 23.57/30.76 kb and 26/34 files 2016-08-25 09:01:49,616 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 24.47/30.76 kb and 27/34 files 2016-08-25 09:01:49,617 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 25.37/30.76 kb and 28/34 files 2016-08-25 09:01:49,617 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 26.27/30.76 kb and 29/34 files 2016-08-25 09:01:49,617 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 27.17/30.76 kb and 30/34 files 2016-08-25 09:01:49,617 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 28.06/30.76 kb and 31/34 files 2016-08-25 09:01:49,618 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 28.96/30.76 kb and 32/34 files 2016-08-25 09:01:49,618 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 29.86/30.76 kb and 33/34 files 2016-08-25 09:01:49,618 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 30.76/30.76 kb and 34/34 files 2016-08-25 09:01:49,619 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:49,622 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:49,623 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:49,623 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) is starting 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000094 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000096.log 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000095.sst 2016-08-25 09:01:49,627 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55419 2016-08-25 09:01:49,628 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:49,628 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) started 2016-08-25 09:01:49,628 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:49,628 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:49,628 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-2 ====================================== 61.4. Browse the queue on new master ====================================== ====================================== 61.5. Found ====================================== ====================================== 62.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:49,642 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) is shutting down 2016-08-25 09:01:49,642 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:49,644 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:49,645 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:49,645 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003e 2016-08-25 09:01:49,645 | INFO | Time-limited test | Session: 0x156bec91725003e closed 2016-08-25 09:01:49,645 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:49,646 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55397 which had sessionid 0x156bec91725003e 2016-08-25 09:01:49,648 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:49,648 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:49,766 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:49,766 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) uptime 3.458 seconds 2016-08-25 09:01:49,766 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) is shutdown Wait for master to start up... 2016-08-25 09:01:49,768 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:49,771 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@39584da9 2016-08-25 09:01:49,771 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:49,771 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:49,772 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:49,772 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55421 2016-08-25 09:01:49,772 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55421 2016-08-25 09:01:49,773 | INFO | SyncThread:0 | Established session 0x156bec917250041 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55421 2016-08-25 09:01:49,773 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250041, negotiated timeout = 10000 2016-08-25 09:01:49,778 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:49,889 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:49,889 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:49,889 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:49,889 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:49,891 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:50,122 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:50,125 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:50,130 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:50,383 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55422 2016-08-25 09:01:50,388 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:50,388 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:50,388 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55422 2016-08-25 09:01:50,388 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:50,389 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55422 2016-08-25 09:01:50,389 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:50,409 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:50,613 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:50,613 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:50,613 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000096.log 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000095.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000094 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:50,616 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000096.log 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000094 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000095.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:50,620 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:50,620 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:50,643 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/30.76 kb and 1/34 files 2016-08-25 09:01:50,644 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/30.76 kb and 2/34 files 2016-08-25 09:01:50,644 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/30.76 kb and 1/34 files 2016-08-25 09:01:50,645 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/30.76 kb and 2/34 files 2016-08-25 09:01:50,645 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/30.76 kb and 3/34 files 2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/30.76 kb and 3/34 files 2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/30.76 kb and 4/34 files 2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.64/30.76 kb and 4/34 files 2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.78/30.76 kb and 5/34 files 2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.54/30.76 kb and 5/34 files 2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.68/30.76 kb and 6/34 files 2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.44/30.76 kb and 6/34 files 2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.58/30.76 kb and 7/34 files 2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.63/30.76 kb and 7/34 files 2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/30.76 kb and 8/34 files 2016-08-25 09:01:50,648 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.53/30.76 kb and 8/34 files 2016-08-25 09:01:50,648 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.38/30.76 kb and 9/34 files 2016-08-25 09:01:50,648 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.54/30.76 kb and 9/34 files 2016-08-25 09:01:50,649 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.44/30.76 kb and 10/34 files 2016-08-25 09:01:50,649 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.28/30.76 kb and 10/34 files 2016-08-25 09:01:50,650 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.34/30.76 kb and 11/34 files 2016-08-25 09:01:50,650 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.18/30.76 kb and 11/34 files 2016-08-25 09:01:50,650 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.24/30.76 kb and 12/34 files 2016-08-25 09:01:50,651 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.08/30.76 kb and 12/34 files 2016-08-25 09:01:50,651 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.87/30.76 kb and 13/34 files 2016-08-25 09:01:50,651 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.98/30.76 kb and 13/34 files 2016-08-25 09:01:50,651 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.77/30.76 kb and 14/34 files 2016-08-25 09:01:50,652 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.67/30.76 kb and 15/34 files 2016-08-25 09:01:50,652 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.88/30.76 kb and 14/34 files 2016-08-25 09:01:50,653 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.57/30.76 kb and 16/34 files 2016-08-25 09:01:50,653 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.78/30.76 kb and 15/34 files 2016-08-25 09:01:50,653 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.68/30.76 kb and 16/34 files 2016-08-25 09:01:50,653 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.47/30.76 kb and 17/34 files 2016-08-25 09:01:50,654 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.37/30.76 kb and 18/34 files 2016-08-25 09:01:50,654 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.58/30.76 kb and 17/34 files 2016-08-25 09:01:50,654 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.27/30.76 kb and 19/34 files 2016-08-25 09:01:50,655 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.48/30.76 kb and 18/34 files 2016-08-25 09:01:50,655 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.16/30.76 kb and 20/34 files 2016-08-25 09:01:50,655 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.37/30.76 kb and 19/34 files 2016-08-25 09:01:50,655 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.06/30.76 kb and 21/34 files 2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.27/30.76 kb and 20/34 files 2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.96/30.76 kb and 22/34 files 2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.17/30.76 kb and 21/34 files 2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.07/30.76 kb and 22/34 files 2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.86/30.76 kb and 23/34 files 2016-08-25 09:01:50,657 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.97/30.76 kb and 23/34 files 2016-08-25 09:01:50,657 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.76/30.76 kb and 24/34 files 2016-08-25 09:01:50,657 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 22.66/30.76 kb and 25/34 files 2016-08-25 09:01:50,657 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.87/30.76 kb and 24/34 files 2016-08-25 09:01:50,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 23.56/30.76 kb and 26/34 files 2016-08-25 09:01:50,658 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.77/30.76 kb and 25/34 files 2016-08-25 09:01:50,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 24.46/30.76 kb and 27/34 files 2016-08-25 09:01:50,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 25.36/30.76 kb and 28/34 files 2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.40/30.76 kb and 26/34 files 2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 26.26/30.76 kb and 29/34 files 2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.30/30.76 kb and 27/34 files 2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 27.16/30.76 kb and 30/34 files 2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.20/30.76 kb and 28/34 files 2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 28.06/30.76 kb and 31/34 files 2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 28.96/30.76 kb and 32/34 files 2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.21/30.76 kb and 29/34 files 2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 29.86/30.76 kb and 33/34 files 2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.11/30.76 kb and 30/34 files 2016-08-25 09:01:50,661 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 30.76/30.76 kb and 34/34 files 2016-08-25 09:01:50,661 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:50,661 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.01/30.76 kb and 31/34 files 2016-08-25 09:01:50,662 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.91/30.76 kb and 32/34 files 2016-08-25 09:01:50,662 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.81/30.76 kb and 33/34 files 2016-08-25 09:01:50,663 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.76/30.76 kb and 34/34 files 2016-08-25 09:01:50,664 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:50,667 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:50,667 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:50,668 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) is starting 2016-08-25 09:01:50,671 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55427 2016-08-25 09:01:50,671 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:50,671 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) started 2016-08-25 09:01:50,672 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:50,672 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:50,672 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-1 ====================================== 62.4. Browse the queue on new master ====================================== ====================================== 62.5. Found ====================================== ====================================== 63.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:50,686 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) is shutting down 2016-08-25 09:01:50,687 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:50,688 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:50,689 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:01:50,689 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:50,689 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250040 2016-08-25 09:01:50,689 | INFO | Time-limited test | Session: 0x156bec917250040 closed 2016-08-25 09:01:50,689 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:50,689 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55413 which had sessionid 0x156bec917250040 2016-08-25 09:01:50,691 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:50,694 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:50,694 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:50,695 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:50,695 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:50,793 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:50,793 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) uptime 2.224 seconds 2016-08-25 09:01:50,794 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) is shutdown Wait for master to start up... 2016-08-25 09:01:50,795 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:50,798 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4736c6df 2016-08-25 09:01:50,798 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:50,799 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:50,799 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:50,799 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55429 2016-08-25 09:01:50,799 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55429 2016-08-25 09:01:50,800 | INFO | SyncThread:0 | Established session 0x156bec917250042 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55429 2016-08-25 09:01:50,800 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250042, negotiated timeout = 10000 2016-08-25 09:01:50,806 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:51,375 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:51,376 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:51,378 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:51,379 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:51,383 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:51,386 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:51,626 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55430 2016-08-25 09:01:51,631 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:51,632 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55430 2016-08-25 09:01:51,632 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:51,633 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:51,638 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55430 2016-08-25 09:01:51,638 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:51,870 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:51,882 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:51,882 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000099.log 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000095.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000098.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000097 2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000099.log 2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000098.sst 2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000097 2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000095.sst 2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:51,910 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/31.71 kb and 1/35 files 2016-08-25 09:01:51,911 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/31.71 kb and 2/35 files 2016-08-25 09:01:51,911 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/31.71 kb and 3/35 files 2016-08-25 09:01:51,912 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/31.71 kb and 4/35 files 2016-08-25 09:01:51,912 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.78/31.71 kb and 5/35 files 2016-08-25 09:01:51,913 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.68/31.71 kb and 6/35 files 2016-08-25 09:01:51,913 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.58/31.71 kb and 7/35 files 2016-08-25 09:01:51,914 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/31.71 kb and 8/35 files 2016-08-25 09:01:51,915 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.38/31.71 kb and 9/35 files 2016-08-25 09:01:51,915 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.28/31.71 kb and 10/35 files 2016-08-25 09:01:51,916 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.18/31.71 kb and 11/35 files 2016-08-25 09:01:51,916 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.08/31.71 kb and 12/35 files 2016-08-25 09:01:51,917 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.97/31.71 kb and 13/35 files 2016-08-25 09:01:51,917 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.87/31.71 kb and 14/35 files 2016-08-25 09:01:51,918 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.77/31.71 kb and 15/35 files 2016-08-25 09:01:51,918 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.67/31.71 kb and 16/35 files 2016-08-25 09:01:51,919 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.57/31.71 kb and 17/35 files 2016-08-25 09:01:51,919 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.52/31.71 kb and 18/35 files 2016-08-25 09:01:51,920 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.42/31.71 kb and 19/35 files 2016-08-25 09:01:51,920 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.32/31.71 kb and 20/35 files 2016-08-25 09:01:51,921 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.22/31.71 kb and 21/35 files 2016-08-25 09:01:51,921 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.12/31.71 kb and 22/35 files 2016-08-25 09:01:51,922 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.01/31.71 kb and 23/35 files 2016-08-25 09:01:51,922 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.91/31.71 kb and 24/35 files 2016-08-25 09:01:51,923 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.81/31.71 kb and 25/35 files 2016-08-25 09:01:51,923 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.71/31.71 kb and 26/35 files 2016-08-25 09:01:51,923 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.61/31.71 kb and 27/35 files 2016-08-25 09:01:51,924 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.51/31.71 kb and 28/35 files 2016-08-25 09:01:51,924 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.53/31.71 kb and 29/35 files 2016-08-25 09:01:51,925 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.43/31.71 kb and 30/35 files 2016-08-25 09:01:51,926 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.33/31.71 kb and 31/35 files 2016-08-25 09:01:51,926 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.23/31.71 kb and 32/35 files 2016-08-25 09:01:51,927 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.13/31.71 kb and 33/35 files 2016-08-25 09:01:51,927 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/31.71 kb and 1/35 files 2016-08-25 09:01:51,927 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.03/31.71 kb and 34/35 files 2016-08-25 09:01:51,928 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.71/31.71 kb and 35/35 files 2016-08-25 09:01:51,928 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.20/31.71 kb and 2/35 files 2016-08-25 09:01:51,928 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:51,928 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.10/31.71 kb and 3/35 files 2016-08-25 09:01:51,929 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.00/31.71 kb and 4/35 files 2016-08-25 09:01:51,929 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.90/31.71 kb and 5/35 files 2016-08-25 09:01:51,930 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.80/31.71 kb and 6/35 files 2016-08-25 09:01:51,930 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.70/31.71 kb and 7/35 files 2016-08-25 09:01:51,931 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.60/31.71 kb and 8/35 files 2016-08-25 09:01:51,931 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.50/31.71 kb and 9/35 files 2016-08-25 09:01:51,932 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.40/31.71 kb and 10/35 files 2016-08-25 09:01:51,932 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.30/31.71 kb and 11/35 files 2016-08-25 09:01:51,933 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.20/31.71 kb and 12/35 files 2016-08-25 09:01:51,934 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.10/31.71 kb and 13/35 files 2016-08-25 09:01:51,934 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.00/31.71 kb and 14/35 files 2016-08-25 09:01:51,935 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.90/31.71 kb and 15/35 files 2016-08-25 09:01:51,935 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:51,935 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.80/31.71 kb and 16/35 files 2016-08-25 09:01:51,936 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.70/31.71 kb and 17/35 files 2016-08-25 09:01:51,936 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) is starting 2016-08-25 09:01:51,937 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.59/31.71 kb and 18/35 files 2016-08-25 09:01:51,937 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.49/31.71 kb and 19/35 files 2016-08-25 09:01:51,938 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.39/31.71 kb and 20/35 files 2016-08-25 09:01:51,939 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.29/31.71 kb and 21/35 files 2016-08-25 09:01:51,939 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.19/31.71 kb and 22/35 files 2016-08-25 09:01:51,940 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55435 2016-08-25 09:01:51,940 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.09/31.71 kb and 23/35 files 2016-08-25 09:01:51,940 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:51,940 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) started 2016-08-25 09:01:51,940 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:51,940 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:51,940 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:51,941 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.77/31.71 kb and 24/35 files 2016-08-25 09:01:51,941 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.67/31.71 kb and 25/35 files Master started: node-3 ====================================== 63.4. Browse the queue on new master ====================================== 2016-08-25 09:01:51,942 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.56/31.71 kb and 26/35 files 2016-08-25 09:01:51,943 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.46/31.71 kb and 27/35 files 2016-08-25 09:01:51,943 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.36/31.71 kb and 28/35 files 2016-08-25 09:01:51,944 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.26/31.71 kb and 29/35 files 2016-08-25 09:01:51,945 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.16/31.71 kb and 30/35 files 2016-08-25 09:01:51,945 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.06/31.71 kb and 31/35 files 2016-08-25 09:01:51,946 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.96/31.71 kb and 32/35 files 2016-08-25 09:01:51,947 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.86/31.71 kb and 33/35 files 2016-08-25 09:01:51,948 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.81/31.71 kb and 34/35 files 2016-08-25 09:01:51,948 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.71/31.71 kb and 35/35 files 2016-08-25 09:01:51,949 | INFO | hawtdispatch-DEFAULT-1 | Attached ====================================== 63.5. Found ====================================== ====================================== 64.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:51,956 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) is shutting down 2016-08-25 09:01:51,956 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:51,958 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:51,959 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:51,960 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:51,960 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003f 2016-08-25 09:01:51,960 | INFO | Time-limited test | Session: 0x156bec91725003f closed 2016-08-25 09:01:51,961 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:51,961 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55405 which had sessionid 0x156bec91725003f 2016-08-25 09:01:51,965 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:51,965 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:51,965 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:51,965 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:52,079 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:52,079 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) uptime 4.455 seconds 2016-08-25 09:01:52,079 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) is shutdown Wait for master to start up... 2016-08-25 09:01:52,081 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:52,083 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@62247009 2016-08-25 09:01:52,083 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:52,084 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:52,085 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:52,085 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55437 2016-08-25 09:01:52,085 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55437 2016-08-25 09:01:52,086 | INFO | SyncThread:0 | Established session 0x156bec917250043 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55437 2016-08-25 09:01:52,086 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250043, negotiated timeout = 10000 2016-08-25 09:01:52,091 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:52,484 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:52,486 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:52,486 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:52,486 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:52,489 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:52,492 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:52,753 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55438 2016-08-25 09:01:52,758 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:52,758 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:52,759 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55438 2016-08-25 09:01:52,760 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:52,762 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55438 2016-08-25 09:01:52,762 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:52,772 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:52,975 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000097 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000099.log 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000095.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000098.sst 2016-08-25 09:01:52,979 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000097 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000099.log 2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000095.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000098.sst 2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:52,997 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/31.71 kb and 1/35 files 2016-08-25 09:01:52,997 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/31.71 kb and 2/35 files 2016-08-25 09:01:52,998 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/31.71 kb and 3/35 files 2016-08-25 09:01:52,998 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/31.71 kb and 4/35 files 2016-08-25 09:01:52,999 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/31.71 kb and 5/35 files 2016-08-25 09:01:52,999 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/31.71 kb and 6/35 files 2016-08-25 09:01:53,000 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.41/31.71 kb and 7/35 files 2016-08-25 09:01:53,000 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.09/31.71 kb and 8/35 files 2016-08-25 09:01:53,001 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.99/31.71 kb and 9/35 files 2016-08-25 09:01:53,001 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.89/31.71 kb and 10/35 files 2016-08-25 09:01:53,002 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.79/31.71 kb and 11/35 files 2016-08-25 09:01:53,002 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.69/31.71 kb and 12/35 files 2016-08-25 09:01:53,003 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.59/31.71 kb and 13/35 files 2016-08-25 09:01:53,004 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.49/31.71 kb and 14/35 files 2016-08-25 09:01:53,005 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.39/31.71 kb and 15/35 files 2016-08-25 09:01:53,005 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.29/31.71 kb and 16/35 files 2016-08-25 09:01:53,006 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.19/31.71 kb and 17/35 files 2016-08-25 09:01:53,007 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.08/31.71 kb and 18/35 files 2016-08-25 09:01:53,008 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.98/31.71 kb and 19/35 files 2016-08-25 09:01:53,008 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.88/31.71 kb and 20/35 files 2016-08-25 09:01:53,009 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.78/31.71 kb and 21/35 files 2016-08-25 09:01:53,009 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.68/31.71 kb and 22/35 files 2016-08-25 09:01:53,010 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.58/31.71 kb and 23/35 files 2016-08-25 09:01:53,010 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.48/31.71 kb and 24/35 files 2016-08-25 09:01:53,011 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.38/31.71 kb and 25/35 files 2016-08-25 09:01:53,011 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.57/31.71 kb and 26/35 files 2016-08-25 09:01:53,011 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.47/31.71 kb and 27/35 files 2016-08-25 09:01:53,012 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.37/31.71 kb and 28/35 files 2016-08-25 09:01:53,012 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.26/31.71 kb and 29/35 files 2016-08-25 09:01:53,012 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.17/31.71 kb and 30/35 files 2016-08-25 09:01:53,013 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.06/31.71 kb and 31/35 files 2016-08-25 09:01:53,013 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.01/31.71 kb and 32/35 files 2016-08-25 09:01:53,014 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.91/31.71 kb and 33/35 files 2016-08-25 09:01:53,014 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.68/31.71 kb and 1/35 files 2016-08-25 09:01:53,014 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.81/31.71 kb and 34/35 files 2016-08-25 09:01:53,015 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.71/31.71 kb and 35/35 files 2016-08-25 09:01:53,015 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.58/31.71 kb and 2/35 files 2016-08-25 09:01:53,015 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:53,015 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.48/31.71 kb and 3/35 files 2016-08-25 09:01:53,016 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.38/31.71 kb and 4/35 files 2016-08-25 09:01:53,016 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.28/31.71 kb and 5/35 files 2016-08-25 09:01:53,017 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:53,017 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.23/31.71 kb and 6/35 files 2016-08-25 09:01:53,017 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.13/31.71 kb and 7/35 files 2016-08-25 09:01:53,018 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) is starting 2016-08-25 09:01:53,018 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.02/31.71 kb and 8/35 files 2016-08-25 09:01:53,018 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/31.71 kb and 9/35 files 2016-08-25 09:01:53,019 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.82/31.71 kb and 10/35 files 2016-08-25 09:01:53,021 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.00/31.71 kb and 11/35 files 2016-08-25 09:01:53,021 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.90/31.71 kb and 12/35 files 2016-08-25 09:01:53,022 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.80/31.71 kb and 13/35 files 2016-08-25 09:01:53,023 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.70/31.71 kb and 14/35 files 2016-08-25 09:01:53,023 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55443 2016-08-25 09:01:53,023 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:53,023 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.60/31.71 kb and 15/35 files 2016-08-25 09:01:53,024 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) started 2016-08-25 09:01:53,024 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:53,024 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:53,024 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:53,024 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.62/31.71 kb and 16/35 files 2016-08-25 09:01:53,024 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.51/31.71 kb and 17/35 files 2016-08-25 09:01:53,025 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.41/31.71 kb and 18/35 files Master started: node-2 ====================================== 64.4. Browse the queue on new master ====================================== 2016-08-25 09:01:53,026 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.31/31.71 kb and 19/35 files 2016-08-25 09:01:53,026 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.21/31.71 kb and 20/35 files 2016-08-25 09:01:53,027 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.11/31.71 kb and 21/35 files 2016-08-25 09:01:53,028 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.01/31.71 kb and 22/35 files 2016-08-25 09:01:53,029 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.92/31.71 kb and 23/35 files 2016-08-25 09:01:53,029 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.81/31.71 kb and 24/35 files 2016-08-25 09:01:53,030 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.71/31.71 kb and 25/35 files 2016-08-25 09:01:53,030 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.61/31.71 kb and 26/35 files 2016-08-25 09:01:53,031 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.51/31.71 kb and 27/35 files 2016-08-25 09:01:53,032 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.41/31.71 kb and 28/35 files 2016-08-25 09:01:53,033 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.31/31.71 kb and 29/35 files 2016-08-25 09:01:53,034 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.21/31.71 kb and 30/35 files 2016-08-25 09:01:53,035 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.11/31.71 kb and 31/35 files 2016-08-25 09:01:53,040 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.01/31.71 kb and 32/35 files 2016-08-25 09:01:53,041 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.91/31.71 kb and 33/35 files 2016-08-25 09:01:53,042 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.81/31.71 kb and 34/35 files 2016-08-25 09:01:53,042 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.71/31.71 kb and 35/35 files 2016-08-25 09:01:53,042 | INFO | hawtdispatch-DEFAULT-1 | Attached ====================================== 64.5. Found ====================================== ====================================== 65.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:53,043 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) is shutting down 2016-08-25 09:01:53,044 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:53,046 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:53,047 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:53,047 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250041 2016-08-25 09:01:53,047 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:53,047 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:53,047 | INFO | Time-limited test | Session: 0x156bec917250041 closed 2016-08-25 09:01:53,047 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:53,048 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55421 which had sessionid 0x156bec917250041 2016-08-25 09:01:53,055 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:53,056 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:53,150 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:53,151 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) uptime 3.384 seconds 2016-08-25 09:01:53,151 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) is shutdown Wait for master to start up... 2016-08-25 09:01:53,153 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:53,156 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7003f305 2016-08-25 09:01:53,156 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:53,157 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:53,157 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:53,157 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55445 2016-08-25 09:01:53,157 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55445 2016-08-25 09:01:53,158 | INFO | SyncThread:0 | Established session 0x156bec917250044 with negotiated timeout 10000 for client /127.0.0.1:55445 2016-08-25 09:01:53,158 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250044, negotiated timeout = 10000 2016-08-25 09:01:53,166 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:53,772 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:53,776 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:53,796 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:53,796 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:53,799 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:53,803 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:54,029 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55446 2016-08-25 09:01:54,034 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:54,034 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:54,035 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55446 2016-08-25 09:01:54,035 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:54,038 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55446 2016-08-25 09:01:54,038 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:54,235 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000102.log 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000098.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000101.sst 2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000095.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000100 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000101.sst 2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000095.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000102.log 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000098.sst 2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000100 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:54,263 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/32.66 kb and 1/36 files 2016-08-25 09:01:54,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/32.66 kb and 2/36 files 2016-08-25 09:01:54,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/32.66 kb and 3/36 files 2016-08-25 09:01:54,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/32.66 kb and 4/36 files 2016-08-25 09:01:54,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/32.66 kb and 5/36 files 2016-08-25 09:01:54,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/32.66 kb and 6/36 files 2016-08-25 09:01:54,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.58/32.66 kb and 7/36 files 2016-08-25 09:01:54,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/32.66 kb and 8/36 files 2016-08-25 09:01:54,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.38/32.66 kb and 9/36 files 2016-08-25 09:01:54,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.28/32.66 kb and 10/36 files 2016-08-25 09:01:54,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.29/32.66 kb and 11/36 files 2016-08-25 09:01:54,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.19/32.66 kb and 12/36 files 2016-08-25 09:01:54,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.09/32.66 kb and 13/36 files 2016-08-25 09:01:54,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.99/32.66 kb and 14/36 files 2016-08-25 09:01:54,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.89/32.66 kb and 15/36 files 2016-08-25 09:01:54,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.79/32.66 kb and 16/36 files 2016-08-25 09:01:54,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.69/32.66 kb and 17/36 files 2016-08-25 09:01:54,272 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.59/32.66 kb and 18/36 files 2016-08-25 09:01:54,272 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.49/32.66 kb and 19/36 files 2016-08-25 09:01:54,273 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.39/32.66 kb and 20/36 files 2016-08-25 09:01:54,273 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.29/32.66 kb and 21/36 files 2016-08-25 09:01:54,274 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.02/32.66 kb and 22/36 files 2016-08-25 09:01:54,274 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.92/32.66 kb and 23/36 files 2016-08-25 09:01:54,275 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.82/32.66 kb and 24/36 files 2016-08-25 09:01:54,275 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.71/32.66 kb and 25/36 files 2016-08-25 09:01:54,276 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.61/32.66 kb and 26/36 files 2016-08-25 09:01:54,276 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.51/32.66 kb and 27/36 files 2016-08-25 09:01:54,277 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.46/32.66 kb and 28/36 files 2016-08-25 09:01:54,277 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.36/32.66 kb and 29/36 files 2016-08-25 09:01:54,278 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.26/32.66 kb and 30/36 files 2016-08-25 09:01:54,278 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.16/32.66 kb and 31/36 files 2016-08-25 09:01:54,279 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.06/32.66 kb and 32/36 files 2016-08-25 09:01:54,279 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.96/32.66 kb and 33/36 files 2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/32.66 kb and 1/36 files 2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.86/32.66 kb and 34/36 files 2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.76/32.66 kb and 35/36 files 2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/32.66 kb and 2/36 files 2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 32.66/32.66 kb and 36/36 files 2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:54,281 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/32.66 kb and 3/36 files 2016-08-25 09:01:54,281 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.60/32.66 kb and 4/36 files 2016-08-25 09:01:54,281 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.50/32.66 kb and 5/36 files 2016-08-25 09:01:54,282 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.39/32.66 kb and 6/36 files 2016-08-25 09:01:54,282 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.29/32.66 kb and 7/36 files 2016-08-25 09:01:54,283 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.19/32.66 kb and 8/36 files 2016-08-25 09:01:54,284 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.09/32.66 kb and 9/36 files 2016-08-25 09:01:54,284 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.04/32.66 kb and 10/36 files 2016-08-25 09:01:54,285 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.94/32.66 kb and 11/36 files 2016-08-25 09:01:54,286 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.84/32.66 kb and 12/36 files 2016-08-25 09:01:54,286 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.74/32.66 kb and 13/36 files 2016-08-25 09:01:54,287 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.64/32.66 kb and 14/36 files 2016-08-25 09:01:54,287 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.54/32.66 kb and 15/36 files 2016-08-25 09:01:54,288 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.44/32.66 kb and 16/36 files 2016-08-25 09:01:54,288 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.34/32.66 kb and 17/36 files 2016-08-25 09:01:54,289 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.24/32.66 kb and 18/36 files 2016-08-25 09:01:54,289 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.14/32.66 kb and 19/36 files 2016-08-25 09:01:54,290 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.32/32.66 kb and 20/36 files 2016-08-25 09:01:54,291 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.22/32.66 kb and 21/36 files 2016-08-25 09:01:54,291 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.12/32.66 kb and 22/36 files 2016-08-25 09:01:54,292 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.02/32.66 kb and 23/36 files 2016-08-25 09:01:54,293 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 22.75/32.66 kb and 24/36 files 2016-08-25 09:01:54,293 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 23.65/32.66 kb and 25/36 files 2016-08-25 09:01:54,294 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 24.55/32.66 kb and 26/36 files 2016-08-25 09:01:54,295 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 25.45/32.66 kb and 27/36 files 2016-08-25 09:01:54,295 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 26.35/32.66 kb and 28/36 files 2016-08-25 09:01:54,296 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 27.25/32.66 kb and 29/36 files 2016-08-25 09:01:54,296 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 28.15/32.66 kb and 30/36 files 2016-08-25 09:01:54,297 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 28.16/32.66 kb and 31/36 files 2016-08-25 09:01:54,297 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 29.06/32.66 kb and 32/36 files 2016-08-25 09:01:54,298 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 29.96/32.66 kb and 33/36 files 2016-08-25 09:01:54,299 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 30.86/32.66 kb and 34/36 files 2016-08-25 09:01:54,299 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 31.76/32.66 kb and 35/36 files 2016-08-25 09:01:54,300 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 32.66/32.66 kb and 36/36 files 2016-08-25 09:01:54,300 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:01:54,305 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:54,305 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:54,306 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) is starting 2016-08-25 09:01:54,310 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55451 2016-08-25 09:01:54,310 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:54,311 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) started 2016-08-25 09:01:54,311 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:54,311 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:54,311 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-1 ====================================== 65.4. Browse the queue on new master ====================================== ====================================== 65.5. Found ====================================== ====================================== 66.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:54,333 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) is shutting down 2016-08-25 09:01:54,334 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:54,338 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:54,339 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:54,339 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:54,339 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:54,339 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:54,340 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:54,340 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250042 2016-08-25 09:01:54,341 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55429 which had sessionid 0x156bec917250042 2016-08-25 09:01:54,341 | INFO | Time-limited test | Session: 0x156bec917250042 closed 2016-08-25 09:01:54,341 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:54,463 | INFO | ActiveMQ BrokerService[node-1] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:54,464 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) uptime 3.669 seconds 2016-08-25 09:01:54,464 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) is shutdown Wait for master to start up... 2016-08-25 09:01:54,466 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:54,469 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@17161429 2016-08-25 09:01:54,469 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:54,470 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:54,470 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:54,471 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55453 2016-08-25 09:01:54,471 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55453 2016-08-25 09:01:54,472 | INFO | SyncThread:0 | Established session 0x156bec917250045 with negotiated timeout 10000 for client /127.0.0.1:55453 2016-08-25 09:01:54,472 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250045, negotiated timeout = 10000 2016-08-25 09:01:54,487 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:54,797 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:54,798 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:54,808 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:54,808 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:54,812 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:54,817 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:55,036 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55454 2016-08-25 09:01:55,041 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:55,041 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:55,043 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55454 2016-08-25 09:01:55,043 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:55,047 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55454 2016-08-25 09:01:55,047 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:55,058 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:55,224 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:55,224 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000098.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000101.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000095.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000102.log 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000100 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:55,227 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:55,227 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:55,228 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000098.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000101.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000100 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000102.log 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000095.sst 2016-08-25 09:01:55,241 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/32.66 kb and 1/36 files 2016-08-25 09:01:55,241 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/32.66 kb and 2/36 files 2016-08-25 09:01:55,242 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/32.66 kb and 3/36 files 2016-08-25 09:01:55,242 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.60/32.66 kb and 4/36 files 2016-08-25 09:01:55,243 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.50/32.66 kb and 5/36 files 2016-08-25 09:01:55,243 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.40/32.66 kb and 6/36 files 2016-08-25 09:01:55,244 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.30/32.66 kb and 7/36 files 2016-08-25 09:01:55,244 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.20/32.66 kb and 8/36 files 2016-08-25 09:01:55,244 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.09/32.66 kb and 9/36 files 2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/32.66 kb and 1/36 files 2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/32.66 kb and 2/36 files 2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.00/32.66 kb and 10/36 files 2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/32.66 kb and 3/36 files 2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.18/32.66 kb and 11/36 files 2016-08-25 09:01:55,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/32.66 kb and 4/36 files 2016-08-25 09:01:55,246 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.08/32.66 kb and 12/36 files 2016-08-25 09:01:55,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/32.66 kb and 5/36 files 2016-08-25 09:01:55,246 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.98/32.66 kb and 13/36 files 2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/32.66 kb and 6/36 files 2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.88/32.66 kb and 14/36 files 2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/32.66 kb and 7/36 files 2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.78/32.66 kb and 15/36 files 2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.20/32.66 kb and 8/36 files 2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.68/32.66 kb and 16/36 files 2016-08-25 09:01:55,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/32.66 kb and 9/36 files 2016-08-25 09:01:55,248 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.58/32.66 kb and 17/36 files 2016-08-25 09:01:55,248 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.53/32.66 kb and 18/36 files 2016-08-25 09:01:55,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/32.66 kb and 10/36 files 2016-08-25 09:01:55,249 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.42/32.66 kb and 19/36 files 2016-08-25 09:01:55,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.89/32.66 kb and 11/36 files 2016-08-25 09:01:55,249 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.32/32.66 kb and 20/36 files 2016-08-25 09:01:55,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.79/32.66 kb and 12/36 files 2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.34/32.66 kb and 21/36 files 2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.69/32.66 kb and 13/36 files 2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.59/32.66 kb and 14/36 files 2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.24/32.66 kb and 22/36 files 2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.14/32.66 kb and 23/36 files 2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.49/32.66 kb and 15/36 files 2016-08-25 09:01:55,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.39/32.66 kb and 16/36 files 2016-08-25 09:01:55,251 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.04/32.66 kb and 24/36 files 2016-08-25 09:01:55,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.29/32.66 kb and 17/36 files 2016-08-25 09:01:55,251 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.94/32.66 kb and 25/36 files 2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.19/32.66 kb and 18/36 files 2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.84/32.66 kb and 26/36 files 2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.09/32.66 kb and 19/36 files 2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 23.74/32.66 kb and 27/36 files 2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.11/32.66 kb and 20/36 files 2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 24.64/32.66 kb and 28/36 files 2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.00/32.66 kb and 21/36 files 2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 26.37/32.66 kb and 29/36 files 2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.91/32.66 kb and 22/36 files 2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 27.26/32.66 kb and 30/36 files 2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.81/32.66 kb and 23/36 files 2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 28.16/32.66 kb and 31/36 files 2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.71/32.66 kb and 24/36 files 2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 29.06/32.66 kb and 32/36 files 2016-08-25 09:01:55,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.61/32.66 kb and 25/36 files 2016-08-25 09:01:55,254 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 29.96/32.66 kb and 33/36 files 2016-08-25 09:01:55,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.55/32.66 kb and 26/36 files 2016-08-25 09:01:55,254 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 30.86/32.66 kb and 34/36 files 2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 31.76/32.66 kb and 35/36 files 2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.45/32.66 kb and 27/36 files 2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.35/32.66 kb and 28/36 files 2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 32.66/32.66 kb and 36/36 files 2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.08/32.66 kb and 29/36 files 2016-08-25 09:01:55,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.27/32.66 kb and 30/36 files 2016-08-25 09:01:55,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.17/32.66 kb and 31/36 files 2016-08-25 09:01:55,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.06/32.66 kb and 32/36 files 2016-08-25 09:01:55,257 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:55,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.96/32.66 kb and 33/36 files 2016-08-25 09:01:55,258 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.86/32.66 kb and 34/36 files 2016-08-25 09:01:55,258 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) is starting 2016-08-25 09:01:55,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.76/32.66 kb and 35/36 files 2016-08-25 09:01:55,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 32.66/32.66 kb and 36/36 files 2016-08-25 09:01:55,259 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:55,262 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55459 2016-08-25 09:01:55,263 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:55,263 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) started 2016-08-25 09:01:55,263 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:55,263 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:01:55,263 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb Master started: node-3 ====================================== 66.4. Browse the queue on new master ====================================== 2016-08-25 09:01:55,266 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 66.5. Found ====================================== ====================================== 67.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:55,279 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) is shutting down 2016-08-25 09:01:55,280 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:55,285 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:55,286 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:55,287 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250043 2016-08-25 09:01:55,287 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55437 which had sessionid 0x156bec917250043 2016-08-25 09:01:55,288 | INFO | Time-limited test | Session: 0x156bec917250043 closed 2016-08-25 09:01:55,288 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:55,289 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:55,290 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:55,290 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:55,290 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:55,395 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:55,395 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) uptime 3.315 seconds 2016-08-25 09:01:55,396 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) is shutdown Wait for master to start up... 2016-08-25 09:01:55,398 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:55,400 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@29a8186c 2016-08-25 09:01:55,400 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:55,400 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:55,401 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:55,401 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55461 2016-08-25 09:01:55,401 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55461 2016-08-25 09:01:55,402 | INFO | SyncThread:0 | Established session 0x156bec917250046 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55461 2016-08-25 09:01:55,402 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250046, negotiated timeout = 10000 2016-08-25 09:01:55,407 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:55,956 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:55,966 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:55,971 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:55,971 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:55,976 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:55,980 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:56,205 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55462 2016-08-25 09:01:56,211 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:56,211 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:56,212 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55462 2016-08-25 09:01:56,212 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:56,215 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55462 2016-08-25 09:01:56,215 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:56,425 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:56,435 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:56,435 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000095.sst 2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000103 2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000101.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000098.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000105.log 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000104.sst 2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000104.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst 2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000095.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000105.log 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000101.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000103 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst 2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000098.sst 2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst 2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst 2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst 2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst 2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst 2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst 2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:56,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/33.61 kb and 1/37 files 2016-08-25 09:01:56,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/33.61 kb and 2/37 files 2016-08-25 09:01:56,455 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/33.61 kb and 1/37 files 2016-08-25 09:01:56,456 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.58/33.61 kb and 3/37 files 2016-08-25 09:01:56,456 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/33.61 kb and 2/37 files 2016-08-25 09:01:56,456 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.48/33.61 kb and 4/37 files 2016-08-25 09:01:56,456 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/33.61 kb and 3/37 files 2016-08-25 09:01:56,457 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/33.61 kb and 4/37 files 2016-08-25 09:01:56,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.38/33.61 kb and 5/37 files 2016-08-25 09:01:56,458 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/33.61 kb and 5/37 files 2016-08-25 09:01:56,458 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.28/33.61 kb and 6/37 files 2016-08-25 09:01:56,458 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/33.61 kb and 6/37 files 2016-08-25 09:01:56,459 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/33.61 kb and 7/37 files 2016-08-25 09:01:56,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.18/33.61 kb and 7/37 files 2016-08-25 09:01:56,460 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.20/33.61 kb and 8/37 files 2016-08-25 09:01:56,460 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.08/33.61 kb and 8/37 files 2016-08-25 09:01:56,461 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/33.61 kb and 9/37 files 2016-08-25 09:01:56,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.98/33.61 kb and 9/37 files 2016-08-25 09:01:56,461 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/33.61 kb and 10/37 files 2016-08-25 09:01:56,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.93/33.61 kb and 10/37 files 2016-08-25 09:01:56,462 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.89/33.61 kb and 11/37 files 2016-08-25 09:01:56,462 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.83/33.61 kb and 11/37 files 2016-08-25 09:01:56,463 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.79/33.61 kb and 12/37 files 2016-08-25 09:01:56,463 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.72/33.61 kb and 12/37 files 2016-08-25 09:01:56,463 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.69/33.61 kb and 13/37 files 2016-08-25 09:01:56,464 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.62/33.61 kb and 13/37 files 2016-08-25 09:01:56,464 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.59/33.61 kb and 14/37 files 2016-08-25 09:01:56,464 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.52/33.61 kb and 14/37 files 2016-08-25 09:01:56,465 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.42/33.61 kb and 15/37 files 2016-08-25 09:01:56,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.49/33.61 kb and 15/37 files 2016-08-25 09:01:56,465 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.32/33.61 kb and 16/37 files 2016-08-25 09:01:56,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.67/33.61 kb and 16/37 files 2016-08-25 09:01:56,466 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.22/33.61 kb and 17/37 files 2016-08-25 09:01:56,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.57/33.61 kb and 17/37 files 2016-08-25 09:01:56,466 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.12/33.61 kb and 18/37 files 2016-08-25 09:01:56,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.47/33.61 kb and 18/37 files 2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.02/33.61 kb and 19/37 files 2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.37/33.61 kb and 19/37 files 2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.92/33.61 kb and 20/37 files 2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.27/33.61 kb and 20/37 files 2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.17/33.61 kb and 21/37 files 2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.82/33.61 kb and 21/37 files 2016-08-25 09:01:56,468 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.07/33.61 kb and 22/37 files 2016-08-25 09:01:56,468 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.72/33.61 kb and 22/37 files 2016-08-25 09:01:56,468 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.97/33.61 kb and 23/37 files 2016-08-25 09:01:56,468 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.62/33.61 kb and 23/37 files 2016-08-25 09:01:56,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.87/33.61 kb and 24/37 files 2016-08-25 09:01:56,469 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.63/33.61 kb and 24/37 files 2016-08-25 09:01:56,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.65/33.61 kb and 25/37 files 2016-08-25 09:01:56,469 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.53/33.61 kb and 25/37 files 2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.55/33.61 kb and 26/37 files 2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.43/33.61 kb and 26/37 files 2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.45/33.61 kb and 27/37 files 2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.33/33.61 kb and 27/37 files 2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.35/33.61 kb and 28/37 files 2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.23/33.61 kb and 28/37 files 2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.25/33.61 kb and 29/37 files 2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.13/33.61 kb and 29/37 files 2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.15/33.61 kb and 30/37 files 2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.03/33.61 kb and 30/37 files 2016-08-25 09:01:56,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.05/33.61 kb and 31/37 files 2016-08-25 09:01:56,472 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.93/33.61 kb and 31/37 files 2016-08-25 09:01:56,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.95/33.61 kb and 32/37 files 2016-08-25 09:01:56,472 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.12/33.61 kb and 32/37 files 2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.85/33.61 kb and 33/37 files 2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.01/33.61 kb and 33/37 files 2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.75/33.61 kb and 34/37 files 2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.91/33.61 kb and 34/37 files 2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 32.70/33.61 kb and 35/37 files 2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.81/33.61 kb and 35/37 files 2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 33.59/33.61 kb and 36/37 files 2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 32.71/33.61 kb and 36/37 files 2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 33.61/33.61 kb and 37/37 files 2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 33.61/33.61 kb and 37/37 files 2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:56,480 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:56,481 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) is starting 2016-08-25 09:01:56,483 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:56,485 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55467 2016-08-25 09:01:56,485 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:56,485 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) started 2016-08-25 09:01:56,486 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:56,486 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:01:56,486 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb Master started: node-2 ====================================== 67.4. Browse the queue on new master ====================================== ====================================== 67.5. Found ====================================== ====================================== 68.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:56,501 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) is shutting down 2016-08-25 09:01:56,502 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:56,504 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:56,504 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:56,504 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250044 2016-08-25 09:01:56,505 | INFO | Time-limited test | Session: 0x156bec917250044 closed 2016-08-25 09:01:56,505 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:56,505 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55445 which had sessionid 0x156bec917250044 2016-08-25 09:01:56,508 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:56,508 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:56,508 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:56,508 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:56,621 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:56,622 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) uptime 3.471 seconds 2016-08-25 09:01:56,622 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) is shutdown Wait for master to start up... 2016-08-25 09:01:56,624 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:56,626 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5af33ce2 2016-08-25 09:01:56,627 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:56,627 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:56,628 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:56,628 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55469 2016-08-25 09:01:56,628 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55469 2016-08-25 09:01:56,629 | INFO | SyncThread:0 | Established session 0x156bec917250047 with negotiated timeout 10000 for client /127.0.0.1:55469 2016-08-25 09:01:56,629 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250047, negotiated timeout = 10000 2016-08-25 09:01:56,636 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:57,031 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:57,033 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:57,034 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:57,034 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:57,044 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:57,049 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:57,298 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55470 2016-08-25 09:01:57,303 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:57,304 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:57,306 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55470 2016-08-25 09:01:57,306 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:57,311 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55470 2016-08-25 09:01:57,311 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:57,325 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:57,537 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:57,538 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000101.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000105.log 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000103 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000095.sst 2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000104.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000098.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:57,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000105.log 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000104.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000103 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst 2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000098.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000095.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000101.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst 2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst 2016-08-25 09:01:57,546 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst 2016-08-25 09:01:57,546 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst 2016-08-25 09:01:57,557 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/33.61 kb and 1/37 files 2016-08-25 09:01:57,557 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/33.61 kb and 2/37 files 2016-08-25 09:01:57,558 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/33.61 kb and 3/37 files 2016-08-25 09:01:57,558 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/33.61 kb and 4/37 files 2016-08-25 09:01:57,559 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/33.61 kb and 5/37 files 2016-08-25 09:01:57,559 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/33.61 kb and 6/37 files 2016-08-25 09:01:57,560 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/33.61 kb and 7/37 files 2016-08-25 09:01:57,560 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.20/33.61 kb and 8/37 files 2016-08-25 09:01:57,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.09/33.61 kb and 9/37 files 2016-08-25 09:01:57,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.99/33.61 kb and 10/37 files 2016-08-25 09:01:57,562 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.89/33.61 kb and 11/37 files 2016-08-25 09:01:57,562 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.79/33.61 kb and 12/37 files 2016-08-25 09:01:57,563 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.69/33.61 kb and 13/37 files 2016-08-25 09:01:57,563 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.58/33.61 kb and 14/37 files 2016-08-25 09:01:57,564 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.60/33.61 kb and 15/37 files 2016-08-25 09:01:57,564 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.50/33.61 kb and 16/37 files 2016-08-25 09:01:57,565 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.40/33.61 kb and 17/37 files 2016-08-25 09:01:57,565 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.59/33.61 kb and 18/37 files 2016-08-25 09:01:57,566 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.49/33.61 kb and 19/37 files 2016-08-25 09:01:57,566 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.39/33.61 kb and 20/37 files 2016-08-25 09:01:57,567 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.29/33.61 kb and 21/37 files 2016-08-25 09:01:57,568 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.18/33.61 kb and 22/37 files 2016-08-25 09:01:57,568 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.96/33.61 kb and 23/37 files 2016-08-25 09:01:57,568 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.86/33.61 kb and 24/37 files 2016-08-25 09:01:57,569 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.76/33.61 kb and 25/37 files 2016-08-25 09:01:57,570 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.66/33.61 kb and 26/37 files 2016-08-25 09:01:57,570 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.56/33.61 kb and 27/37 files 2016-08-25 09:01:57,571 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.46/33.61 kb and 28/37 files 2016-08-25 09:01:57,571 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.36/33.61 kb and 29/37 files 2016-08-25 09:01:57,572 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.31/33.61 kb and 30/37 files 2016-08-25 09:01:57,572 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.21/33.61 kb and 31/37 files 2016-08-25 09:01:57,573 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.11/33.61 kb and 32/37 files 2016-08-25 09:01:57,573 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.01/33.61 kb and 33/37 files 2016-08-25 09:01:57,574 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.91/33.61 kb and 34/37 files 2016-08-25 09:01:57,574 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.81/33.61 kb and 35/37 files 2016-08-25 09:01:57,574 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 32.71/33.61 kb and 36/37 files 2016-08-25 09:01:57,575 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 33.61/33.61 kb and 37/37 files 2016-08-25 09:01:57,575 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:57,578 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:57,579 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) is starting 2016-08-25 09:01:57,583 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55475 2016-08-25 09:01:57,583 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:57,583 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) started 2016-08-25 09:01:57,583 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:57,584 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:01:57,584 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:01:57,585 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/33.61 kb and 1/37 files 2016-08-25 09:01:57,585 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/33.61 kb and 2/37 files 2016-08-25 09:01:57,586 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/33.61 kb and 3/37 files 2016-08-25 09:01:57,586 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/33.61 kb and 4/37 files 2016-08-25 09:01:57,587 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/33.61 kb and 5/37 files 2016-08-25 09:01:57,588 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.68/33.61 kb and 6/37 files Master started: node-1 ====================================== 68.4. Browse the queue on new master ====================================== 2016-08-25 09:01:57,588 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/33.61 kb and 7/37 files 2016-08-25 09:01:57,589 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.48/33.61 kb and 8/37 files 2016-08-25 09:01:57,590 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.38/33.61 kb and 9/37 files 2016-08-25 09:01:57,590 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.28/33.61 kb and 10/37 files 2016-08-25 09:01:57,591 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.18/33.61 kb and 11/37 files 2016-08-25 09:01:57,592 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.08/33.61 kb and 12/37 files 2016-08-25 09:01:57,592 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.86/33.61 kb and 13/37 files 2016-08-25 09:01:57,593 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.76/33.61 kb and 14/37 files 2016-08-25 09:01:57,593 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.66/33.61 kb and 15/37 files 2016-08-25 09:01:57,594 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.56/33.61 kb and 16/37 files 2016-08-25 09:01:57,599 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.46/33.61 kb and 17/37 files 2016-08-25 09:01:57,600 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.36/33.61 kb and 18/37 files 2016-08-25 09:01:57,601 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.25/33.61 kb and 19/37 files 2016-08-25 09:01:57,601 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.15/33.61 kb and 20/37 files 2016-08-25 09:01:57,602 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.05/33.61 kb and 21/37 files 2016-08-25 09:01:57,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.07/33.61 kb and 22/37 files 2016-08-25 09:01:57,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.02/33.61 kb and 23/37 files 2016-08-25 09:01:57,604 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.92/33.61 kb and 24/37 files 2016-08-25 09:01:57,605 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.82/33.61 kb and 25/37 files 2016-08-25 09:01:57,605 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.72/33.61 kb and 26/37 files 2016-08-25 09:01:57,606 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.62/33.61 kb and 27/37 files 2016-08-25 09:01:57,607 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.52/33.61 kb and 28/37 files 2016-08-25 09:01:57,607 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.42/33.61 kb and 29/37 files 2016-08-25 09:01:57,608 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.32/33.61 kb and 30/37 files 2016-08-25 09:01:57,608 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.21/33.61 kb and 31/37 files 2016-08-25 09:01:57,609 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.11/33.61 kb and 32/37 files 2016-08-25 09:01:57,611 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.01/33.61 kb and 33/37 files 2016-08-25 09:01:57,611 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.91/33.61 kb and 34/37 files 2016-08-25 09:01:57,612 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.81/33.61 kb and 35/37 files 2016-08-25 09:01:57,612 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 32.71/33.61 kb and 36/37 files 2016-08-25 09:01:57,613 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 33.61/33.61 kb and 37/37 files 2016-08-25 09:01:57,613 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:01:57,623 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 68.5. Found ====================================== ====================================== 69.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:57,633 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) is shutting down 2016-08-25 09:01:58,636 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:58,644 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:58,644 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:58,644 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250045 2016-08-25 09:01:58,645 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:58,645 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:58,645 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:58,645 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:58,648 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55453 which had sessionid 0x156bec917250045 2016-08-25 09:01:58,648 | INFO | Time-limited test | Session: 0x156bec917250045 closed 2016-08-25 09:01:58,648 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:58,661 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:01:58,661 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:58,661 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:58,662 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:58,664 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:58,666 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:58,696 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55477 2016-08-25 09:01:58,700 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:58,701 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55477 2016-08-25 09:01:58,701 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:01:58,711 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:58,713 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:58,713 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:58,714 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000108.log 2016-08-25 09:01:58,714 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:01:58,714 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000106 2016-08-25 09:01:58,716 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.02/4.17 kb and 1/4 files 2016-08-25 09:01:58,716 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.20/4.17 kb and 2/4 files 2016-08-25 09:01:58,717 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.14/4.17 kb and 3/4 files 2016-08-25 09:01:58,717 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.17/4.17 kb and 4/4 files 2016-08-25 09:01:58,717 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:58,719 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:58,720 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) is starting 2016-08-25 09:01:58,724 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55480 2016-08-25 09:01:58,724 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:58,724 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) started 2016-08-25 09:01:58,724 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:58,725 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb 2016-08-25 09:01:58,725 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb 2016-08-25 09:01:58,753 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:58,753 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) uptime 4.289 seconds 2016-08-25 09:01:58,754 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) is shutdown Wait for master to start up... Master started: node-3 ====================================== 69.4. Browse the queue on new master ====================================== 2016-08-25 09:01:58,756 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:58,758 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4c6425fa 2016-08-25 09:01:58,758 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:58,759 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:58,759 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:58,759 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55482 2016-08-25 09:01:58,759 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55482 2016-08-25 09:01:58,760 | INFO | SyncThread:0 | Established session 0x156bec917250048 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55482 2016-08-25 09:01:58,760 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250048, negotiated timeout = 10000 ====================================== 69.5. Found ====================================== ====================================== 70.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:58,770 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) is shutting down 2016-08-25 09:01:58,771 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:58,772 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:58,772 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55477 2016-08-25 09:01:58,772 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:58,775 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:58,777 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:58,778 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250046 2016-08-25 09:01:58,778 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:58,778 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:58,778 | INFO | Time-limited test | Session: 0x156bec917250046 closed 2016-08-25 09:01:58,778 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:58,778 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55461 which had sessionid 0x156bec917250046 2016-08-25 09:01:58,781 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:58,898 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:58,899 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) uptime 3.503 seconds 2016-08-25 09:01:58,899 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) is shutdown Wait for master to start up... 2016-08-25 09:01:58,901 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:58,903 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@41b861ab 2016-08-25 09:01:58,903 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:58,904 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:58,905 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:58,905 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55484 2016-08-25 09:01:58,905 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55484 2016-08-25 09:01:58,906 | INFO | SyncThread:0 | Established session 0x156bec917250049 with negotiated timeout 10000 for client /127.0.0.1:55484 2016-08-25 09:01:58,907 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250049, negotiated timeout = 10000 2016-08-25 09:01:58,918 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,158 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:59,158 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,158 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,159 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:59,161 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:59,164 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,181 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55485 2016-08-25 09:01:59,184 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,184 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,184 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55485 2016-08-25 09:01:59,184 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:59,185 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55485 2016-08-25 09:01:59,185 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:59,199 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,365 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:59,365 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:59,366 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:59,366 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:01:59,366 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000108.log 2016-08-25 09:01:59,366 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000106 2016-08-25 09:01:59,370 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/4.17 kb and 1/4 files 2016-08-25 09:01:59,370 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:59,370 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.96/4.17 kb and 2/4 files 2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.14/4.17 kb and 3/4 files 2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000106 2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000108.log 2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.17/4.17 kb and 4/4 files 2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:59,375 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,376 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.03/4.17 kb and 1/4 files 2016-08-25 09:01:59,376 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.22/4.17 kb and 2/4 files 2016-08-25 09:01:59,377 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.23/4.17 kb and 3/4 files 2016-08-25 09:01:59,377 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.17/4.17 kb and 4/4 files 2016-08-25 09:01:59,377 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:59,378 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) is starting 2016-08-25 09:01:59,382 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55490 2016-08-25 09:01:59,382 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:59,383 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) started 2016-08-25 09:01:59,383 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:59,383 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:01:59,383 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb Master started: node-2 ====================================== 70.4. Browse the queue on new master ====================================== 2016-08-25 09:01:59,386 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d ====================================== 70.5. Found ====================================== ====================================== 71.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:59,399 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) is shutting down 2016-08-25 09:01:59,399 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:59,399 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | socketQueue interrupted - stopping 2016-08-25 09:01:59,399 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | Could not accept connection during shutdown : java.lang.InterruptedException 2016-08-25 09:01:59,402 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,403 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,403 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,404 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,404 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:59,404 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:59,404 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250047 2016-08-25 09:01:59,405 | INFO | Time-limited test | Session: 0x156bec917250047 closed 2016-08-25 09:01:59,405 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:59,405 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55469 which had sessionid 0x156bec917250047 2016-08-25 09:01:59,438 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:59,440 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,443 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:59,443 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,446 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:59,451 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,480 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55492 2016-08-25 09:01:59,483 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,484 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55492 2016-08-25 09:01:59,485 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:01:59,500 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,502 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000112.log 2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000110 2016-08-25 09:01:59,507 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/3.22 kb and 1/5 files 2016-08-25 09:01:59,508 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.20/3.22 kb and 2/5 files 2016-08-25 09:01:59,508 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.14/3.22 kb and 3/5 files 2016-08-25 09:01:59,509 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.04/3.22 kb and 4/5 files 2016-08-25 09:01:59,509 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:59,510 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.22/3.22 kb and 5/5 files 2016-08-25 09:01:59,510 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) uptime 2.888 seconds 2016-08-25 09:01:59,510 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:01:59,510 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) is shutdown Wait for master to start up... 2016-08-25 09:01:59,512 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:59,513 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,514 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) is starting 2016-08-25 09:01:59,516 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@28370f7b 2016-08-25 09:01:59,516 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:59,517 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:59,518 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:59,518 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55495 2016-08-25 09:01:59,518 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55496 2016-08-25 09:01:59,518 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:59,518 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) started 2016-08-25 09:01:59,518 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:59,518 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55496 2016-08-25 09:01:59,519 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:01:59,519 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:01:59,519 | INFO | SyncThread:0 | Established session 0x156bec91725004a with negotiated timeout 10000 for client /127.0.0.1:55496 2016-08-25 09:01:59,520 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725004a, negotiated timeout = 10000 Master started: node-1 ====================================== 71.4. Browse the queue on new master ====================================== 2016-08-25 09:01:59,529 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,530 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55492 2016-08-25 09:01:59,530 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started ====================================== 71.5. Found ====================================== ====================================== 72.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:59,537 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) is shutting down 2016-08-25 09:01:59,538 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:59,541 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,542 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:59,542 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250048 2016-08-25 09:01:59,543 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55482 which had sessionid 0x156bec917250048 2016-08-25 09:01:59,543 | INFO | Time-limited test | Session: 0x156bec917250048 closed 2016-08-25 09:01:59,543 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:59,552 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,559 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:01:59,559 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,560 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,561 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,571 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:01:59,571 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,574 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:01:59,578 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,606 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55499 2016-08-25 09:01:59,609 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,609 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55499 2016-08-25 09:01:59,609 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:59,618 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:59,620 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000110 2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000112.log 2016-08-25 09:01:59,625 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/3.22 kb and 1/5 files 2016-08-25 09:01:59,627 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.92/3.22 kb and 2/5 files 2016-08-25 09:01:59,628 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.86/3.22 kb and 3/5 files 2016-08-25 09:01:59,628 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.04/3.22 kb and 4/5 files 2016-08-25 09:01:59,629 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.22/3.22 kb and 5/5 files 2016-08-25 09:01:59,629 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:59,631 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:59,635 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) is starting 2016-08-25 09:01:59,638 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55502 2016-08-25 09:01:59,639 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:59,639 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) started 2016-08-25 09:01:59,639 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:59,639 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb 2016-08-25 09:01:59,639 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb 2016-08-25 09:01:59,668 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:01:59,669 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) uptime 0.915 seconds 2016-08-25 09:01:59,669 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) is shutdown Wait for master to start up... Master started: node-3 ====================================== 72.4. Browse the queue on new master ====================================== 2016-08-25 09:01:59,671 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:01:59,673 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@61a8a328 2016-08-25 09:01:59,673 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:59,673 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:59,674 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:59,674 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55504 2016-08-25 09:01:59,674 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55504 2016-08-25 09:01:59,675 | INFO | SyncThread:0 | Established session 0x156bec91725004b with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55504 2016-08-25 09:01:59,675 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725004b, negotiated timeout = 10000 ====================================== 72.5. Found ====================================== ====================================== 73.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:01:59,681 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) is shutting down 2016-08-25 09:01:59,681 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:59,682 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,683 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55499 2016-08-25 09:01:59,683 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:01:59,683 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,685 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,685 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:59,685 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,685 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:01:59,685 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250049 2016-08-25 09:01:59,685 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:01:59,686 | INFO | Time-limited test | Session: 0x156bec917250049 closed 2016-08-25 09:01:59,686 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55484 which had sessionid 0x156bec917250049 2016-08-25 09:01:59,686 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:01:59,687 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,701 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:01:59,701 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,704 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:01:59,707 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,739 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55506 2016-08-25 09:01:59,742 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,742 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55506 2016-08-25 09:01:59,742 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:01:59,764 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:59,767 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:59,767 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:01:59,767 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:01:59,767 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:01:59,768 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000115.log 2016-08-25 09:01:59,768 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:01:59,768 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000113 2016-08-25 09:01:59,771 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/4.17 kb and 1/6 files 2016-08-25 09:01:59,772 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/4.17 kb and 2/6 files 2016-08-25 09:01:59,772 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.86/4.17 kb and 3/6 files 2016-08-25 09:01:59,772 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.04/4.17 kb and 4/6 files 2016-08-25 09:01:59,773 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.94/4.17 kb and 5/6 files 2016-08-25 09:01:59,773 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files 2016-08-25 09:01:59,773 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:59,775 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:59,776 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) is starting 2016-08-25 09:01:59,781 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55509 2016-08-25 09:01:59,782 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:59,782 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) started 2016-08-25 09:01:59,782 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:59,782 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb 2016-08-25 09:01:59,782 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb 2016-08-25 09:01:59,790 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:01:59,791 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) uptime 0.892 seconds 2016-08-25 09:01:59,791 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) is shutdown Wait for master to start up... Master started: node-2 ====================================== 73.4. Browse the queue on new master ====================================== 2016-08-25 09:01:59,794 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:01:59,798 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@79ea3abe 2016-08-25 09:01:59,798 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:59,799 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:59,799 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:01:59,799 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55511 2016-08-25 09:01:59,800 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55511 2016-08-25 09:01:59,801 | INFO | SyncThread:0 | Established session 0x156bec91725004c with negotiated timeout 10000 for client /127.0.0.1:55511 2016-08-25 09:01:59,801 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725004c, negotiated timeout = 10000 ====================================== 73.5. Found ====================================== ====================================== 74.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:01:59,809 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) is shutting down 2016-08-25 09:01:59,810 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:59,811 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:59,812 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,812 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004a 2016-08-25 09:01:59,813 | INFO | Time-limited test | Session: 0x156bec91725004a closed 2016-08-25 09:01:59,813 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:01:59,813 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55496 which had sessionid 0x156bec91725004a 2016-08-25 09:01:59,814 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,816 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55506 2016-08-25 09:01:59,816 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:01:59,817 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,817 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:01:59,817 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Slave stopped 2016-08-25 09:01:59,817 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,819 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,838 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:01:59,838 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,842 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:01:59,844 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,879 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55513 2016-08-25 09:01:59,882 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,882 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Attaching to master: tcp://localhost:55513 2016-08-25 09:01:59,882 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Slave started 2016-08-25 09:01:59,900 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,902 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000113 2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000115.log 2016-08-25 09:01:59,906 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/4.17 kb and 1/6 files 2016-08-25 09:01:59,906 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.17/4.17 kb and 2/6 files 2016-08-25 09:01:59,907 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.07/4.17 kb and 3/6 files 2016-08-25 09:01:59,907 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.09/4.17 kb and 4/6 files 2016-08-25 09:01:59,907 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.99/4.17 kb and 5/6 files 2016-08-25 09:01:59,908 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files 2016-08-25 09:01:59,908 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:01:59,910 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,910 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) is starting 2016-08-25 09:01:59,914 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55516 2016-08-25 09:01:59,914 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:01:59,914 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) started 2016-08-25 09:01:59,915 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:01:59,915 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb 2016-08-25 09:01:59,915 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb 2016-08-25 09:01:59,948 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:01:59,949 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) uptime 0.439 seconds 2016-08-25 09:01:59,949 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) is shutdown Wait for master to start up... Master started: node-1 ====================================== 74.4. Browse the queue on new master ====================================== 2016-08-25 09:01:59,951 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:01:59,955 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@301ed798 2016-08-25 09:01:59,955 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:01:59,956 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:01:59,957 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:01:59,957 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55518 2016-08-25 09:01:59,957 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55518 2016-08-25 09:01:59,958 | INFO | SyncThread:0 | Established session 0x156bec91725004d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55518 2016-08-25 09:01:59,958 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725004d, negotiated timeout = 10000 ====================================== 74.5. Found ====================================== ====================================== 75.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:01:59,966 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) is shutting down 2016-08-25 09:01:59,967 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:01:59,969 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:01:59,970 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004b 2016-08-25 09:01:59,970 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:01:59,970 | INFO | Time-limited test | Session: 0x156bec91725004b closed 2016-08-25 09:01:59,971 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:01:59,971 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55504 which had sessionid 0x156bec91725004b 2016-08-25 09:01:59,971 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,971 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:01:59,972 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:01:59,974 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55513 2016-08-25 09:01:59,974 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:01:59,975 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave stopped 2016-08-25 09:01:59,975 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:01:59,977 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,003 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:00,003 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,007 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:00,009 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,049 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Master started: tcp://localhost:55520 2016-08-25 09:02:00,052 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,052 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Attaching to master: tcp://localhost:55520 2016-08-25 09:02:00,052 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave started 2016-08-25 09:02:00,079 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:00,079 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) uptime 0.410 seconds 2016-08-25 09:02:00,079 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) is shutdown Wait for master to start up... 2016-08-25 09:02:00,081 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:00,083 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,083 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@318be84d 2016-08-25 09:02:00,084 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:00,085 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:00,086 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:00,086 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55523 2016-08-25 09:02:00,086 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55523 2016-08-25 09:02:00,087 | INFO | SyncThread:0 | Established session 0x156bec91725004e with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55523 2016-08-25 09:02:00,087 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,087 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725004e, negotiated timeout = 10000 2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000118.log 2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000116 2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:00,095 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/5.12 kb and 1/7 files 2016-08-25 09:02:00,096 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/5.12 kb and 2/7 files 2016-08-25 09:02:00,097 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.74/5.12 kb and 3/7 files 2016-08-25 09:02:00,097 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.92/5.12 kb and 4/7 files 2016-08-25 09:02:00,097 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,098 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.21/5.12 kb and 5/7 files 2016-08-25 09:02:00,098 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.22/5.12 kb and 6/7 files 2016-08-25 09:02:00,099 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files 2016-08-25 09:02:00,099 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:00,099 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55520 2016-08-25 09:02:00,099 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:00,104 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,106 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) is starting 2016-08-25 09:02:00,112 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55525 2016-08-25 09:02:00,112 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:00,113 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) started 2016-08-25 09:02:00,113 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:00,113 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:00,113 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb Master started: node-3 ====================================== 75.4. Browse the queue on new master ====================================== ====================================== 75.5. Found ====================================== ====================================== 76.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:00,129 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) is shutting down 2016-08-25 09:02:00,130 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:00,134 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,137 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:00,138 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004c 2016-08-25 09:02:00,139 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55511 which had sessionid 0x156bec91725004c 2016-08-25 09:02:00,139 | INFO | Time-limited test | Session: 0x156bec91725004c closed 2016-08-25 09:02:00,139 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:00,140 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,140 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,164 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:00,164 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,167 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,189 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave stopped 2016-08-25 09:02:00,189 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,192 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:00,195 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,230 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55527 2016-08-25 09:02:00,234 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,234 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55527 2016-08-25 09:02:00,234 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:02:00,253 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:00,254 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) uptime 0.463 seconds 2016-08-25 09:02:00,254 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) is shutdown Wait for master to start up... 2016-08-25 09:02:00,255 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:00,256 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:00,258 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@567172fc 2016-08-25 09:02:00,258 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:00,258 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,259 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:00,259 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000118.log 2016-08-25 09:02:00,259 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55530 2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:00,259 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55530 2016-08-25 09:02:00,260 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000116 2016-08-25 09:02:00,260 | INFO | SyncThread:0 | Established session 0x156bec91725004f with negotiated timeout 10000 for client /127.0.0.1:55530 2016-08-25 09:02:00,260 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725004f, negotiated timeout = 10000 2016-08-25 09:02:00,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/5.12 kb and 1/7 files 2016-08-25 09:02:00,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.96/5.12 kb and 2/7 files 2016-08-25 09:02:00,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.86/5.12 kb and 3/7 files 2016-08-25 09:02:00,267 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.04/5.12 kb and 4/7 files 2016-08-25 09:02:00,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.94/5.12 kb and 5/7 files 2016-08-25 09:02:00,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.84/5.12 kb and 6/7 files 2016-08-25 09:02:00,268 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55527 2016-08-25 09:02:00,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files 2016-08-25 09:02:00,268 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:00,268 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:00,271 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:00,272 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) is starting 2016-08-25 09:02:00,276 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55532 2016-08-25 09:02:00,276 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:00,276 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) started 2016-08-25 09:02:00,276 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:00,276 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:00,277 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb Master started: node-2 ====================================== 76.4. Browse the queue on new master ====================================== 2016-08-25 09:02:00,292 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 76.5. Found ====================================== ====================================== 77.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:00,295 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) is shutting down 2016-08-25 09:02:00,295 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:00,298 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,299 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,299 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:00,299 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:00,299 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004d 2016-08-25 09:02:00,300 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,300 | INFO | Time-limited test | Session: 0x156bec91725004d closed 2016-08-25 09:02:00,301 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55518 which had sessionid 0x156bec91725004d 2016-08-25 09:02:00,301 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000118.log 2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000116 2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:00,302 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,303 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,303 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:00,313 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:00,314 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,375 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:02:00,375 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,378 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:00,381 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,407 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:00,408 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) uptime 0.458 seconds 2016-08-25 09:02:00,408 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) is shutdown Wait for master to start up... 2016-08-25 09:02:00,410 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:00,412 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@2c6bedc9 2016-08-25 09:02:00,412 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:00,413 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:00,413 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:00,413 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55535 2016-08-25 09:02:00,413 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55535 2016-08-25 09:02:00,414 | INFO | SyncThread:0 | Established session 0x156bec917250050 with negotiated timeout 10000 for client /127.0.0.1:55535 2016-08-25 09:02:00,414 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250050, negotiated timeout = 10000 2016-08-25 09:02:00,429 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55536 2016-08-25 09:02:00,432 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,432 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,433 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55536 2016-08-25 09:02:00,433 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:02:00,434 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55536 2016-08-25 09:02:00,434 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:00,449 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:00,464 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,465 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000119 2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000121.log 2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:00,467 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:00,467 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:00,467 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000119 2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000121.log 2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:00,470 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.33/6.07 kb and 1/8 files 2016-08-25 09:02:00,470 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.23/6.07 kb and 2/8 files 2016-08-25 09:02:00,471 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.42/6.07 kb and 3/8 files 2016-08-25 09:02:00,472 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.32/6.07 kb and 4/8 files 2016-08-25 09:02:00,472 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.22/6.07 kb and 5/8 files 2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.23/6.07 kb and 6/8 files 2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.13/6.07 kb and 7/8 files 2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files 2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.94/6.07 kb and 1/8 files 2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:00,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.28/6.07 kb and 2/8 files 2016-08-25 09:02:00,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.29/6.07 kb and 3/8 files 2016-08-25 09:02:00,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.19/6.07 kb and 4/8 files 2016-08-25 09:02:00,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.38/6.07 kb and 5/8 files 2016-08-25 09:02:00,476 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.28/6.07 kb and 6/8 files 2016-08-25 09:02:00,476 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.17/6.07 kb and 7/8 files 2016-08-25 09:02:00,476 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:00,477 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files 2016-08-25 09:02:00,477 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:00,477 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) is starting 2016-08-25 09:02:00,480 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,482 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55541 2016-08-25 09:02:00,482 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:00,482 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) started 2016-08-25 09:02:00,482 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:00,482 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:00,482 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb Master started: node-1 ====================================== 77.4. Browse the queue on new master ====================================== ====================================== 77.5. Found ====================================== ====================================== 78.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:00,499 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) is shutting down 2016-08-25 09:02:00,500 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:00,503 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,504 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:00,504 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004e 2016-08-25 09:02:00,505 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55523 which had sessionid 0x156bec91725004e 2016-08-25 09:02:00,505 | INFO | Time-limited test | Session: 0x156bec91725004e closed 2016-08-25 09:02:00,505 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:00,509 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,509 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,509 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:00,509 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,578 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:00,578 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:00,578 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,580 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,583 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:00,586 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,621 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:00,622 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) uptime 0.543 seconds 2016-08-25 09:02:00,622 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) is shutdown Wait for master to start up... 2016-08-25 09:02:00,624 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:00,626 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1a984a63 2016-08-25 09:02:00,626 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:00,627 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:00,627 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:00,627 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55543 2016-08-25 09:02:00,628 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55543 2016-08-25 09:02:00,628 | INFO | SyncThread:0 | Established session 0x156bec917250051 with negotiated timeout 10000 for client /127.0.0.1:55543 2016-08-25 09:02:00,628 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250051, negotiated timeout = 10000 2016-08-25 09:02:00,636 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55544 2016-08-25 09:02:00,640 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,640 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,641 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55544 2016-08-25 09:02:00,641 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:00,641 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55544 2016-08-25 09:02:00,641 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:02:00,674 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,683 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:00,683 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,683 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000120.sst 2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000119 2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000121.log 2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:00,688 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/6.07 kb and 1/8 files 2016-08-25 09:02:00,688 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/6.07 kb and 2/8 files 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000121.log 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000120.sst 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.81/6.07 kb and 3/8 files 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000119 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:00,690 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.71/6.07 kb and 4/8 files 2016-08-25 09:02:00,690 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.66/6.07 kb and 5/8 files 2016-08-25 09:02:00,691 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.99/6.07 kb and 6/8 files 2016-08-25 09:02:00,691 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.17/6.07 kb and 7/8 files 2016-08-25 09:02:00,692 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files 2016-08-25 09:02:00,692 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:02:00,694 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/6.07 kb and 1/8 files 2016-08-25 09:02:00,694 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/6.07 kb and 2/8 files 2016-08-25 09:02:00,695 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,695 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.98/6.07 kb and 3/8 files 2016-08-25 09:02:00,695 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.92/6.07 kb and 4/8 files 2016-08-25 09:02:00,696 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) is starting 2016-08-25 09:02:00,696 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.82/6.07 kb and 5/8 files 2016-08-25 09:02:00,696 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.16/6.07 kb and 6/8 files 2016-08-25 09:02:00,697 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.06/6.07 kb and 7/8 files 2016-08-25 09:02:00,697 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files 2016-08-25 09:02:00,697 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:00,700 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55549 2016-08-25 09:02:00,700 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:00,700 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) started 2016-08-25 09:02:00,700 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:00,700 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:00,700 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:00,701 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d Master started: node-3 ====================================== 78.4. Browse the queue on new master ====================================== ====================================== 78.5. Found ====================================== ====================================== 79.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:00,717 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) is shutting down 2016-08-25 09:02:00,718 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:00,720 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,721 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:00,721 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004f 2016-08-25 09:02:00,721 | INFO | Time-limited test | Session: 0x156bec91725004f closed 2016-08-25 09:02:00,721 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:00,722 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55530 which had sessionid 0x156bec91725004f 2016-08-25 09:02:00,724 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,724 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:00,724 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:00,724 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:00,826 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:00,826 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) uptime 0.572 seconds 2016-08-25 09:02:00,826 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) is shutdown Wait for master to start up... 2016-08-25 09:02:00,828 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:00,832 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5652fdf4 2016-08-25 09:02:00,832 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:00,833 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:00,833 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,833 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:00,834 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55551 2016-08-25 09:02:00,834 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:00,834 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55551 2016-08-25 09:02:00,835 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:00,836 | INFO | SyncThread:0 | Established session 0x156bec917250052 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55551 2016-08-25 09:02:00,836 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250052, negotiated timeout = 10000 2016-08-25 09:02:00,837 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:00,840 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:00,847 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,901 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55552 2016-08-25 09:02:00,905 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,905 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:00,907 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55552 2016-08-25 09:02:00,907 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:00,907 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55552 2016-08-25 09:02:00,907 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:02:00,953 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:00,953 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000123.sst 2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000124.log 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000123.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000122 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000124.log 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000122 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:00,963 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/7.02 kb and 1/9 files 2016-08-25 09:02:00,963 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/7.02 kb and 2/9 files 2016-08-25 09:02:00,964 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/7.02 kb and 3/9 files 2016-08-25 09:02:00,964 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/7.02 kb and 4/9 files 2016-08-25 09:02:00,965 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.78/7.02 kb and 5/9 files 2016-08-25 09:02:00,965 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.80/7.02 kb and 6/9 files 2016-08-25 09:02:00,966 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.18/7.02 kb and 7/9 files 2016-08-25 09:02:00,967 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.08/7.02 kb and 8/9 files 2016-08-25 09:02:00,967 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files 2016-08-25 09:02:00,967 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:00,968 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/7.02 kb and 1/9 files 2016-08-25 09:02:00,968 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/7.02 kb and 2/9 files 2016-08-25 09:02:00,969 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.74/7.02 kb and 3/9 files 2016-08-25 09:02:00,969 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.92/7.02 kb and 4/9 files 2016-08-25 09:02:00,970 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.31/7.02 kb and 5/9 files 2016-08-25 09:02:00,970 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:00,970 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.21/7.02 kb and 6/9 files 2016-08-25 09:02:00,971 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.11/7.02 kb and 7/9 files 2016-08-25 09:02:00,971 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) is starting 2016-08-25 09:02:00,971 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.01/7.02 kb and 8/9 files 2016-08-25 09:02:00,972 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files 2016-08-25 09:02:00,972 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:00,976 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55557 2016-08-25 09:02:00,976 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:00,977 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:00,977 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) started 2016-08-25 09:02:00,977 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:00,977 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:00,977 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb Master started: node-2 ====================================== 79.4. Browse the queue on new master ====================================== ====================================== 79.5. Found ====================================== ====================================== 80.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:00,998 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) is shutting down 2016-08-25 09:02:00,998 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:01,006 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,009 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:01,010 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250050 2016-08-25 09:02:01,011 | INFO | Time-limited test | Session: 0x156bec917250050 closed 2016-08-25 09:02:01,012 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:01,012 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55535 which had sessionid 0x156bec917250050 2016-08-25 09:02:01,013 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:01,014 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:01,014 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:01,014 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:01,102 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:01,104 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,104 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:02:01,104 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,108 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:01,111 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,127 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:01,127 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) uptime 0.719 seconds 2016-08-25 09:02:01,127 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) is shutdown Wait for master to start up... 2016-08-25 09:02:01,129 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:01,131 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@33fca8e4 2016-08-25 09:02:01,131 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:01,132 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:01,132 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:01,132 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55559 2016-08-25 09:02:01,133 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55559 2016-08-25 09:02:01,133 | INFO | SyncThread:0 | Established session 0x156bec917250053 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55559 2016-08-25 09:02:01,133 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250053, negotiated timeout = 10000 2016-08-25 09:02:01,164 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55560 2016-08-25 09:02:01,168 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,169 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,170 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55560 2016-08-25 09:02:01,170 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:01,170 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55560 2016-08-25 09:02:01,170 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:02:01,181 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:01,184 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000122 2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000120.sst 2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:01,186 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:01,186 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000123.sst 2016-08-25 09:02:01,186 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000124.log 2016-08-25 09:02:01,190 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/7.02 kb and 1/9 files 2016-08-25 09:02:01,191 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.96/7.02 kb and 2/9 files 2016-08-25 09:02:01,191 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.34/7.02 kb and 3/9 files 2016-08-25 09:02:01,192 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.24/7.02 kb and 4/9 files 2016-08-25 09:02:01,193 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.14/7.02 kb and 5/9 files 2016-08-25 09:02:01,195 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.04/7.02 kb and 6/9 files 2016-08-25 09:02:01,195 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.94/7.02 kb and 7/9 files 2016-08-25 09:02:01,196 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.84/7.02 kb and 8/9 files 2016-08-25 09:02:01,196 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files 2016-08-25 09:02:01,196 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:01,199 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:01,200 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) is starting 2016-08-25 09:02:01,204 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55564 2016-08-25 09:02:01,204 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:01,205 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) started 2016-08-25 09:02:01,205 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:01,205 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb 2016-08-25 09:02:01,205 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb Master started: node-1 ====================================== 80.4. Browse the queue on new master ====================================== ====================================== 80.5. Found ====================================== ====================================== 81.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:01,217 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) is shutting down 2016-08-25 09:02:01,217 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:01,220 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,221 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:01,221 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:01,221 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:01,221 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250051 2016-08-25 09:02:01,222 | INFO | Time-limited test | Session: 0x156bec917250051 closed 2016-08-25 09:02:01,222 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55543 which had sessionid 0x156bec917250051 2016-08-25 09:02:01,222 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:01,234 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:01,274 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:01,275 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,278 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,327 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:01,328 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) uptime 0.706 seconds 2016-08-25 09:02:01,328 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) is shutdown Wait for master to start up... 2016-08-25 09:02:01,330 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:01,332 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3f91bd18 2016-08-25 09:02:01,332 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:01,333 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:01,333 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:01,333 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55566 2016-08-25 09:02:01,334 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55566 2016-08-25 09:02:01,335 | INFO | SyncThread:0 | Established session 0x156bec917250054 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55566 2016-08-25 09:02:01,335 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250054, negotiated timeout = 10000 2016-08-25 09:02:01,342 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,349 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:01,353 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:01,356 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,411 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55567 2016-08-25 09:02:01,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,415 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55567 2016-08-25 09:02:01,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:01,417 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55567 2016-08-25 09:02:01,417 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:01,454 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:01,493 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:01,494 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:01,494 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000127.log 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000126.sst 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000123.sst 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000125 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:01,500 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/7.98 kb and 1/10 files 2016-08-25 09:02:01,500 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/7.98 kb and 2/10 files 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000123.sst 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000126.sst 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.10/7.98 kb and 3/10 files 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000125 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000127.log 2016-08-25 09:02:01,502 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.00/7.98 kb and 4/10 files 2016-08-25 09:02:01,502 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.90/7.98 kb and 5/10 files 2016-08-25 09:02:01,503 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.80/7.98 kb and 6/10 files 2016-08-25 09:02:01,503 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.70/7.98 kb and 7/10 files 2016-08-25 09:02:01,504 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.13/7.98 kb and 8/10 files 2016-08-25 09:02:01,505 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.03/7.98 kb and 9/10 files 2016-08-25 09:02:01,505 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.98/7.98 kb and 10/10 files 2016-08-25 09:02:01,505 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:02:01,507 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/7.98 kb and 1/10 files 2016-08-25 09:02:01,507 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/7.98 kb and 2/10 files 2016-08-25 09:02:01,508 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.74/7.98 kb and 3/10 files 2016-08-25 09:02:01,508 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:01,509 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.64/7.98 kb and 4/10 files 2016-08-25 09:02:01,509 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) is starting 2016-08-25 09:02:01,509 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.54/7.98 kb and 5/10 files 2016-08-25 09:02:01,510 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.55/7.98 kb and 6/10 files 2016-08-25 09:02:01,510 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.99/7.98 kb and 7/10 files 2016-08-25 09:02:01,511 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.89/7.98 kb and 8/10 files 2016-08-25 09:02:01,511 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.79/7.98 kb and 9/10 files 2016-08-25 09:02:01,512 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.98/7.98 kb and 10/10 files 2016-08-25 09:02:01,512 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:01,513 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55572 2016-08-25 09:02:01,514 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:01,514 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) started 2016-08-25 09:02:01,514 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:01,514 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb 2016-08-25 09:02:01,514 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb Master started: node-2 ====================================== 81.4. Browse the queue on new master ====================================== 2016-08-25 09:02:01,516 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d ====================================== 81.5. Found ====================================== ====================================== 82.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:01,531 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) is shutting down 2016-08-25 09:02:01,532 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:01,537 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:02:01,537 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,543 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:01,543 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250053 2016-08-25 09:02:01,544 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:01,544 | INFO | Time-limited test | Session: 0x156bec917250053 closed 2016-08-25 09:02:01,544 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:01,544 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55559 which had sessionid 0x156bec917250053 2016-08-25 09:02:01,545 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,546 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:01,546 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:01,547 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:01,631 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:01,631 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,633 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:02:01,633 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,635 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:01,638 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,661 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:01,662 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) uptime 0.534 seconds 2016-08-25 09:02:01,662 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) is shutdown Wait for master to start up... 2016-08-25 09:02:01,665 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:01,666 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@51698ab9 2016-08-25 09:02:01,666 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:01,667 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:01,668 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:01,668 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55574 2016-08-25 09:02:01,668 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55574 2016-08-25 09:02:01,669 | INFO | SyncThread:0 | Established session 0x156bec917250055 with negotiated timeout 10000 for client /127.0.0.1:55574 2016-08-25 09:02:01,669 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250055, negotiated timeout = 10000 2016-08-25 09:02:01,707 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55575 2016-08-25 09:02:01,714 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,714 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:01,715 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55575 2016-08-25 09:02:01,716 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:01,716 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55575 2016-08-25 09:02:01,716 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:01,729 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:01,763 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:01,763 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000123.sst 2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000120.sst 2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000127.log 2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000125 2016-08-25 09:02:01,765 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000126.sst 2016-08-25 09:02:01,765 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:01,769 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:01,769 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/7.98 kb and 1/10 files 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/7.98 kb and 2/10 files 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000120.sst 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000126.sst 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000123.sst 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/7.98 kb and 3/10 files 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000125 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000127.log 2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:01,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/7.98 kb and 4/10 files 2016-08-25 09:02:01,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.61/7.98 kb and 5/10 files 2016-08-25 09:02:01,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.80/7.98 kb and 6/10 files 2016-08-25 09:02:01,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.70/7.98 kb and 7/10 files 2016-08-25 09:02:01,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.13/7.98 kb and 8/10 files 2016-08-25 09:02:01,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.03/7.98 kb and 9/10 files 2016-08-25 09:02:01,774 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.98/7.98 kb and 10/10 files 2016-08-25 09:02:01,774 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:01,775 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/7.98 kb and 1/10 files 2016-08-25 09:02:01,776 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/7.98 kb and 2/10 files 2016-08-25 09:02:01,776 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:01,776 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/7.98 kb and 3/10 files 2016-08-25 09:02:01,777 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/7.98 kb and 4/10 files 2016-08-25 09:02:01,777 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) is starting 2016-08-25 09:02:01,777 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/7.98 kb and 5/10 files 2016-08-25 09:02:01,778 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.51/7.98 kb and 6/10 files 2016-08-25 09:02:01,778 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.95/7.98 kb and 7/10 files 2016-08-25 09:02:01,779 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.85/7.98 kb and 8/10 files 2016-08-25 09:02:01,779 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.03/7.98 kb and 9/10 files 2016-08-25 09:02:01,780 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.98/7.98 kb and 10/10 files 2016-08-25 09:02:01,780 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:01,781 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55580 2016-08-25 09:02:01,782 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:01,782 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) started 2016-08-25 09:02:01,782 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:01,782 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb 2016-08-25 09:02:01,782 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb Master started: node-3 ====================================== 82.4. Browse the queue on new master ====================================== 2016-08-25 09:02:01,784 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d ====================================== 82.5. Found ====================================== ====================================== 83.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:01,799 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) is shutting down 2016-08-25 09:02:01,800 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:01,805 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,806 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:01,806 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:01,806 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:01,806 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:01,806 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250052 2016-08-25 09:02:01,806 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:01,807 | INFO | Time-limited test | Session: 0x156bec917250052 closed 2016-08-25 09:02:01,807 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:01,807 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55551 which had sessionid 0x156bec917250052 2016-08-25 09:02:01,913 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:01,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) uptime 1.086 seconds 2016-08-25 09:02:01,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) is shutdown Wait for master to start up... 2016-08-25 09:02:01,916 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:01,918 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3018e40b 2016-08-25 09:02:01,918 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:01,919 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:01,920 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:01,920 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55582 2016-08-25 09:02:01,920 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55582 2016-08-25 09:02:01,921 | INFO | SyncThread:0 | Established session 0x156bec917250056 with negotiated timeout 10000 for client /127.0.0.1:55582 2016-08-25 09:02:01,921 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250056, negotiated timeout = 10000 2016-08-25 09:02:01,927 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,965 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:01,966 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:01,967 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,968 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:01,971 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:01,975 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,036 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55583 2016-08-25 09:02:02,041 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,041 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,042 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55583 2016-08-25 09:02:02,042 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:02:02,043 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55583 2016-08-25 09:02:02,043 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:02,100 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:02,103 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000126.sst 2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000123.sst 2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000129.sst 2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000130.log 2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000128 2016-08-25 09:02:02,108 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000130.log 2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000126.sst 2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000129.sst 2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000128 2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000123.sst 2016-08-25 09:02:02,112 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/8.93 kb and 1/11 files 2016-08-25 09:02:02,113 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/8.93 kb and 2/11 files 2016-08-25 09:02:02,113 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.74/8.93 kb and 3/11 files 2016-08-25 09:02:02,113 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.64/8.93 kb and 4/11 files 2016-08-25 09:02:02,114 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.54/8.93 kb and 5/11 files 2016-08-25 09:02:02,114 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.44/8.93 kb and 6/11 files 2016-08-25 09:02:02,115 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.34/8.93 kb and 7/11 files 2016-08-25 09:02:02,115 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.35/8.93 kb and 8/11 files 2016-08-25 09:02:02,115 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.25/8.93 kb and 9/11 files 2016-08-25 09:02:02,116 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.44/8.93 kb and 10/11 files 2016-08-25 09:02:02,116 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.93/8.93 kb and 11/11 files 2016-08-25 09:02:02,116 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:02,117 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/8.93 kb and 1/11 files 2016-08-25 09:02:02,118 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/8.93 kb and 2/11 files 2016-08-25 09:02:02,118 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.98/8.93 kb and 3/11 files 2016-08-25 09:02:02,119 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.88/8.93 kb and 4/11 files 2016-08-25 09:02:02,119 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:02,120 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.78/8.93 kb and 5/11 files 2016-08-25 09:02:02,120 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.72/8.93 kb and 6/11 files 2016-08-25 09:02:02,121 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) is starting 2016-08-25 09:02:02,121 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.62/8.93 kb and 7/11 files 2016-08-25 09:02:02,122 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.52/8.93 kb and 8/11 files 2016-08-25 09:02:02,122 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.01/8.93 kb and 9/11 files 2016-08-25 09:02:02,123 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.03/8.93 kb and 10/11 files 2016-08-25 09:02:02,123 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.93/8.93 kb and 11/11 files 2016-08-25 09:02:02,123 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:02:02,125 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55588 2016-08-25 09:02:02,125 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:02,125 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) started 2016-08-25 09:02:02,125 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:02,125 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:02,126 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-1 ====================================== 83.4. Browse the queue on new master ====================================== 2016-08-25 09:02:02,128 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 83.5. Found ====================================== ====================================== 84.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:02,147 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) is shutting down 2016-08-25 09:02:02,148 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:02,151 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,153 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:02,154 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250054 2016-08-25 09:02:02,155 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:02,155 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:02,155 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:02,155 | INFO | Time-limited test | Session: 0x156bec917250054 closed 2016-08-25 09:02:02,155 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:02,155 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:02,155 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55566 which had sessionid 0x156bec917250054 2016-08-25 09:02:02,266 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:02,266 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:02,266 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,268 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,271 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:02,275 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,277 | INFO | ActiveMQ BrokerService[node-1] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:02,277 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) uptime 0.949 seconds 2016-08-25 09:02:02,277 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) is shutdown Wait for master to start up... 2016-08-25 09:02:02,279 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:02,280 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@c30944a 2016-08-25 09:02:02,280 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:02,281 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:02,281 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:02,281 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55590 2016-08-25 09:02:02,281 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55590 2016-08-25 09:02:02,282 | INFO | SyncThread:0 | Established session 0x156bec917250057 with negotiated timeout 10000 for client /127.0.0.1:55590 2016-08-25 09:02:02,282 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250057, negotiated timeout = 10000 2016-08-25 09:02:02,334 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55591 2016-08-25 09:02:02,337 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,337 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,338 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55591 2016-08-25 09:02:02,338 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:02,339 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55591 2016-08-25 09:02:02,339 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:02:02,352 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:02,393 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:02,394 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000123.sst 2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000129.sst 2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000126.sst 2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000120.sst 2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000130.log 2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000128 2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:02,397 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:02,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:02,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000120.sst 2016-08-25 09:02:02,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000123.sst 2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000128 2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000129.sst 2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000130.log 2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000126.sst 2016-08-25 09:02:02,401 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/8.93 kb and 1/11 files 2016-08-25 09:02:02,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/8.93 kb and 2/11 files 2016-08-25 09:02:02,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/8.93 kb and 3/11 files 2016-08-25 09:02:02,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/8.93 kb and 4/11 files 2016-08-25 09:02:02,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/8.93 kb and 5/11 files 2016-08-25 09:02:02,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/8.93 kb and 6/11 files 2016-08-25 09:02:02,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/8.93 kb and 7/11 files 2016-08-25 09:02:02,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/8.93 kb and 8/11 files 2016-08-25 09:02:02,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.50/8.93 kb and 9/11 files 2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.99/8.93 kb and 10/11 files 2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/8.93 kb and 1/11 files 2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.93/8.93 kb and 11/11 files 2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/8.93 kb and 2/11 files 2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:02,407 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/8.93 kb and 3/11 files 2016-08-25 09:02:02,407 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/8.93 kb and 4/11 files 2016-08-25 09:02:02,408 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/8.93 kb and 5/11 files 2016-08-25 09:02:02,409 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:02,409 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.44/8.93 kb and 6/11 files 2016-08-25 09:02:02,410 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.93/8.93 kb and 7/11 files 2016-08-25 09:02:02,410 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) is starting 2016-08-25 09:02:02,411 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.83/8.93 kb and 8/11 files 2016-08-25 09:02:02,412 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.84/8.93 kb and 9/11 files 2016-08-25 09:02:02,412 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.03/8.93 kb and 10/11 files 2016-08-25 09:02:02,413 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.93/8.93 kb and 11/11 files 2016-08-25 09:02:02,413 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:02,414 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55596 2016-08-25 09:02:02,414 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:02,414 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) started 2016-08-25 09:02:02,414 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:02,414 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:02,415 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 84.4. Browse the queue on new master ====================================== 2016-08-25 09:02:02,418 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 84.5. Found ====================================== ====================================== 85.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:02,432 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) is shutting down 2016-08-25 09:02:02,433 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:02,436 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,437 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:02,437 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:02,437 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:02,437 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:02,437 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:02,438 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250055 2016-08-25 09:02:02,439 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55574 which had sessionid 0x156bec917250055 2016-08-25 09:02:02,439 | INFO | Time-limited test | Session: 0x156bec917250055 closed 2016-08-25 09:02:02,439 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:02,546 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:02,546 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) uptime 0.884 seconds 2016-08-25 09:02:02,546 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) is shutdown Wait for master to start up... 2016-08-25 09:02:02,548 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:02,556 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@645abf3c 2016-08-25 09:02:02,556 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:02,557 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:02,558 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:02,558 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55598 2016-08-25 09:02:02,558 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55598 2016-08-25 09:02:02,559 | INFO | SyncThread:0 | Established session 0x156bec917250058 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55598 2016-08-25 09:02:02,559 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250058, negotiated timeout = 10000 2016-08-25 09:02:02,568 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,623 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:02,625 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,632 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:02,632 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,636 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:02,639 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,702 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55599 2016-08-25 09:02:02,705 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,705 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:02,707 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55599 2016-08-25 09:02:02,707 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:02,707 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55599 2016-08-25 09:02:02,708 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:02:02,769 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:02,769 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:02,774 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:02,775 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000126.sst 2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000129.sst 2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000123.sst 2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000132.sst 2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000131 2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000133.log 2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:02,778 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000123.sst 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000129.sst 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000120.sst 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000126.sst 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000117.sst 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000132.sst 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000131 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000111.sst 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000133.log 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000114.sst 2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000109.sst 2016-08-25 09:02:02,783 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.94/9.88 kb and 1/12 files 2016-08-25 09:02:02,783 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.84/9.88 kb and 2/12 files 2016-08-25 09:02:02,784 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.74/9.88 kb and 3/12 files 2016-08-25 09:02:02,784 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.64/9.88 kb and 4/12 files 2016-08-25 09:02:02,785 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/9.88 kb and 5/12 files 2016-08-25 09:02:02,785 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.55/9.88 kb and 6/12 files 2016-08-25 09:02:02,786 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.45/9.88 kb and 7/12 files 2016-08-25 09:02:02,786 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.00/9.88 kb and 8/12 files 2016-08-25 09:02:02,787 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.89/9.88 kb and 9/12 files 2016-08-25 09:02:02,787 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.79/9.88 kb and 10/12 files 2016-08-25 09:02:02,788 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.98/9.88 kb and 11/12 files 2016-08-25 09:02:02,788 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.88/9.88 kb and 12/12 files 2016-08-25 09:02:02,788 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:02,789 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.88 kb and 1/12 files 2016-08-25 09:02:02,789 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/9.88 kb and 2/12 files 2016-08-25 09:02:02,791 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/9.88 kb and 3/12 files 2016-08-25 09:02:02,791 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/9.88 kb and 4/12 files 2016-08-25 09:02:02,792 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/9.88 kb and 5/12 files 2016-08-25 09:02:02,792 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/9.88 kb and 6/12 files 2016-08-25 09:02:02,793 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:02,793 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.94/9.88 kb and 7/12 files 2016-08-25 09:02:02,794 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.95/9.88 kb and 8/12 files 2016-08-25 09:02:02,794 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) is starting 2016-08-25 09:02:02,794 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.85/9.88 kb and 9/12 files 2016-08-25 09:02:02,796 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.04/9.88 kb and 10/12 files 2016-08-25 09:02:02,797 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.94/9.88 kb and 11/12 files 2016-08-25 09:02:02,798 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.88/9.88 kb and 12/12 files 2016-08-25 09:02:02,798 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:02,798 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55604 2016-08-25 09:02:02,798 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:02,799 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) started 2016-08-25 09:02:02,799 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:02,799 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:02,799 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 85.4. Browse the queue on new master ====================================== 2016-08-25 09:02:02,803 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 85.5. Found ====================================== ====================================== 86.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:02,818 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) is shutting down 2016-08-25 09:02:02,819 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:02,823 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,825 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:02,825 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250056 2016-08-25 09:02:02,826 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55582 which had sessionid 0x156bec917250056 2016-08-25 09:02:02,826 | INFO | Time-limited test | Session: 0x156bec917250056 closed 2016-08-25 09:02:02,826 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:02,829 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:02,829 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:02,829 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:02,829 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:02,940 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:02,941 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) uptime 1.027 seconds 2016-08-25 09:02:02,941 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) is shutdown Wait for master to start up... 2016-08-25 09:02:02,943 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:02,947 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@322c2ddd 2016-08-25 09:02:02,947 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:02,948 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:02,949 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:02,949 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55606 2016-08-25 09:02:02,949 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55606 2016-08-25 09:02:02,950 | INFO | SyncThread:0 | Established session 0x156bec917250059 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55606 2016-08-25 09:02:02,950 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250059, negotiated timeout = 10000 2016-08-25 09:02:02,960 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,969 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:02,969 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,972 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:02,979 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:02,982 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:02,985 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,054 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55607 2016-08-25 09:02:03,061 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,061 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,062 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55607 2016-08-25 09:02:03,062 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55607 2016-08-25 09:02:03,062 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:03,062 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:02:03,073 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:03,076 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:03,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000131 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000120.sst 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000133.log 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000114.sst 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000117.sst 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000132.sst 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000111.sst 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000123.sst 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000126.sst 2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000129.sst 2016-08-25 09:02:03,086 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.54/9.88 kb and 1/12 files 2016-08-25 09:02:03,087 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.44/9.88 kb and 2/12 files 2016-08-25 09:02:03,087 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.63/9.88 kb and 3/12 files 2016-08-25 09:02:03,088 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.57/9.88 kb and 4/12 files 2016-08-25 09:02:03,088 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.47/9.88 kb and 5/12 files 2016-08-25 09:02:03,088 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.48/9.88 kb and 6/12 files 2016-08-25 09:02:03,089 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.38/9.88 kb and 7/12 files 2016-08-25 09:02:03,089 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.28/9.88 kb and 8/12 files 2016-08-25 09:02:03,089 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.18/9.88 kb and 9/12 files 2016-08-25 09:02:03,090 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.08/9.88 kb and 10/12 files 2016-08-25 09:02:03,090 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.98/9.88 kb and 11/12 files 2016-08-25 09:02:03,090 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.88/9.88 kb and 12/12 files 2016-08-25 09:02:03,091 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:03,093 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:03,094 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) is starting 2016-08-25 09:02:03,098 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55611 2016-08-25 09:02:03,098 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:03,098 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) started 2016-08-25 09:02:03,098 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:03,098 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:03,098 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-1 ====================================== 86.4. Browse the queue on new master ====================================== ====================================== 86.5. Found ====================================== ====================================== 87.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:03,114 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) is shutting down 2016-08-25 09:02:03,114 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:03,117 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,118 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,118 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:03,119 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:03,119 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250057 2016-08-25 09:02:03,120 | INFO | Time-limited test | Session: 0x156bec917250057 closed 2016-08-25 09:02:03,120 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55590 which had sessionid 0x156bec917250057 2016-08-25 09:02:03,120 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:03,138 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,198 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:03,198 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,200 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,228 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:03,228 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) uptime 0.951 seconds 2016-08-25 09:02:03,228 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) is shutdown Wait for master to start up... 2016-08-25 09:02:03,230 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:03,232 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1d3e3a05 2016-08-25 09:02:03,232 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:03,233 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:03,233 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:03,234 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55613 2016-08-25 09:02:03,234 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55613 2016-08-25 09:02:03,234 | INFO | SyncThread:0 | Established session 0x156bec91725005a with negotiated timeout 10000 for client /127.0.0.1:55613 2016-08-25 09:02:03,235 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725005a, negotiated timeout = 10000 2016-08-25 09:02:03,241 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,245 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:03,249 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:03,253 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,276 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55614 2016-08-25 09:02:03,280 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,280 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,280 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55614 2016-08-25 09:02:03,280 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:02:03,281 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55614 2016-08-25 09:02:03,281 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:03,347 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:03,347 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000136.log 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000136.log 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000134 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000134 2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:03,352 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/2.86 kb and 1/4 files 2016-08-25 09:02:03,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.18/2.86 kb and 1/4 files 2016-08-25 09:02:03,353 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.12/2.86 kb and 2/4 files 2016-08-25 09:02:03,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.20/2.86 kb and 2/4 files 2016-08-25 09:02:03,353 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.85/2.86 kb and 3/4 files 2016-08-25 09:02:03,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.14/2.86 kb and 3/4 files 2016-08-25 09:02:03,354 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.86/2.86 kb and 4/4 files 2016-08-25 09:02:03,354 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.86/2.86 kb and 4/4 files 2016-08-25 09:02:03,354 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:03,354 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:03,358 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:03,358 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:03,359 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) is starting 2016-08-25 09:02:03,363 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55619 2016-08-25 09:02:03,363 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:03,363 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) started 2016-08-25 09:02:03,364 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:03,364 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:03,364 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 87.4. Browse the queue on new master ====================================== ====================================== 87.5. Found ====================================== ====================================== 88.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:03,383 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) is shutting down 2016-08-25 09:02:03,384 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:03,388 | INFO | Starting broker node: node-2-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:02:03,388 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,390 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,390 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:03,391 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,391 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:03,393 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:03,394 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:03,394 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,394 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:03,394 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250059 2016-08-25 09:02:03,395 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,395 | INFO | Time-limited test | Session: 0x156bec917250059 closed 2016-08-25 09:02:03,395 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55606 which had sessionid 0x156bec917250059 2016-08-25 09:02:03,395 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:03,397 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,401 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:03,404 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,425 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55621 2016-08-25 09:02:03,430 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,431 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55621 2016-08-25 09:02:03,432 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:02:03,444 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:03,446 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:03,446 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:03,446 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000134 2016-08-25 09:02:03,447 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000136.log 2016-08-25 09:02:03,447 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:03,449 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/2.86 kb and 1/4 files 2016-08-25 09:02:03,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.74/2.86 kb and 2/4 files 2016-08-25 09:02:03,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.92/2.86 kb and 3/4 files 2016-08-25 09:02:03,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.86/2.86 kb and 4/4 files 2016-08-25 09:02:03,450 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:03,453 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:03,454 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) is starting 2016-08-25 09:02:03,457 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55624 2016-08-25 09:02:03,457 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:03,457 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) started 2016-08-25 09:02:03,457 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:03,457 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:03,457 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:03,514 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:03,514 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) uptime 0.573 seconds 2016-08-25 09:02:03,514 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) is shutdown Wait for master to start up... Master started: node-2 ====================================== 88.4. Browse the queue on new master ====================================== 2016-08-25 09:02:03,516 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:03,518 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3ab65057 2016-08-25 09:02:03,518 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:03,518 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:03,519 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:03,519 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55626 2016-08-25 09:02:03,519 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55626 2016-08-25 09:02:03,519 | INFO | SyncThread:0 | Established session 0x156bec91725005b with negotiated timeout 10000 for client /127.0.0.1:55626 2016-08-25 09:02:03,520 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725005b, negotiated timeout = 10000 ====================================== 88.5. Found ====================================== ====================================== 89.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:03,527 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) is shutting down 2016-08-25 09:02:03,527 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:03,528 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,530 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55621 2016-08-25 09:02:03,530 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:03,530 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,531 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,531 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:03,531 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:03,531 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,534 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:03,534 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,534 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250058 2016-08-25 09:02:03,534 | INFO | Time-limited test | Session: 0x156bec917250058 closed 2016-08-25 09:02:03,535 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:03,535 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55598 which had sessionid 0x156bec917250058 2016-08-25 09:02:03,547 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:02:03,547 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,550 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:03,553 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,581 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55628 2016-08-25 09:02:03,584 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,585 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55628 2016-08-25 09:02:03,585 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:02:03,602 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:03,604 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000138 2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000140.log 2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:03,608 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/3.22 kb and 1/5 files 2016-08-25 09:02:03,608 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/3.22 kb and 2/5 files 2016-08-25 09:02:03,609 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.02/3.22 kb and 3/5 files 2016-08-25 09:02:03,609 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.20/3.22 kb and 4/5 files 2016-08-25 09:02:03,610 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.22/3.22 kb and 5/5 files 2016-08-25 09:02:03,610 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:03,612 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:03,612 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) is starting 2016-08-25 09:02:03,615 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55631 2016-08-25 09:02:03,615 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:03,615 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) started 2016-08-25 09:02:03,615 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:03,615 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:03,616 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:03,636 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:03,637 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) uptime 1.091 seconds 2016-08-25 09:02:03,637 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) is shutdown Wait for master to start up... Master started: node-1 ====================================== 89.4. Browse the queue on new master ====================================== 2016-08-25 09:02:03,639 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:03,642 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@47a034e7 2016-08-25 09:02:03,642 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:03,643 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:03,644 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:03,644 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55633 2016-08-25 09:02:03,645 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55633 2016-08-25 09:02:03,645 | INFO | SyncThread:0 | Established session 0x156bec91725005c with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55633 2016-08-25 09:02:03,645 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725005c, negotiated timeout = 10000 ====================================== 89.5. Found ====================================== ====================================== 90.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:03,652 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) is shutting down 2016-08-25 09:02:03,652 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:03,656 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,661 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55628 2016-08-25 09:02:03,661 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:03,662 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:03,662 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,663 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,663 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:03,663 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005a 2016-08-25 09:02:03,664 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,664 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:03,664 | INFO | Time-limited test | Session: 0x156bec91725005a closed 2016-08-25 09:02:03,664 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:03,665 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55613 which had sessionid 0x156bec91725005a 2016-08-25 09:02:03,666 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,666 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:03,666 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,669 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:03,673 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,700 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55635 2016-08-25 09:02:03,703 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,703 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55635 2016-08-25 09:02:03,703 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:03,714 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:03,717 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:03,718 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:03,718 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000140.log 2016-08-25 09:02:03,719 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:03,719 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000138 2016-08-25 09:02:03,719 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:03,721 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/3.22 kb and 1/5 files 2016-08-25 09:02:03,721 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.12/3.22 kb and 2/5 files 2016-08-25 09:02:03,722 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.14/3.22 kb and 3/5 files 2016-08-25 09:02:03,722 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.32/3.22 kb and 4/5 files 2016-08-25 09:02:03,723 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.22/3.22 kb and 5/5 files 2016-08-25 09:02:03,723 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:03,725 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:03,727 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) is starting 2016-08-25 09:02:03,731 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55638 2016-08-25 09:02:03,732 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:03,732 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) started 2016-08-25 09:02:03,732 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:03,732 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:03,732 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:03,792 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:03,792 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) uptime 0.563 seconds 2016-08-25 09:02:03,793 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) is shutdown Wait for master to start up... Master started: node-3 ====================================== 90.4. Browse the queue on new master ====================================== 2016-08-25 09:02:03,794 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:03,796 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7371ece 2016-08-25 09:02:03,796 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:03,797 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:03,798 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:03,798 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55640 2016-08-25 09:02:03,798 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55640 2016-08-25 09:02:03,798 | INFO | SyncThread:0 | Established session 0x156bec91725005d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55640 2016-08-25 09:02:03,798 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725005d, negotiated timeout = 10000 2016-08-25 09:02:03,805 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. ====================================== 90.5. Found ====================================== ====================================== 91.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:03,806 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) is shutting down 2016-08-25 09:02:03,806 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:03,806 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55635 2016-08-25 09:02:03,807 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:03,811 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,816 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:03,816 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005b 2016-08-25 09:02:03,817 | INFO | Time-limited test | Session: 0x156bec91725005b closed 2016-08-25 09:02:03,817 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55626 which had sessionid 0x156bec91725005b 2016-08-25 09:02:03,817 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:03,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:03,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,839 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:03,840 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,859 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:02:03,860 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:03,860 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:03,863 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:03,866 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,910 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55642 2016-08-25 09:02:03,913 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,914 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55642 2016-08-25 09:02:03,914 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:03,920 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:03,921 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) uptime 0.406 seconds 2016-08-25 09:02:03,921 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) is shutdown Wait for master to start up... 2016-08-25 09:02:03,923 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:03,925 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6488a31b 2016-08-25 09:02:03,926 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:03,926 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:03,927 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:03,927 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55644 2016-08-25 09:02:03,927 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55644 2016-08-25 09:02:03,928 | INFO | SyncThread:0 | Established session 0x156bec91725005e with negotiated timeout 10000 for client /127.0.0.1:55644 2016-08-25 09:02:03,928 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725005e, negotiated timeout = 10000 2016-08-25 09:02:03,941 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:03,942 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55642 2016-08-25 09:02:03,942 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:03,947 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:03,961 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:03,962 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000143.log 2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000141 2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:03,967 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000141 2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000143.log 2016-08-25 09:02:03,969 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/4.17 kb and 1/6 files 2016-08-25 09:02:03,969 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.12/4.17 kb and 2/6 files 2016-08-25 09:02:03,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.02/4.17 kb and 3/6 files 2016-08-25 09:02:03,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.04/4.17 kb and 4/6 files 2016-08-25 09:02:03,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.27/4.17 kb and 5/6 files 2016-08-25 09:02:03,971 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files 2016-08-25 09:02:03,971 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:03,972 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/4.17 kb and 1/6 files 2016-08-25 09:02:03,972 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.91/4.17 kb and 2/6 files 2016-08-25 09:02:03,973 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.81/4.17 kb and 3/6 files 2016-08-25 09:02:03,973 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.75/4.17 kb and 4/6 files 2016-08-25 09:02:03,974 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.98/4.17 kb and 5/6 files 2016-08-25 09:02:03,974 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:03,975 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files 2016-08-25 09:02:03,975 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:02:03,976 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) is starting 2016-08-25 09:02:03,979 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:03,981 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55648 2016-08-25 09:02:03,982 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:03,982 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) started 2016-08-25 09:02:03,982 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:03,982 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:03,982 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 91.4. Browse the queue on new master ====================================== ====================================== 91.5. Found ====================================== ====================================== 92.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:04,002 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) is shutting down 2016-08-25 09:02:04,003 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:04,009 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,014 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,015 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:04,015 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,016 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:04,020 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:04,021 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005c 2016-08-25 09:02:04,021 | INFO | Time-limited test | Session: 0x156bec91725005c closed 2016-08-25 09:02:04,021 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:04,021 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55633 which had sessionid 0x156bec91725005c 2016-08-25 09:02:04,050 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:04,050 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:04,050 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,052 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,053 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:04,055 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,100 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55650 2016-08-25 09:02:04,103 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,104 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55650 2016-08-25 09:02:04,104 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:04,131 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:04,134 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,134 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:04,134 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:04,135 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:04,135 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000141 2016-08-25 09:02:04,135 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000143.log 2016-08-25 09:02:04,135 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:04,140 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/4.17 kb and 1/6 files 2016-08-25 09:02:04,141 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.84/4.17 kb and 2/6 files 2016-08-25 09:02:04,141 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/4.17 kb and 3/6 files 2016-08-25 09:02:04,142 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.09/4.17 kb and 4/6 files 2016-08-25 09:02:04,143 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.27/4.17 kb and 5/6 files 2016-08-25 09:02:04,143 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files 2016-08-25 09:02:04,143 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:04,146 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:04,147 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) is starting 2016-08-25 09:02:04,150 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55653 2016-08-25 09:02:04,150 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:04,151 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) started 2016-08-25 09:02:04,151 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:04,151 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:02:04,151 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb 2016-08-25 09:02:04,160 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:04,161 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) uptime 0.523 seconds 2016-08-25 09:02:04,161 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) is shutdown Wait for master to start up... Master started: node-1 ====================================== 92.4. Browse the queue on new master ====================================== 2016-08-25 09:02:04,163 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:04,166 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@609f293a 2016-08-25 09:02:04,166 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:04,166 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:04,167 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:04,167 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55655 2016-08-25 09:02:04,167 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55655 2016-08-25 09:02:04,168 | INFO | SyncThread:0 | Established session 0x156bec91725005f with negotiated timeout 10000 for client /127.0.0.1:55655 2016-08-25 09:02:04,168 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725005f, negotiated timeout = 10000 ====================================== 92.5. Found ====================================== ====================================== 93.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:04,177 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) is shutting down 2016-08-25 09:02:04,177 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:04,179 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,180 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:04,180 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005d 2016-08-25 09:02:04,181 | INFO | Time-limited test | Session: 0x156bec91725005d closed 2016-08-25 09:02:04,181 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:04,181 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55640 which had sessionid 0x156bec91725005d 2016-08-25 09:02:04,181 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,181 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:04,187 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,189 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55650 2016-08-25 09:02:04,189 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:04,189 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave stopped 2016-08-25 09:02:04,191 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,237 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:04,237 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,239 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:04,243 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,283 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55657 2016-08-25 09:02:04,284 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:04,285 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) uptime 0.492 seconds 2016-08-25 09:02:04,285 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) is shutdown Wait for master to start up... 2016-08-25 09:02:04,287 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:04,290 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@20bb0006 2016-08-25 09:02:04,290 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:04,290 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,290 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Attaching to master: tcp://localhost:55657 2016-08-25 09:02:04,291 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave started 2016-08-25 09:02:04,291 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:04,291 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:04,292 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55659 2016-08-25 09:02:04,292 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55659 2016-08-25 09:02:04,293 | INFO | SyncThread:0 | Established session 0x156bec917250060 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55659 2016-08-25 09:02:04,293 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250060, negotiated timeout = 10000 2016-08-25 09:02:04,302 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,303 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55657 2016-08-25 09:02:04,303 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:04,333 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:04,337 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000146.log 2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000144 2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:04,344 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/5.12 kb and 1/7 files 2016-08-25 09:02:04,345 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,345 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.20/5.12 kb and 2/7 files 2016-08-25 09:02:04,345 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:04,345 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.49/5.12 kb and 3/7 files 2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000146.log 2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.38/5.12 kb and 4/7 files 2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000144 2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:04,347 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.28/5.12 kb and 5/7 files 2016-08-25 09:02:04,347 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.18/5.12 kb and 6/7 files 2016-08-25 09:02:04,348 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files 2016-08-25 09:02:04,348 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:04,351 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:04,351 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/5.12 kb and 1/7 files 2016-08-25 09:02:04,351 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.20/5.12 kb and 2/7 files 2016-08-25 09:02:04,352 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) is starting 2016-08-25 09:02:04,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.10/5.12 kb and 3/7 files 2016-08-25 09:02:04,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.00/5.12 kb and 4/7 files 2016-08-25 09:02:04,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.28/5.12 kb and 5/7 files 2016-08-25 09:02:04,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.18/5.12 kb and 6/7 files 2016-08-25 09:02:04,354 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files 2016-08-25 09:02:04,354 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:04,355 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55663 2016-08-25 09:02:04,355 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:04,356 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) started 2016-08-25 09:02:04,356 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:04,356 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:04,356 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:04,357 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d Master started: node-3 ====================================== 93.4. Browse the queue on new master ====================================== ====================================== 93.5. Found ====================================== ====================================== 94.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:04,371 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) is shutting down 2016-08-25 09:02:04,371 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:04,374 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,376 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:04,377 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,377 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,377 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:04,377 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:04,377 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005e 2016-08-25 09:02:04,377 | INFO | Time-limited test | Session: 0x156bec91725005e closed 2016-08-25 09:02:04,377 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:04,377 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55644 which had sessionid 0x156bec91725005e 2016-08-25 09:02:04,440 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:04,443 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,444 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:04,444 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,449 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:04,455 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,489 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:04,489 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) uptime 0.567 seconds 2016-08-25 09:02:04,489 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) is shutdown Wait for master to start up... 2016-08-25 09:02:04,491 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:04,493 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@61c28cfb 2016-08-25 09:02:04,493 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:04,494 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:04,494 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:04,494 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55665 2016-08-25 09:02:04,494 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55665 2016-08-25 09:02:04,495 | INFO | SyncThread:0 | Established session 0x156bec917250061 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55665 2016-08-25 09:02:04,495 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250061, negotiated timeout = 10000 2016-08-25 09:02:04,498 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Master started: tcp://localhost:55666 2016-08-25 09:02:04,505 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,505 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,507 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55666 2016-08-25 09:02:04,507 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:02:04,507 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55666 2016-08-25 09:02:04,507 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:04,526 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:04,531 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,531 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:04,531 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:04,531 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000144 2016-08-25 09:02:04,532 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:04,532 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:04,532 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:04,532 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000146.log 2016-08-25 09:02:04,537 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/5.12 kb and 1/7 files 2016-08-25 09:02:04,537 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.79/5.12 kb and 2/7 files 2016-08-25 09:02:04,538 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/5.12 kb and 3/7 files 2016-08-25 09:02:04,538 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.02/5.12 kb and 4/7 files 2016-08-25 09:02:04,539 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.92/5.12 kb and 5/7 files 2016-08-25 09:02:04,539 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.93/5.12 kb and 6/7 files 2016-08-25 09:02:04,540 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files 2016-08-25 09:02:04,540 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:04,542 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:04,543 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) is starting 2016-08-25 09:02:04,547 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55670 2016-08-25 09:02:04,547 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:04,547 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) started 2016-08-25 09:02:04,547 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:04,548 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:04,548 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 94.4. Browse the queue on new master ====================================== 2016-08-25 09:02:04,558 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d ====================================== 94.5. Found ====================================== ====================================== 95.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:04,563 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) is shutting down 2016-08-25 09:02:04,563 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:04,565 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,566 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:04,566 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005f 2016-08-25 09:02:04,567 | INFO | Time-limited test | Session: 0x156bec91725005f closed 2016-08-25 09:02:04,567 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55655 which had sessionid 0x156bec91725005f 2016-08-25 09:02:04,567 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:04,567 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000144 2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000146.log 2016-08-25 09:02:04,571 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,571 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,571 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,571 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:04,572 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:04,609 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:04,609 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,611 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,666 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:04,669 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:04,671 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:04,672 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) uptime 0.511 seconds 2016-08-25 09:02:04,672 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) is shutdown Wait for master to start up... 2016-08-25 09:02:04,673 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,674 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:04,676 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@62399a23 2016-08-25 09:02:04,676 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:04,677 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:04,677 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:04,677 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55673 2016-08-25 09:02:04,677 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55673 2016-08-25 09:02:04,678 | INFO | SyncThread:0 | Established session 0x156bec917250062 with negotiated timeout 10000 for client /127.0.0.1:55673 2016-08-25 09:02:04,678 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250062, negotiated timeout = 10000 2016-08-25 09:02:04,725 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55674 2016-08-25 09:02:04,728 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,728 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,729 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55674 2016-08-25 09:02:04,729 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:02:04,730 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55674 2016-08-25 09:02:04,730 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:04,763 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:04,764 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:04,766 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,766 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000149.log 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000147 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000147 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000149.log 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:04,770 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/6.07 kb and 1/8 files 2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/6.07 kb and 1/8 files 2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.08/6.07 kb and 2/8 files 2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.24/6.07 kb and 2/8 files 2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.25/6.07 kb and 3/8 files 2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/6.07 kb and 3/8 files 2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.15/6.07 kb and 4/8 files 2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.92/6.07 kb and 4/8 files 2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.09/6.07 kb and 5/8 files 2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.26/6.07 kb and 5/8 files 2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.27/6.07 kb and 6/8 files 2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.15/6.07 kb and 6/8 files 2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.17/6.07 kb and 7/8 files 2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.05/6.07 kb and 7/8 files 2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files 2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:02:04,774 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files 2016-08-25 09:02:04,774 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:04,776 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:04,776 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:04,777 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) is starting 2016-08-25 09:02:04,781 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55679 2016-08-25 09:02:04,781 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:04,781 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) started 2016-08-25 09:02:04,781 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:04,781 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:04,781 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 95.4. Browse the queue on new master ====================================== ====================================== 95.5. Found ====================================== ====================================== 96.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:04,796 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) is shutting down 2016-08-25 09:02:04,796 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:04,800 | INFO | Starting broker node: node-1-EventThread | Previously elected master is not online, staring new election 2016-08-25 09:02:04,800 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,801 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,802 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:04,802 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:04,803 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:04,803 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:04,803 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250061 2016-08-25 09:02:04,804 | INFO | Time-limited test | Session: 0x156bec917250061 closed 2016-08-25 09:02:04,804 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:04,804 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55665 which had sessionid 0x156bec917250061 2016-08-25 09:02:04,805 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,892 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:04,892 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:04,892 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,895 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:04,898 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:04,901 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,913 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:04,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) uptime 0.423 seconds 2016-08-25 09:02:04,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) is shutdown Wait for master to start up... 2016-08-25 09:02:04,914 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:04,916 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3cf21ac1 2016-08-25 09:02:04,916 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:04,917 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:04,917 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:04,917 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55681 2016-08-25 09:02:04,917 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55681 2016-08-25 09:02:04,918 | INFO | SyncThread:0 | Established session 0x156bec917250063 with negotiated timeout 10000 for client /127.0.0.1:55681 2016-08-25 09:02:04,918 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250063, negotiated timeout = 10000 2016-08-25 09:02:04,950 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55682 2016-08-25 09:02:04,954 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,954 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:04,955 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55682 2016-08-25 09:02:04,955 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:04,956 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55682 2016-08-25 09:02:04,956 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:02:04,967 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:04,993 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,993 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:04,993 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:04,993 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000149.log 2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000147 2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000149.log 2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000147 2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:04,997 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:04,997 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:04,997 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:04,998 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/6.07 kb and 1/8 files 2016-08-25 09:02:04,998 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.84/6.07 kb and 2/8 files 2016-08-25 09:02:04,999 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.73/6.07 kb and 3/8 files 2016-08-25 09:02:04,999 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.63/6.07 kb and 4/8 files 2016-08-25 09:02:05,000 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.53/6.07 kb and 5/8 files 2016-08-25 09:02:05,000 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.54/6.07 kb and 6/8 files 2016-08-25 09:02:05,001 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/6.07 kb and 7/8 files 2016-08-25 09:02:05,001 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.18/6.07 kb and 1/8 files 2016-08-25 09:02:05,001 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files 2016-08-25 09:02:05,001 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:05,002 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.20/6.07 kb and 2/8 files 2016-08-25 09:02:05,002 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.54/6.07 kb and 3/8 files 2016-08-25 09:02:05,003 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.48/6.07 kb and 4/8 files 2016-08-25 09:02:05,003 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.38/6.07 kb and 5/8 files 2016-08-25 09:02:05,004 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:05,004 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.27/6.07 kb and 6/8 files 2016-08-25 09:02:05,004 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.17/6.07 kb and 7/8 files 2016-08-25 09:02:05,005 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) is starting 2016-08-25 09:02:05,005 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files 2016-08-25 09:02:05,005 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:05,008 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:05,009 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55687 2016-08-25 09:02:05,009 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:05,009 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) started 2016-08-25 09:02:05,009 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:05,009 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:05,009 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-1 ====================================== 96.4. Browse the queue on new master ====================================== ====================================== 96.5. Found ====================================== ====================================== 97.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:05,025 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) is shutting down 2016-08-25 09:02:05,025 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:05,028 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,029 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:05,029 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250060 2016-08-25 09:02:05,029 | INFO | Time-limited test | Session: 0x156bec917250060 closed 2016-08-25 09:02:05,029 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:05,029 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55659 which had sessionid 0x156bec917250060 2016-08-25 09:02:05,033 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:05,033 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:05,033 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:05,033 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:05,136 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:05,136 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) uptime 0.851 seconds 2016-08-25 09:02:05,137 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) is shutdown Wait for master to start up... 2016-08-25 09:02:05,139 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:05,139 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:05,139 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,139 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:05,141 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,141 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3407a533 2016-08-25 09:02:05,141 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:05,142 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:05,143 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:05,143 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55689 2016-08-25 09:02:05,143 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55689 2016-08-25 09:02:05,143 | INFO | SyncThread:0 | Established session 0x156bec917250064 with negotiated timeout 10000 for client /127.0.0.1:55689 2016-08-25 09:02:05,143 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250064, negotiated timeout = 10000 2016-08-25 09:02:05,144 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:05,147 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,201 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55690 2016-08-25 09:02:05,205 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,205 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,206 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55690 2016-08-25 09:02:05,206 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:05,206 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55690 2016-08-25 09:02:05,206 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:02:05,240 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:05,278 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:05,280 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000151.sst 2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000150 2016-08-25 09:02:05,282 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000152.log 2016-08-25 09:02:05,283 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:05,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000151.sst 2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000150 2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000152.log 2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:05,287 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/7.02 kb and 1/9 files 2016-08-25 09:02:05,287 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/7.02 kb and 1/9 files 2016-08-25 09:02:05,288 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/7.02 kb and 2/9 files 2016-08-25 09:02:05,288 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/7.02 kb and 2/9 files 2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.69/7.02 kb and 3/9 files 2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.85/7.02 kb and 3/9 files 2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/7.02 kb and 4/9 files 2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.75/7.02 kb and 4/9 files 2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.98/7.02 kb and 5/9 files 2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.65/7.02 kb and 5/9 files 2016-08-25 09:02:05,290 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.88/7.02 kb and 6/9 files 2016-08-25 09:02:05,290 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.54/7.02 kb and 6/9 files 2016-08-25 09:02:05,290 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.82/7.02 kb and 7/9 files 2016-08-25 09:02:05,290 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.44/7.02 kb and 7/9 files 2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.00/7.02 kb and 8/9 files 2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.83/7.02 kb and 8/9 files 2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files 2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files 2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:05,294 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:05,294 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:05,295 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) is starting 2016-08-25 09:02:05,299 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55695 2016-08-25 09:02:05,299 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:05,299 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) started 2016-08-25 09:02:05,299 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:05,299 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:05,299 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 97.4. Browse the queue on new master ====================================== ====================================== 97.5. Found ====================================== ====================================== 98.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:05,314 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) is shutting down 2016-08-25 09:02:05,314 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:05,317 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,318 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:05,318 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:05,318 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:05,318 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:05,318 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:05,319 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250062 2016-08-25 09:02:05,319 | INFO | Time-limited test | Session: 0x156bec917250062 closed 2016-08-25 09:02:05,319 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55673 which had sessionid 0x156bec917250062 2016-08-25 09:02:05,319 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:05,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:05,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,415 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:05,417 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,420 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:05,424 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,436 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:05,437 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) uptime 0.765 seconds 2016-08-25 09:02:05,437 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) is shutdown Wait for master to start up... 2016-08-25 09:02:05,440 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:05,442 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@227ec5dd 2016-08-25 09:02:05,442 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:05,443 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:05,443 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:05,443 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55697 2016-08-25 09:02:05,444 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55697 2016-08-25 09:02:05,446 | INFO | SyncThread:0 | Established session 0x156bec917250065 with negotiated timeout 10000 for client /127.0.0.1:55697 2016-08-25 09:02:05,446 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250065, negotiated timeout = 10000 2016-08-25 09:02:05,487 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55698 2016-08-25 09:02:05,492 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,492 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,494 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55698 2016-08-25 09:02:05,494 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:05,494 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55698 2016-08-25 09:02:05,495 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:02:05,510 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:05,540 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:05,542 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:05,542 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000151.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000150 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000152.log 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000150 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000152.log 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000151.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:05,544 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:05,547 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/7.02 kb and 1/9 files 2016-08-25 09:02:05,547 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/7.02 kb and 1/9 files 2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.79/7.02 kb and 2/9 files 2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.91/7.02 kb and 2/9 files 2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.73/7.02 kb and 3/9 files 2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.30/7.02 kb and 3/9 files 2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.75/7.02 kb and 4/9 files 2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.20/7.02 kb and 4/9 files 2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.10/7.02 kb and 5/9 files 2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.93/7.02 kb and 5/9 files 2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.00/7.02 kb and 6/9 files 2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.32/7.02 kb and 6/9 files 2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.18/7.02 kb and 7/9 files 2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.22/7.02 kb and 7/9 files 2016-08-25 09:02:05,550 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.08/7.02 kb and 8/9 files 2016-08-25 09:02:05,550 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.12/7.02 kb and 8/9 files 2016-08-25 09:02:05,551 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files 2016-08-25 09:02:05,551 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:05,551 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files 2016-08-25 09:02:05,551 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:05,554 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:05,554 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:05,555 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) is starting 2016-08-25 09:02:05,558 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55703 2016-08-25 09:02:05,558 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:05,559 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) started 2016-08-25 09:02:05,559 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:05,559 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:05,559 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 98.4. Browse the queue on new master ====================================== ====================================== 98.5. Found ====================================== ====================================== 99.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:05,574 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) is shutting down 2016-08-25 09:02:05,575 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:05,577 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,578 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:05,578 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:05,578 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:05,578 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:05,579 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:05,579 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250063 2016-08-25 09:02:05,580 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55681 which had sessionid 0x156bec917250063 2016-08-25 09:02:05,580 | INFO | Time-limited test | Session: 0x156bec917250063 closed 2016-08-25 09:02:05,580 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:05,688 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:05,689 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) uptime 0.776 seconds 2016-08-25 09:02:05,689 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) is shutdown Wait for master to start up... 2016-08-25 09:02:05,692 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:05,695 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@769bca67 2016-08-25 09:02:05,695 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:05,696 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:05,697 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:05,697 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55705 2016-08-25 09:02:05,698 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55705 2016-08-25 09:02:05,698 | INFO | SyncThread:0 | Established session 0x156bec917250066 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55705 2016-08-25 09:02:05,699 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250066, negotiated timeout = 10000 2016-08-25 09:02:05,708 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,720 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:05,721 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,726 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:05,726 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,729 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:05,733 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,797 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55706 2016-08-25 09:02:05,816 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,817 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:05,818 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55706 2016-08-25 09:02:05,818 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:05,819 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55706 2016-08-25 09:02:05,819 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:05,859 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:05,864 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:05,869 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:05,870 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:05,870 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:05,870 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000154.sst 2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000155.log 2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000151.sst 2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000153 2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:05,872 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:05,872 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000153 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000155.log 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000154.sst 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000151.sst 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:05,877 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/7.97 kb and 1/10 files 2016-08-25 09:02:05,877 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.84/7.97 kb and 2/10 files 2016-08-25 09:02:05,878 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.73/7.97 kb and 3/10 files 2016-08-25 09:02:05,879 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.63/7.97 kb and 4/10 files 2016-08-25 09:02:05,880 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.65/7.97 kb and 5/10 files 2016-08-25 09:02:05,880 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.44/7.97 kb and 1/10 files 2016-08-25 09:02:05,880 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/7.97 kb and 6/10 files 2016-08-25 09:02:05,880 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.63/7.97 kb and 2/10 files 2016-08-25 09:02:05,881 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/7.97 kb and 7/10 files 2016-08-25 09:02:05,881 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.52/7.97 kb and 3/10 files 2016-08-25 09:02:05,881 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.17/7.97 kb and 8/10 files 2016-08-25 09:02:05,881 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.42/7.97 kb and 4/10 files 2016-08-25 09:02:05,882 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.07/7.97 kb and 9/10 files 2016-08-25 09:02:05,882 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.36/7.97 kb and 5/10 files 2016-08-25 09:02:05,882 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.97/7.97 kb and 10/10 files 2016-08-25 09:02:05,883 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:05,882 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.26/7.97 kb and 6/10 files 2016-08-25 09:02:05,883 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.16/7.97 kb and 7/10 files 2016-08-25 09:02:05,884 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.05/7.97 kb and 8/10 files 2016-08-25 09:02:05,884 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.07/7.97 kb and 9/10 files 2016-08-25 09:02:05,885 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.97/7.97 kb and 10/10 files 2016-08-25 09:02:05,885 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:05,886 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:05,887 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) is starting 2016-08-25 09:02:05,888 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:05,891 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55711 2016-08-25 09:02:05,892 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:05,892 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) started 2016-08-25 09:02:05,892 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:05,892 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:05,892 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-1 ====================================== 99.4. Browse the queue on new master ====================================== ====================================== 99.5. Found ====================================== ====================================== 100.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:05,910 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) is shutting down 2016-08-25 09:02:05,911 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:05,914 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:05,914 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:05,914 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250064 2016-08-25 09:02:05,915 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:05,915 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55689 which had sessionid 0x156bec917250064 2016-08-25 09:02:05,915 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:05,915 | INFO | Time-limited test | Session: 0x156bec917250064 closed 2016-08-25 09:02:05,915 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:05,915 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:05,915 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:06,021 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:06,022 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:06,022 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,024 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,026 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:06,030 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,031 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:06,032 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) uptime 0.895 seconds 2016-08-25 09:02:06,032 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) is shutdown Wait for master to start up... 2016-08-25 09:02:06,034 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:06,036 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@74ddb0f9 2016-08-25 09:02:06,036 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:06,037 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:06,038 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:06,038 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55713 2016-08-25 09:02:06,038 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55713 2016-08-25 09:02:06,039 | INFO | SyncThread:0 | Established session 0x156bec917250067 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55713 2016-08-25 09:02:06,039 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250067, negotiated timeout = 10000 2016-08-25 09:02:06,095 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55714 2016-08-25 09:02:06,100 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,100 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,101 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55714 2016-08-25 09:02:06,101 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:06,102 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55714 2016-08-25 09:02:06,102 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:06,116 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:06,154 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:06,155 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000155.log 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000153 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000154.sst 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000151.sst 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:06,157 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000153 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000155.log 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000151.sst 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000154.sst 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:06,162 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/7.97 kb and 1/10 files 2016-08-25 09:02:06,163 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.12/7.97 kb and 2/10 files 2016-08-25 09:02:06,164 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.02/7.97 kb and 3/10 files 2016-08-25 09:02:06,165 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.92/7.97 kb and 4/10 files 2016-08-25 09:02:06,165 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.36/7.97 kb and 5/10 files 2016-08-25 09:02:06,166 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.26/7.97 kb and 6/10 files 2016-08-25 09:02:06,167 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.16/7.97 kb and 7/10 files 2016-08-25 09:02:06,167 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.05/7.97 kb and 8/10 files 2016-08-25 09:02:06,168 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.44/7.97 kb and 1/10 files 2016-08-25 09:02:06,168 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.07/7.97 kb and 9/10 files 2016-08-25 09:02:06,169 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.63/7.97 kb and 2/10 files 2016-08-25 09:02:06,169 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.97/7.97 kb and 10/10 files 2016-08-25 09:02:06,169 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:06,169 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.52/7.97 kb and 3/10 files 2016-08-25 09:02:06,170 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.46/7.97 kb and 4/10 files 2016-08-25 09:02:06,170 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.36/7.97 kb and 5/10 files 2016-08-25 09:02:06,170 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.26/7.97 kb and 6/10 files 2016-08-25 09:02:06,171 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.27/7.97 kb and 7/10 files 2016-08-25 09:02:06,171 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:06,172 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.17/7.97 kb and 8/10 files 2016-08-25 09:02:06,172 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.07/7.97 kb and 9/10 files 2016-08-25 09:02:06,172 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) is starting 2016-08-25 09:02:06,173 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.97/7.97 kb and 10/10 files 2016-08-25 09:02:06,173 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:06,176 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:06,177 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55719 2016-08-25 09:02:06,177 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:06,177 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) started 2016-08-25 09:02:06,177 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:06,177 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:06,177 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-2 ====================================== 100.4. Browse the queue on new master ====================================== ====================================== 100.5. Found ====================================== ====================================== 101.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:06,193 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) is shutting down 2016-08-25 09:02:06,194 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:06,197 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,199 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:06,199 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:06,199 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:06,199 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:06,200 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:06,200 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250065 2016-08-25 09:02:06,201 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55697 which had sessionid 0x156bec917250065 2016-08-25 09:02:06,201 | INFO | Time-limited test | Session: 0x156bec917250065 closed 2016-08-25 09:02:06,201 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:06,308 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:06,308 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) uptime 0.871 seconds 2016-08-25 09:02:06,308 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) is shutdown Wait for master to start up... 2016-08-25 09:02:06,310 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:06,313 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5180a0e5 2016-08-25 09:02:06,313 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:06,314 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:06,315 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:06,315 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55721 2016-08-25 09:02:06,315 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55721 2016-08-25 09:02:06,316 | INFO | SyncThread:0 | Established session 0x156bec917250068 with negotiated timeout 10000 for client /127.0.0.1:55721 2016-08-25 09:02:06,316 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250068, negotiated timeout = 10000 2016-08-25 09:02:06,336 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,397 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:02:06,400 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,402 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:06,402 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,406 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:06,411 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,481 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55722 2016-08-25 09:02:06,485 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,485 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,486 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55722 2016-08-25 09:02:06,487 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:06,487 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55722 2016-08-25 09:02:06,487 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:06,548 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:06,553 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:06,553 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:06,553 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000157.sst 2016-08-25 09:02:06,553 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000151.sst 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000158.log 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000154.sst 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000156 2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:06,559 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/8.92 kb and 1/11 files 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/8.92 kb and 2/11 files 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000151.sst 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/8.92 kb and 3/11 files 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000157.sst 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000154.sst 2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.59/8.92 kb and 4/11 files 2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000156 2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000158.log 2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.77/8.92 kb and 5/11 files 2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.67/8.92 kb and 6/11 files 2016-08-25 09:02:06,562 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.57/8.92 kb and 7/11 files 2016-08-25 09:02:06,562 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.47/8.92 kb and 8/11 files 2016-08-25 09:02:06,563 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.41/8.92 kb and 9/11 files 2016-08-25 09:02:06,563 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.90/8.92 kb and 10/11 files 2016-08-25 09:02:06,564 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.92/8.92 kb and 11/11 files 2016-08-25 09:02:06,564 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:06,566 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/8.92 kb and 1/11 files 2016-08-25 09:02:06,566 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.84/8.92 kb and 2/11 files 2016-08-25 09:02:06,566 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.73/8.92 kb and 3/11 files 2016-08-25 09:02:06,566 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:06,567 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.63/8.92 kb and 4/11 files 2016-08-25 09:02:06,567 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) is starting 2016-08-25 09:02:06,567 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.53/8.92 kb and 5/11 files 2016-08-25 09:02:06,568 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.43/8.92 kb and 6/11 files 2016-08-25 09:02:06,568 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.32/8.92 kb and 7/11 files 2016-08-25 09:02:06,569 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.82/8.92 kb and 8/11 files 2016-08-25 09:02:06,569 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.83/8.92 kb and 9/11 files 2016-08-25 09:02:06,570 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.73/8.92 kb and 10/11 files 2016-08-25 09:02:06,570 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/8.92 kb and 11/11 files 2016-08-25 09:02:06,570 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:06,571 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55727 2016-08-25 09:02:06,572 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:06,573 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) started 2016-08-25 09:02:06,573 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:06,573 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:06,573 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 101.4. Browse the queue on new master ====================================== 2016-08-25 09:02:06,577 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d ====================================== 101.5. Found ====================================== ====================================== 102.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:06,596 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) is shutting down 2016-08-25 09:02:06,597 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:06,599 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,599 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:06,599 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250066 2016-08-25 09:02:06,600 | INFO | Time-limited test | Session: 0x156bec917250066 closed 2016-08-25 09:02:06,600 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55705 which had sessionid 0x156bec917250066 2016-08-25 09:02:06,600 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:06,604 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:06,604 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:06,604 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:06,604 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:06,718 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:06,718 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) uptime 1.028 seconds 2016-08-25 09:02:06,719 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) is shutdown Wait for master to start up... 2016-08-25 09:02:06,720 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:06,722 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1c18c548 2016-08-25 09:02:06,723 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:06,723 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:06,723 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:06,724 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:06,724 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55729 2016-08-25 09:02:06,724 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55729 2016-08-25 09:02:06,725 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:06,725 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,725 | INFO | SyncThread:0 | Established session 0x156bec917250069 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55729 2016-08-25 09:02:06,725 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250069, negotiated timeout = 10000 2016-08-25 09:02:06,726 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,728 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:06,730 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,811 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55730 2016-08-25 09:02:06,815 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,815 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:06,817 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55730 2016-08-25 09:02:06,817 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:06,817 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55730 2016-08-25 09:02:06,817 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:02:06,835 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:06,882 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:06,883 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000154.sst 2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000158.log 2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000157.sst 2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000156 2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000151.sst 2016-08-25 09:02:06,889 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:06,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:06,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000151.sst 2016-08-25 09:02:06,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:06,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000158.log 2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000154.sst 2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000157.sst 2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000156 2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:06,897 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/8.92 kb and 1/11 files 2016-08-25 09:02:06,897 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.91/8.92 kb and 2/11 files 2016-08-25 09:02:06,898 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.10/8.92 kb and 3/11 files 2016-08-25 09:02:06,899 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.00/8.92 kb and 4/11 files 2016-08-25 09:02:06,899 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.93/8.92 kb and 5/11 files 2016-08-25 09:02:06,900 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/8.92 kb and 6/11 files 2016-08-25 09:02:06,901 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.33/8.92 kb and 7/11 files 2016-08-25 09:02:06,901 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.22/8.92 kb and 8/11 files 2016-08-25 09:02:06,902 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.12/8.92 kb and 9/11 files 2016-08-25 09:02:06,902 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.02/8.92 kb and 10/11 files 2016-08-25 09:02:06,902 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.92/8.92 kb and 11/11 files 2016-08-25 09:02:06,902 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:06,903 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/8.92 kb and 1/11 files 2016-08-25 09:02:06,904 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.91/8.92 kb and 2/11 files 2016-08-25 09:02:06,904 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/8.92 kb and 3/11 files 2016-08-25 09:02:06,905 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:06,905 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.00/8.92 kb and 4/11 files 2016-08-25 09:02:06,905 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.89/8.92 kb and 5/11 files 2016-08-25 09:02:06,906 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) is starting 2016-08-25 09:02:06,906 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.79/8.92 kb and 6/11 files 2016-08-25 09:02:06,906 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.69/8.92 kb and 7/11 files 2016-08-25 09:02:06,907 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/8.92 kb and 8/11 files 2016-08-25 09:02:06,908 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.08/8.92 kb and 9/11 files 2016-08-25 09:02:06,908 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.02/8.92 kb and 10/11 files 2016-08-25 09:02:06,909 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/8.92 kb and 11/11 files 2016-08-25 09:02:06,909 | INFO | hawtdispatch-DEFAULT-1 | Attached 2016-08-25 09:02:06,909 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55735 2016-08-25 09:02:06,910 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:06,910 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) started 2016-08-25 09:02:06,910 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:06,910 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:06,910 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-1 ====================================== 102.4. Browse the queue on new master ====================================== 2016-08-25 09:02:06,913 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 ====================================== 102.5. Found ====================================== ====================================== 103.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:06,924 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) is shutting down 2016-08-25 09:02:06,925 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:06,927 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:06,927 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:06,927 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:06,927 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:06,927 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250067 2016-08-25 09:02:06,928 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:06,928 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:06,928 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55713 which had sessionid 0x156bec917250067 2016-08-25 09:02:06,928 | INFO | Time-limited test | Session: 0x156bec917250067 closed 2016-08-25 09:02:06,928 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:07,036 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:07,036 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) uptime 1.004 seconds 2016-08-25 09:02:07,036 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) is shutdown Wait for master to start up... 2016-08-25 09:02:07,039 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:07,041 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4c0906ae 2016-08-25 09:02:07,041 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:07,042 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:07,043 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:07,043 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55737 2016-08-25 09:02:07,043 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55737 2016-08-25 09:02:07,044 | INFO | SyncThread:0 | Established session 0x156bec91725006a with negotiated timeout 10000 for client /127.0.0.1:55737 2016-08-25 09:02:07,045 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725006a, negotiated timeout = 10000 2016-08-25 09:02:07,052 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,125 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:07,129 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,134 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:07,134 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,137 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:07,141 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,234 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55738 2016-08-25 09:02:07,238 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,238 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,239 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55738 2016-08-25 09:02:07,239 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:07,240 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55738 2016-08-25 09:02:07,240 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:07,304 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:07,315 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:07,315 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000157.sst 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000151.sst 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000159 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000154.sst 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000161.log 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000160.sst 2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.86 kb and 1/12 files 2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/9.86 kb and 2/12 files 2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst 2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/9.86 kb and 3/12 files 2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000151.sst 2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000159 2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.59/9.86 kb and 4/12 files 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000154.sst 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000157.sst 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000160.sst 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000161.log 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.49/9.86 kb and 5/12 files 2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.03/9.86 kb and 6/12 files 2016-08-25 09:02:07,324 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.93/9.86 kb and 7/12 files 2016-08-25 09:02:07,324 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.87/9.86 kb and 8/12 files 2016-08-25 09:02:07,324 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.05/9.86 kb and 9/12 files 2016-08-25 09:02:07,325 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.95/9.86 kb and 10/12 files 2016-08-25 09:02:07,325 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.97/9.86 kb and 11/12 files 2016-08-25 09:02:07,325 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.86/9.86 kb and 12/12 files 2016-08-25 09:02:07,325 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:07,328 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:07,329 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) is starting 2016-08-25 09:02:07,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/9.86 kb and 1/12 files 2016-08-25 09:02:07,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/9.86 kb and 2/12 files 2016-08-25 09:02:07,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.34/9.86 kb and 3/12 files 2016-08-25 09:02:07,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.24/9.86 kb and 4/12 files 2016-08-25 09:02:07,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.13/9.86 kb and 5/12 files 2016-08-25 09:02:07,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.15/9.86 kb and 6/12 files 2016-08-25 09:02:07,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.05/9.86 kb and 7/12 files 2016-08-25 09:02:07,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.95/9.86 kb and 8/12 files 2016-08-25 09:02:07,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.84/9.86 kb and 9/12 files 2016-08-25 09:02:07,333 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55743 2016-08-25 09:02:07,333 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:07,333 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) started 2016-08-25 09:02:07,333 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:07,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.78/9.86 kb and 10/12 files 2016-08-25 09:02:07,333 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:07,333 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:07,334 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.68/9.86 kb and 11/12 files 2016-08-25 09:02:07,334 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.86/9.86 kb and 12/12 files 2016-08-25 09:02:07,334 | INFO | hawtdispatch-DEFAULT-1 | Attached Master started: node-2 ====================================== 103.4. Browse the queue on new master ====================================== 2016-08-25 09:02:07,339 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 103.5. Found ====================================== ====================================== 104.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:07,352 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) is shutting down 2016-08-25 09:02:07,354 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:07,357 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,358 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:07,359 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:07,359 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:07,359 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:07,360 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:07,360 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250068 2016-08-25 09:02:07,361 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55721 which had sessionid 0x156bec917250068 2016-08-25 09:02:07,361 | INFO | Time-limited test | Session: 0x156bec917250068 closed 2016-08-25 09:02:07,361 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:07,475 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:07,476 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) uptime 1.167 seconds 2016-08-25 09:02:07,476 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) is shutdown Wait for master to start up... 2016-08-25 09:02:07,478 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:07,480 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1d25b88f 2016-08-25 09:02:07,480 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:07,481 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:07,481 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:07,481 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55745 2016-08-25 09:02:07,481 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55745 2016-08-25 09:02:07,482 | INFO | SyncThread:0 | Established session 0x156bec91725006b with negotiated timeout 10000 for client /127.0.0.1:55745 2016-08-25 09:02:07,482 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725006b, negotiated timeout = 10000 2016-08-25 09:02:07,487 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,499 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped 2016-08-25 09:02:07,501 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,512 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:07,513 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,515 | INFO | Starting broker node: node-3-EventThread | Promoted to master 2016-08-25 09:02:07,518 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,595 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55746 2016-08-25 09:02:07,600 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,600 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,601 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55746 2016-08-25 09:02:07,601 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55746 2016-08-25 09:02:07,601 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started 2016-08-25 09:02:07,601 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started 2016-08-25 09:02:07,616 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:07,667 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:07,668 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000157.sst 2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000159 2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000161.log 2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000151.sst 2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000160.sst 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000154.sst 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000145.sst 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000161.log 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000154.sst 2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000139.sst 2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000159 2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000157.sst 2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000151.sst 2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000137.sst 2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000148.sst 2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000142.sst 2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000160.sst 2016-08-25 09:02:07,674 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/9.86 kb and 1/12 files 2016-08-25 09:02:07,674 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.91/9.86 kb and 2/12 files 2016-08-25 09:02:07,674 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.46/9.86 kb and 3/12 files 2016-08-25 09:02:07,675 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.36/9.86 kb and 4/12 files 2016-08-25 09:02:07,675 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.86 kb and 1/12 files 2016-08-25 09:02:07,675 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.29/9.86 kb and 5/12 files 2016-08-25 09:02:07,676 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.08/9.86 kb and 2/12 files 2016-08-25 09:02:07,676 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.48/9.86 kb and 6/12 files 2016-08-25 09:02:07,676 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.38/9.86 kb and 7/12 files 2016-08-25 09:02:07,676 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/9.86 kb and 3/12 files 2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/9.86 kb and 4/12 files 2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.27/9.86 kb and 8/12 files 2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.17/9.86 kb and 9/12 files 2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.42/9.86 kb and 5/12 files 2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.32/9.86 kb and 6/12 files 2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.07/9.86 kb and 10/12 files 2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.97/9.86 kb and 11/12 files 2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.22/9.86 kb and 7/12 files 2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.16/9.86 kb and 8/12 files 2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.86/9.86 kb and 12/12 files 2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:07,679 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.05/9.86 kb and 9/12 files 2016-08-25 09:02:07,679 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.07/9.86 kb and 10/12 files 2016-08-25 09:02:07,680 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.97/9.86 kb and 11/12 files 2016-08-25 09:02:07,680 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.86/9.86 kb and 12/12 files 2016-08-25 09:02:07,680 | INFO | hawtdispatch-DEFAULT-3 | Attached 2016-08-25 09:02:07,681 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:07,681 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) is starting 2016-08-25 09:02:07,684 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:07,685 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55751 2016-08-25 09:02:07,685 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:07,685 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) started 2016-08-25 09:02:07,685 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:07,685 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:07,685 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-3 ====================================== 104.4. Browse the queue on new master ====================================== ====================================== 104.5. Found ====================================== ====================================== 105.3. Restart master node ====================================== Stopping node-3 2016-08-25 09:02:07,705 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) is shutting down 2016-08-25 09:02:07,706 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:07,709 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,711 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:07,711 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250069 2016-08-25 09:02:07,711 | INFO | Time-limited test | Session: 0x156bec917250069 closed 2016-08-25 09:02:07,711 | INFO | Starting broker node: node-3-EventThread | EventThread shut down 2016-08-25 09:02:07,711 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55729 which had sessionid 0x156bec917250069 2016-08-25 09:02:07,715 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:07,715 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:07,716 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:07,716 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d 2016-08-25 09:02:07,817 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3] 2016-08-25 09:02:07,818 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) uptime 1.099 seconds 2016-08-25 09:02:07,818 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) is shutdown Wait for master to start up... 2016-08-25 09:02:07,820 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default] 2016-08-25 09:02:07,822 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@63de5fd6 2016-08-25 09:02:07,822 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:07,823 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:07,823 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:07,823 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55753 2016-08-25 09:02:07,823 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55753 2016-08-25 09:02:07,824 | INFO | SyncThread:0 | Established session 0x156bec91725006c with negotiated timeout 10000 for client /127.0.0.1:55753 2016-08-25 09:02:07,824 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725006c, negotiated timeout = 10000 2016-08-25 09:02:07,830 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,875 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped 2016-08-25 09:02:07,875 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped 2016-08-25 09:02:07,875 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,877 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:07,879 | INFO | Starting broker node: node-1-EventThread | Promoted to master 2016-08-25 09:02:07,883 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,910 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55754 2016-08-25 09:02:07,915 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,916 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:07,916 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55754 2016-08-25 09:02:07,917 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started 2016-08-25 09:02:07,917 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55754 2016-08-25 09:02:07,917 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started 2016-08-25 09:02:07,932 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:07,991 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:07,992 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:07,993 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000165.sst 2016-08-25 09:02:07,993 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000162 2016-08-25 09:02:07,993 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:07,993 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000164.log 2016-08-25 09:02:07,994 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:07,995 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT 2016-08-25 09:02:07,995 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000162 2016-08-25 09:02:07,995 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000165.sst 2016-08-25 09:02:07,995 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000164.log 2016-08-25 09:02:07,996 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/2.87 kb and 1/4 files 2016-08-25 09:02:07,996 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.67/2.87 kb and 2/4 files 2016-08-25 09:02:07,997 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.69/2.87 kb and 3/4 files 2016-08-25 09:02:07,997 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.87/2.87 kb and 4/4 files 2016-08-25 09:02:07,997 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:07,999 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:08,000 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) is starting 2016-08-25 09:02:08,001 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/2.87 kb and 1/4 files 2016-08-25 09:02:08,002 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.75/2.87 kb and 2/4 files 2016-08-25 09:02:08,003 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.69/2.87 kb and 3/4 files 2016-08-25 09:02:08,004 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.87/2.87 kb and 4/4 files 2016-08-25 09:02:08,004 | INFO | hawtdispatch-DEFAULT-4 | Attached 2016-08-25 09:02:08,005 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55759 2016-08-25 09:02:08,005 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:08,005 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) started 2016-08-25 09:02:08,005 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:08,005 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb 2016-08-25 09:02:08,006 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb Master started: node-1 ====================================== 105.4. Browse the queue on new master ====================================== 2016-08-25 09:02:08,008 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 ====================================== 105.5. Found ====================================== ====================================== 106.3. Restart master node ====================================== Stopping node-1 2016-08-25 09:02:08,021 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) is shutting down 2016-08-25 09:02:08,021 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:08,024 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:08,025 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:08,025 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:08,025 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:08,025 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5 2016-08-25 09:02:08,025 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:08,025 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725006a 2016-08-25 09:02:08,026 | INFO | Time-limited test | Session: 0x156bec91725006a closed 2016-08-25 09:02:08,026 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55737 which had sessionid 0x156bec91725006a 2016-08-25 09:02:08,026 | INFO | Starting broker node: node-1-EventThread | EventThread shut down 2016-08-25 09:02:08,041 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped 2016-08-25 09:02:08,041 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:08,042 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:08,048 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped 2016-08-25 09:02:08,052 | INFO | Starting broker node: node-2-EventThread | Promoted to master 2016-08-25 09:02:08,054 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:08,078 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55761 2016-08-25 09:02:08,081 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation. 2016-08-25 09:02:08,083 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55761 2016-08-25 09:02:08,083 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started 2016-08-25 09:02:08,097 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:08,100 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log 2016-08-25 09:02:08,100 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000164.log 2016-08-25 09:02:08,101 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000162 2016-08-25 09:02:08,101 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT 2016-08-25 09:02:08,101 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000165.sst 2016-08-25 09:02:08,104 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/2.87 kb and 1/4 files 2016-08-25 09:02:08,104 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.92/2.87 kb and 2/4 files 2016-08-25 09:02:08,105 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.93/2.87 kb and 3/4 files 2016-08-25 09:02:08,105 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.87/2.87 kb and 4/4 files 2016-08-25 09:02:08,105 | INFO | hawtdispatch-DEFAULT-2 | Attached 2016-08-25 09:02:08,107 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:08,108 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) is starting 2016-08-25 09:02:08,112 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55764 2016-08-25 09:02:08,112 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started 2016-08-25 09:02:08,112 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) started 2016-08-25 09:02:08,112 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org 2016-08-25 09:02:08,112 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:08,112 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb 2016-08-25 09:02:08,145 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1] 2016-08-25 09:02:08,146 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) uptime 1.109 seconds 2016-08-25 09:02:08,146 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) is shutdown Wait for master to start up... Master started: node-2 ====================================== 106.4. Browse the queue on new master ====================================== 2016-08-25 09:02:08,148 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default] 2016-08-25 09:02:08,149 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1288fe70 2016-08-25 09:02:08,149 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:08,150 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:08,151 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session 2016-08-25 09:02:08,151 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55766 2016-08-25 09:02:08,151 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55766 2016-08-25 09:02:08,152 | INFO | SyncThread:0 | Established session 0x156bec91725006d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55766 2016-08-25 09:02:08,152 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725006d, negotiated timeout = 10000 ====================================== 106.5. Found ====================================== ====================================== 107.3. Restart master node ====================================== Stopping node-2 2016-08-25 09:02:08,158 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) is shutting down 2016-08-25 09:02:08,159 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped 2016-08-25 09:02:08,159 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation. 2016-08-25 09:02:08,160 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated. 2016-08-25 09:02:08,161 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55761 2016-08-25 09:02:08,161 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725006b 2016-08-25 09:02:08,161 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started 2016-08-25 09:02:08,161 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:08,161 | INFO | Time-limited test | Session: 0x156bec91725006b closed 2016-08-25 09:02:08,161 | INFO | Starting broker node: node-2-EventThread | EventThread shut down 2016-08-25 09:02:08,161 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55745 which had sessionid 0x156bec91725006b 2016-08-25 09:02:08,162 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer 2016-08-25 09:02:08,162 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956 2016-08-25 09:02:08,175 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped 2016-08-25 09:02:08,175 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:08,177 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet. 2016-08-25 09:02:08,267 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2] 2016-08-25 09:02:08,268 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) uptime 0.792 seconds 2016-08-25 09:02:08,268 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) is shutdown Wait for master to start up... 2016-08-25 09:02:08,270 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default] 2016-08-25 09:02:08,272 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5f8c78e 2016-08-25 09:02:08,272 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher 2016-08-25 09:02:08,272 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error) 2016-08-25 09:02:08,273 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session 2016-08-25 09:02:08,273 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55767 2016-08-25 09:02:08,273 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55767 2016-08-25 09:02:08,273 | INFO | SyncThread:0 | Established session 0x156bec91725006e with negotiated timeout 10000 for client /127.0.0.1:55767 2016-08-25 09:02:08,273 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725006e, negotiated timeout = 10000 2016-08-25 09:02:08,277 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.