I do have confirmation that a session is established for every client (all 45 of them) before beginning the disk load with dd.
I see, I was just trying to reduce variables. that should be fine then.
I see this in the logs:
2010-10-07 14:49:13,956 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_0
2010-10-07 14:49:13,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
2010-10-07 14:49:13,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
I'm not sure many users are running openjdk, also 1.6.0_0 is very old (I have 1.6.0_18 openjdk on my system). You should upgrade to a recent version of openjdk at the least, although I'd highly suggest running with the official (and recent) sun jdk. (again, this is to reduce variables)
Also I noticed this in the server log for 1 server, it seems to be misconfigured, perhaps you can fix that? (normal_3.3.1/192.168.131.12.log)
2010-10-07 14:49:13,979 - FATAL [main:QuorumPeerMain@83] - Invalid config, exiting abnormally
Should I enable more verbose logging?
Yes, give that a try, perhaps run with TRACE logging turned on. If you can upload one of those logs I'll take a look.
Right now we have this in the server log:
2010-10-07 14:51:32,961 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@633] - EndOfStreamException: Unable to read additional data from client sessionid 0x22b872ad9ff000c, likely client has closed socket
2010-10-07 14:51:32,962 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1434] - Closed socket connection for client /10.23.4.95:59738 which had sessionid 0x22b872ad9ff000c
This indicates that the client is closing the connection (EOS).
Please capture the logs on your client and upload one of them. Perhaps run that at TRACE level as well. That will give us more insight into why the client is closing it's side of the connection (at least from the server's perspective).
Thanks for the help on this!