Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError YCSB Client 0.1 Command line: -db com.yahoo.ycsb.db.KuduYCSBClient -P workloads/5nodes_workload -p sync_ops=false -p print_row_errors=true -p pre_split_num_tablets=200 -p table_name=ycsb -p masterQuorum=172.26.3.231:7051 -threads 16 -s -load Loading workload... Starting test. 0 sec: 0 operations; SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/todd/ycsb-kudu/kudu/target/kudu-binding-0.1.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/todd/ycsb-kudu/kudu/target/archive-tmp/kudu-binding-0.1.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2015-04-12 22:48:56,258 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet ~~~masterTableHack~~~ for table ~~~masterTableHack~~~ with start key "" and endkey "" 2015-04-12 22:48:56,356 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 39c3319fc9284a1f84318891817eb2f9 for table ycsb with start key "" and endkey "user1045" 2015-04-12 22:48:56,400 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 4a1614a7cb4f4e4198f1a31cfdf1a844 for table ycsb with start key "user6265" and endkey "user6310" 2015-04-12 22:48:56,406 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet fc891c9e679b4ba4b02700c0729851b7 for table ycsb with start key "user8515" and endkey "user8560" 2015-04-12 22:48:56,418 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet afecf7b47f7c42ad8be8ff7b7fcca5e2 for table ycsb with start key "user1810" and endkey "user1855" 2015-04-12 22:48:56,472 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 221e204614454e2c92b75763c2aeb661 for table ycsb with start key "user4015" and endkey "user4060" 2015-04-12 22:48:56,487 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-209' to 172.26.3.209 in 11441965ns 2015-04-12 22:48:56,500 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 9e1cb968bc434681a4c08aafb09bff5f for table ycsb with start key "user3205" and endkey "user3250" 2015-04-12 22:48:56,507 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 4ba5a22e14a84329b925dc3e17d20fb8 for table ycsb with start key "user7660" and endkey "user7705" 2015-04-12 22:48:56,531 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 8c82cbb7f2fc44009a7dd7f31b35f687 for table ycsb with start key "user5455" and endkey "user5500" 2015-04-12 22:48:56,556 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 2b82ac405be945e783d4c493789dc378 for table ycsb with start key "user6850" and endkey "user6895" 2015-04-12 22:48:56,566 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 41207f4c0685405d964530dbdd8a6065 for table ycsb with start key "user9100" and endkey "user9145" 2015-04-12 22:48:56,583 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 90fe5afa6aaf4d23a7bc5112e6a509ea for table ycsb with start key "user2395" and endkey "user2440" 1 sec: 284 operations; 275.46 current ops/sec; [INSERT AverageLatency(us)=7883.07] 2015-04-12 22:48:56,585 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0155e276bead45a8a477e3649198cdbe for table ycsb with start key "user4600" and endkey "user4645" 2015-04-12 22:48:56,606 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 778d3039dc464916af771fc0b2c9787a for table ycsb with start key "user2620" and endkey "user2665" 2015-04-12 22:48:56,623 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 063b13cae2cc465288520e3d5179c718 for table ycsb with start key "user7075" and endkey "user7120" 2015-04-12 22:48:56,638 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet ab40d29079234dad9eea3ed93b9bc9d7 for table ycsb with start key "user4870" and endkey "user4915" 2015-04-12 22:48:56,654 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 676f80348c464864be94c2db881fb749 for table ycsb with start key "user7435" and endkey "user7480" 2015-04-12 22:48:56,811 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 50009c48118540b29a4d84caa7c79289 for table ycsb with start key "user5185" and endkey "user5230" 2015-04-12 22:48:56,826 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b582c518624547bbaac69b5b860b7606 for table ycsb with start key "user2035" and endkey "user2080" 2015-04-12 22:48:56,828 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 80065d1bf1b848a6af30738d50ed5efd for table ycsb with start key "user1720" and endkey "user1765" 2015-04-12 22:48:56,847 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet dad8029d22bd47bfafb1b0168e58fe4c for table ycsb with start key "user6490" and endkey "user6535" 2015-04-12 22:48:56,851 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet c6e4c091611e4eb7a6fdcb1a3bde69f2 for table ycsb with start key "user4285" and endkey "user4330" 2015-04-12 22:48:56,870 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 7b9acf4843804553b62b32637790b682 for table ycsb with start key "user8020" and endkey "user8065" 2015-04-12 22:48:56,876 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 32949ffabbce443fbc16d6390720fade for table ycsb with start key "user8155" and endkey "user8200" 2015-04-12 22:48:56,880 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet be7b707c3d8c408b9ca99766bede14cc for table ycsb with start key "user5815" and endkey "user5860" 2015-04-12 22:48:56,891 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 3dc3d317bdbf4366a809b13812551239 for table ycsb with start key "user1450" and endkey "user1495" 2015-04-12 22:48:56,896 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet cf9b1b0aca0744c4a456c74d8df8b31f for table ycsb with start key "user7615" and endkey "user7660" 2015-04-12 22:48:56,899 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 304955d9af0b412594ee7eac5ec48b55 for table ycsb with start key "user5905" and endkey "user5950" 2015-04-12 22:48:56,910 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 55fcbcc8b5024d16bd9bac9827a34a41 for table ycsb with start key "user3700" and endkey "user3745" 2015-04-12 22:48:56,911 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 7ac446106fdb45f7a34f59d1c9062006 for table ycsb with start key "user6130" and endkey "user6175" 2015-04-12 22:48:56,913 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 05ac82e5cdcf4d7494929a7191ca76d2 for table ycsb with start key "user5320" and endkey "user5365" 2015-04-12 22:48:56,918 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 568e7aa39c9d416f95113ae674b2607d for table ycsb with start key "user1675" and endkey "user1720" 2015-04-12 22:48:56,926 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e91c5b97259c40b4b3301e52b27b36b2 for table ycsb with start key "user5545" and endkey "user5590" 2015-04-12 22:48:56,998 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e09f57dde36646f6be9f1a093f7f8984 for table ycsb with start key "user8380" and endkey "user8425" 2015-04-12 22:48:57,007 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 7cd739b3c5914119a5800a5ec9f83882 for table ycsb with start key "user3340" and endkey "user3385" 2015-04-12 22:48:57,008 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 6d6c3aab62344669864adfbc18e91340 for table ycsb with start key "user7795" and endkey "user7840" 2015-04-12 22:48:57,011 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 6486d35136cc4bb4a279e71bb8ba4978 for table ycsb with start key "user4510" and endkey "user4555" 2015-04-12 22:48:57,013 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet fd2e9ee0b59d4451be808b25d714a2b6 for table ycsb with start key "user6715" and endkey "user6760" 2015-04-12 22:48:57,014 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet a4cba0a673a34febacbee732870b67f0 for table ycsb with start key "user2260" and endkey "user2305" 2015-04-12 22:48:57,015 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 3a910f1013c24607b6804b7390f53ef3 for table ycsb with start key "user4960" and endkey "user5005" 2015-04-12 22:48:57,025 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 6a23c56677d446bdbb6a4f4bc0800fc8 for table ycsb with start key "user7210" and endkey "user7255" 2015-04-12 22:48:57,026 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet aa7e8cb61dc843cb9bcb40998e88436f for table ycsb with start key "user5095" and endkey "user5140" 2015-04-12 22:48:57,029 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 8db6003ef43948a0a806a1ecd312c7eb for table ycsb with start key "user8740" and endkey "user8785" 2015-04-12 22:48:57,030 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 4cdb522ac5144879bfe5e56f0c04c9d8 for table ycsb with start key "user7300" and endkey "user7345" 2015-04-12 22:48:57,031 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 59445b81fb98424ba7fff0dde2f10788 for table ycsb with start key "user6400" and endkey "user6445" 2015-04-12 22:48:57,034 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 76d4ae4f2ac1446d9378448bb47addaa for table ycsb with start key "user2845" and endkey "user2890" 2015-04-12 22:48:57,035 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 43799cb9c04a4be5967a82e76a66fbda for table ycsb with start key "user4375" and endkey "user4420" 2015-04-12 22:48:57,060 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet f051f354d76e4582bd1714515ae72c29 for table ycsb with start key "user2170" and endkey "user2215" 2015-04-12 22:48:57,061 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e815f3695c1a4123a69811ae1f28d8cc for table ycsb with start key "user8830" and endkey "user8875" 2015-04-12 22:48:57,071 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 794fd028476141bab133ff334c623583 for table ycsb with start key "user6625" and endkey "user6670" 2015-04-12 22:48:57,074 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 87de27b6d47d4b2b8e9c9f11a827c93a for table ycsb with start key "user5680" and endkey "user5725" 2015-04-12 22:48:57,081 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 1854e52c70654edab674d4b632198cf1 for table ycsb with start key "user3430" and endkey "user3475" 2015-04-12 22:48:57,210 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 3ffd715a76444454bffbff02d8b0c86c for table ycsb with start key "user1585" and endkey "user1630" 2015-04-12 22:48:57,214 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 5f48d1bdbfa84880ba72d0ca71ecc6e4 for table ycsb with start key "user8245" and endkey "user8290" 2015-04-12 22:48:57,226 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 79d622bf67e5444abb8d13e0ae463cc6 for table ycsb with start key "user3790" and endkey "user3835" 2015-04-12 22:48:57,228 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 5dd9d72bacf74e498e188514b2e906ff for table ycsb with start key "user6040" and endkey "user6085" 2015-04-12 22:48:57,232 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b959dbd4945b4ceca63974f4ad5cb562 for table ycsb with start key "user8965" and endkey "user9010" 2015-04-12 22:48:57,235 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 758e6a5fa0e64ef99390a5c4011c4e5e for table ycsb with start key "user2980" and endkey "user3025" 2015-04-12 22:48:57,240 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0ed3afbf6d304ddb8d463ce89d0ceff8 for table ycsb with start key "user7570" and endkey "user7615" 2015-04-12 22:48:57,244 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 3089b9ddffb7460f81168666098577dd for table ycsb with start key "user3925" and endkey "user3970" 2015-04-12 22:48:57,256 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0896a3f819444888a2e54a61257a68bd for table ycsb with start key "user2755" and endkey "user2800" 2015-04-12 22:48:57,261 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 19b2be7173f64f9a90ed886253ec12c5 for table ycsb with start key "user8470" and endkey "user8515" 2015-04-12 22:48:57,267 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet fac7c03d320440679149f7c3419174de for table ycsb with start key "user4105" and endkey "user4150" 2015-04-12 22:48:57,355 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 3bfab595ced947fda4a6355f7de42b66 for table ycsb with start key "user4150" and endkey "user4195" 2015-04-12 22:48:57,360 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 25e461781b7d4af5a803e8af41ddc3b0 for table ycsb with start key "user7885" and endkey "user7930" 2015-04-12 22:48:57,362 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e5b5258c3f8444d58d99e1bc59ce5f3d for table ycsb with start key "user1225" and endkey "user1270" 2015-04-12 22:48:57,366 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 5207814386754341981f5d662f51e2f3 for table ycsb with start key "user1315" and endkey "user1360" 2015-04-12 22:48:57,371 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet ce0df3e006ae4fa38c5d87f6faf3d7b9 for table ycsb with start key "user9190" and endkey "user9235" 2015-04-12 22:48:57,372 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet d186b56ebe2e4e009a836258e8849f42 for table ycsb with start key "user6985" and endkey "user7030" 2015-04-12 22:48:57,373 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet ae51f0105b264219821a479498695c9f for table ycsb with start key "user8605" and endkey "user8650" 2015-04-12 22:48:57,375 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 6a1e42c0bc924e5ca9af3e9b0a695e61 for table ycsb with start key "user3070" and endkey "user3115" 2015-04-12 22:48:57,382 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet a893d5e2864d44c1a6ea1a56d269d756 for table ycsb with start key "user1900" and endkey "user1945" 2015-04-12 22:48:57,386 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e6f33f2dbad147f185f9ba6551e9bdbd for table ycsb with start key "user8785" and endkey "user8830" 2015-04-12 22:48:57,389 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 609c8ecdfd794c66802f75391b6635f0 for table ycsb with start key "user8290" and endkey "user8335" 2015-04-12 22:48:57,390 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 724df91129074bdba35a8902dec67802 for table ycsb with start key "user3565" and endkey "user3610" 2015-04-12 22:48:57,392 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 7390487a192b4225af6797af771ae8f8 for table ycsb with start key "user2485" and endkey "user2530" 2015-04-12 22:48:57,395 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e4fe37d2f1d6457780f2f1d5e11a35ba for table ycsb with start key "user4735" and endkey "user4780" 2015-04-12 22:48:57,401 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-115' to 172.26.0.115 in 4929818ns 2015-04-12 22:48:57,403 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 407650376d2045648cac33cc849ee69f for table ycsb with start key "user2935" and endkey "user2980" 2015-04-12 22:48:57,495 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet fdc1d51f50a24eb0be3e00eaa3c31de3 for table ycsb with start key "user2890" and endkey "user2935" 2015-04-12 22:48:57,522 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet d880b8c9f4a24dafaacc2d5e69b353b2 for table ycsb with start key "user3115" and endkey "user3160" 2015-04-12 22:48:57,523 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 812448b33302481c942cf3adc65072a8 for table ycsb with start key "user2710" and endkey "user2755" 2015-04-12 22:48:57,535 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet fc97fd20e21742c2bbab608eae50a048 for table ycsb with start key "user5140" and endkey "user5185" 2015-04-12 22:48:57,573 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet f843f12e91b74ab39e2587badd5ff13c for table ycsb with start key "user5995" and endkey "user6040" 2015-04-12 22:48:57,584 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 235d8e0664814f3da9ae81b3fe543ce5 for table ycsb with start key "user6760" and endkey "user6805" 2015-04-12 22:48:57,586 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 7823220768a548788bdba28b558352b0 for table ycsb with start key "user2125" and endkey "user2170" 2 sec: 4064 operations; 3776.22 current ops/sec; [INSERT AverageLatency(us)=4190.98] 2015-04-12 22:48:57,596 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 27c587303bec424586cabce85ebab2fe for table ycsb with start key "user1540" and endkey "user1585" 2015-04-12 22:48:57,602 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 11c8079ea2da4587899ff07de3edaaeb for table ycsb with start key "user6580" and endkey "user6625" 2015-04-12 22:48:57,620 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-99' to 172.26.0.99 in 5199050ns 2015-04-12 22:48:57,622 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet d438088934d84b8698281544952630b4 for table ycsb with start key "user9820" and endkey "user9865" 2015-04-12 22:48:57,632 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet ee12377863f54824b3fadbb4f2b487d6 for table ycsb with start key "user5410" and endkey "user5455" 2015-04-12 22:48:57,634 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 5ab76debd43549c5b838b2cbe20dee81 for table ycsb with start key "user6175" and endkey "user6220" 2015-04-12 22:48:57,635 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet ee6f559224624922887eef4ca0ad9ea4 for table ycsb with start key "user7930" and endkey "user7975" 2015-04-12 22:48:57,639 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet f836a48e19fe49f99776c7b5fe5e5c12 for table ycsb with start key "user5770" and endkey "user5815" 2015-04-12 22:48:57,645 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 80d80b334abf48b69918e3fda9f50329 for table ycsb with start key "user3475" and endkey "user3520" 2015-04-12 22:48:57,653 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 6cfd594c7fac420090441d55bc7174c7 for table ycsb with start key "user7345" and endkey "user7390" 2015-04-12 22:48:57,662 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 5c20b2b2cbfd41149d11068755ce6d7d for table ycsb with start key "user8875" and endkey "user8920" 2015-04-12 22:48:57,673 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e3924acd6e48462fb4e8d04c3664e036 for table ycsb with start key "user6355" and endkey "user6400" 2015-04-12 22:48:57,681 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 3d05b1860a0549e0bfa403c28d61e229 for table ycsb with start key "user5005" and endkey "user5050" 2015-04-12 22:48:57,894 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 53993e7cdf464e9c9cd5204df08decee for table ycsb with start key "user3295" and endkey "user3340" 2015-04-12 22:48:57,900 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet fb55343f179547808b203dee1a9c7c51 for table ycsb with start key "user5365" and endkey "user5410" 2015-04-12 22:48:57,905 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 54e547f98a6d4e569a23ede2a2828a83 for table ycsb with start key "user1090" and endkey "user1135" 2015-04-12 22:48:57,917 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 55537115ebb24a0498f71913f7106518 for table ycsb with start key "user5590" and endkey "user5635" 2015-04-12 22:48:57,927 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b6ef90b60bfd49e68621680fbf266d0c for table ycsb with start key "user1945" and endkey "user1990" 2015-04-12 22:48:57,932 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 1bab3b084f374671a768eadb8d37b8ce for table ycsb with start key "user2530" and endkey "user2575" 2015-04-12 22:48:57,935 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b18c47f8b11c495ea60f2ade6ed10a6a for table ycsb with start key "user8650" and endkey "user8695" 2015-04-12 22:48:57,940 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 193b513bbb8348838276bea48c102371 for table ycsb with start key "user6940" and endkey "user6985" 2015-04-12 22:48:57,975 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 1b4c1836283542a5aad5da8e27f160b8 for table ycsb with start key "user7525" and endkey "user7570" 2015-04-12 22:48:57,977 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e39aa4e04b2c411fa14e66efd4924587 for table ycsb with start key "user5950" and endkey "user5995" 2015-04-12 22:48:57,978 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 2b221b5a3264493ca51b5fbd9f3a17b7 for table ycsb with start key "user4825" and endkey "user4870" 2015-04-12 22:48:57,990 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 37ea00dfed7a4694af52000257a344e1 for table ycsb with start key "user4060" and endkey "user4105" 2015-04-12 22:48:57,992 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 99c4e9b15f9a4642932a666528706d1d for table ycsb with start key "user4240" and endkey "user4285" 2015-04-12 22:48:57,999 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet d84c3633a0e54403a92c4b5bf09027c2 for table ycsb with start key "user3970" and endkey "user4015" 2015-04-12 22:48:58,001 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 941a3f6fd55144c88da7462b1fb999d4 for table ycsb with start key "user4645" and endkey "user4690" 2015-04-12 22:48:58,056 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet d1ee46c96b9d4a3187936f69d316fe1c for table ycsb with start key "user3655" and endkey "user3700" 2015-04-12 22:48:58,057 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 55739e98262d4d1f8b9e311dc3753784 for table ycsb with start key "user8695" and endkey "user8740" 2015-04-12 22:48:58,061 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 3d2a04020cd54df29d2f72784db74eb8 for table ycsb with start key "user5230" and endkey "user5275" 2015-04-12 22:48:58,073 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 65c9c301df1a4b288a6924b805f8130b for table ycsb with start key "user1360" and endkey "user1405" 2015-04-12 22:48:58,333 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 19ce84717eea4585b9a1696f383447a0 for table ycsb with start key "user7750" and endkey "user7795" 2015-04-12 22:48:58,341 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 6e1d8107f384430cbabea9c8711b5d86 for table ycsb with start key "user8110" and endkey "user8155" 2015-04-12 22:48:58,373 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 29e50b6b9df249beb72def616de450fe for table ycsb with start key "user6445" and endkey "user6490" 2015-04-12 22:48:58,405 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 82cc726f2a1f47a6bd8c7716b2b49729 for table ycsb with start key "user5275" and endkey "user5320" 2015-04-12 22:48:58,440 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-1-80' to 172.26.1.80 in 7537037ns 2015-04-12 22:48:58,442 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 292b2c6e55004754ad389b24a8275071 for table ycsb with start key "user9955" and endkey "" 2015-04-12 22:48:58,467 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-1-51' to 172.26.1.51 in 5532003ns 2015-04-12 22:48:58,474 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 5ebd6ef592244001b8a568ffedec51d3 for table ycsb with start key "user1765" and endkey "user1810" 2015-04-12 22:48:58,477 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 83b5688c7bed4302a9e2d8828bc1c2b2 for table ycsb with start key "user9865" and endkey "user9910" 2015-04-12 22:48:58,488 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 94d504613dd54b5192674b44af42620f for table ycsb with start key "user5500" and endkey "user5545" 3 sec: 9035 operations; 4966.03 current ops/sec; [INSERT AverageLatency(us)=3187.67] 2015-04-12 22:48:58,747 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-33' to 172.26.0.33 in 3675224ns 2015-04-12 22:48:58,767 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet c0ea31aaabc944bba27b4fe6a3a31eae for table ycsb with start key "user1630" and endkey "user1675" 2015-04-12 22:48:58,792 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 99f9f1eaa18b4645afcaf29dc9b15880 for table ycsb with start key "user4195" and endkey "user4240" 2015-04-12 22:48:58,794 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet df31ca08502f4d58adc30bd0c27ec214 for table ycsb with start key "user9055" and endkey "user9100" 2015-04-12 22:48:58,799 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 100f480db4fe44b1bbed803d08324229 for table ycsb with start key "user7480" and endkey "user7525" 2015-04-12 22:48:58,826 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-215' to 172.26.0.215 in 3320953ns 2015-04-12 22:48:58,830 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 01864d7c02dc40ff818da7436686990c for table ycsb with start key "user1855" and endkey "user1900" 2015-04-12 22:48:58,847 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 01db1eae9f6842a3a6da8c05b33269e0 for table ycsb with start key "user7705" and endkey "user7750" 2015-04-12 22:48:58,916 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet c335263a849d4463ab09828f901eced2 for table ycsb with start key "user5050" and endkey "user5095" 2015-04-12 22:48:59,076 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet cf8642a46b27475698ec364d329f2527 for table ycsb with start key "user9910" and endkey "user9955" 2015-04-12 22:48:59,466 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-238' to 172.26.3.238 in 8704182ns 2015-04-12 22:48:59,473 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-39' to 172.26.0.39 in 3761657ns 2015-04-12 22:48:59,560 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 22c5b8d0fd944b95bf656134e59c6561 for table ycsb with start key "user4465" and endkey "user4510" 2015-04-12 22:48:59,568 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 2cdc818b226c48a38cd60998fa420fdc for table ycsb with start key "user3880" and endkey "user3925" 2015-04-12 22:48:59,591 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-210' to 172.26.0.210 in 3899098ns 4 sec: 14045 operations; 5010 current ops/sec; [INSERT AverageLatency(us)=3154.31] 2015-04-12 22:48:59,678 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-168' to 172.26.0.168 in 4243957ns 2015-04-12 22:48:59,693 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 50beb22aef7649efb442bd32b9dd698d for table ycsb with start key "user6535" and endkey "user6580" 2015-04-12 22:48:59,706 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b7aecd5368774804ab1e16285fca8434 for table ycsb with start key "user7840" and endkey "user7885" 2015-04-12 22:48:59,730 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet d2b34753240441f9b88483645427781c for table ycsb with start key "user7120" and endkey "user7165" 2015-04-12 22:48:59,740 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 8c4058addb27446a9a79719ab1a87412 for table ycsb with start key "user2350" and endkey "user2395" 2015-04-12 22:48:59,831 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet c919e4b1b5b940caafb7d09ff8369ed7 for table ycsb with start key "user9775" and endkey "user9820" 5 sec: 25135 operations; 11078.92 current ops/sec; [INSERT AverageLatency(us)=1420.09] 2015-04-12 22:49:01,149 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 68a6dd22e5634e97b27e4656a212cb18 for table ycsb with start key "user7030" and endkey "user7075" 2015-04-12 22:49:01,200 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet cc79343ea6ca409d99587e843d24054d for table ycsb with start key "user8920" and endkey "user8965" 2015-04-12 22:49:01,213 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 49ca3d948a5649bbbec6000ff1f6a0c1 for table ycsb with start key "user3025" and endkey "user3070" 2015-04-12 22:49:01,225 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0d7a95cfe5794ec886d2cfad7ff3c9f3 for table ycsb with start key "user2800" and endkey "user2845" 2015-04-12 22:49:01,229 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet a33230b6b6e7450484f9bcb05f6724c9 for table ycsb with start key "user1180" and endkey "user1225" 2015-04-12 22:49:01,275 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b06a40fa59ad4038bf280da0dcb5dba9 for table ycsb with start key "user4420" and endkey "user4465" 2015-04-12 22:49:01,296 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 51ceab7c90cf467f9d25e624598dbd41 for table ycsb with start key "user6220" and endkey "user6265" 2015-04-12 22:49:01,299 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 364f71ae824140fca856dee2e90429bd for table ycsb with start key "user2305" and endkey "user2350" 2015-04-12 22:49:01,311 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 82ba0cb62280430596448f39a3b52e65 for table ycsb with start key "user3520" and endkey "user3565" 2015-04-12 22:49:01,312 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet ea8878afbfc149ff8fb15cf04dde6f6f for table ycsb with start key "user6310" and endkey "user6355" 2015-04-12 22:49:01,318 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 49b5a06519a4411fb0f05904a3577791 for table ycsb with start key "user4555" and endkey "user4600" 2015-04-12 22:49:01,335 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 7449e718e31d4108ac71be70b8a8c22c for table ycsb with start key "user3835" and endkey "user3880" 2015-04-12 22:49:01,402 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-160' to 172.26.0.160 in 4522498ns 2015-04-12 22:49:01,405 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0dc7e3ae77b34d41b07b9596079abd58 for table ycsb with start key "user6085" and endkey "user6130" 6 sec: 30295 operations; 5160 current ops/sec; [INSERT AverageLatency(us)=3084.59] 2015-04-12 22:49:02,095 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-231' to 172.26.3.231 in 4316441ns 2015-04-12 22:49:02,154 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-92' to 172.26.0.92 in 5372931ns 2015-04-12 22:49:02,169 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-211' to 172.26.3.211 in 3276075ns 2015-04-12 22:49:02,171 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0377831a23ca4854924d7a8f1ecd2697 for table ycsb with start key "user9730" and endkey "user9775" 2015-04-12 22:49:02,190 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-111' to 172.26.0.111 in 3230013ns 2015-04-12 22:49:02,191 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 930f13ce130f4333bdba0bc4b4900673 for table ycsb with start key "user6895" and endkey "user6940" 2015-04-12 22:49:02,191 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 1481f0b5b9024d70883392336d607dae for table ycsb with start key "user1990" and endkey "user2035" 2015-04-12 22:49:02,212 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b0cca8cc157c4957ba8466717c47f7c1 for table ycsb with start key "user1135" and endkey "user1180" 2015-04-12 22:49:02,242 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-210' to 172.26.0.210 in 4796032ns 2015-04-12 22:49:02,274 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-164' to 172.26.0.164 in 5311531ns 2015-04-12 22:49:02,391 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet cb0a1d070bb349d5ad68e1953559de7e for table ycsb with start key "user7165" and endkey "user7210" 7 sec: 43208 operations; 12900.1 current ops/sec; [INSERT AverageLatency(us)=1234.49] 2015-04-12 22:49:03,336 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-247' to 172.26.0.247 in 10060083ns 2015-04-12 22:49:03,359 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b1ba53af715c493aa175e83d5e674bfd for table ycsb with start key "user4915" and endkey "user4960" 2015-04-12 22:49:03,361 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 346cd14c69094045a7a853d9f974800d for table ycsb with start key "user1495" and endkey "user1540" 2015-04-12 22:49:03,501 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 95cbdee3bfd0424fbf1ac5b60f5bb7a7 for table ycsb with start key "user8065" and endkey "user8110" 2015-04-12 22:49:03,507 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-28' to 172.26.0.28 in 3588611ns 2015-04-12 22:49:03,519 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet cf096948621a49e6a7a5b55be20eb7c6 for table ycsb with start key "user3160" and endkey "user3205" 8 sec: 56079 operations; 12871 current ops/sec; [INSERT AverageLatency(us)=1237.36] 2015-04-12 22:49:03,594 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-231' to 172.26.3.231 in 4316474ns 2015-04-12 22:49:03,615 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-68' to 172.26.0.68 in 6044693ns 2015-04-12 22:49:03,651 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e59a4f9ef47d4e209c98fd73e66073df for table ycsb with start key "user3250" and endkey "user3295" 2015-04-12 22:49:03,660 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 182c18e3050349e391a3ebb9634f805e for table ycsb with start key "user7975" and endkey "user8020" 2015-04-12 22:49:03,665 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 4f45341b1f3c4092bb73aadb05df5736 for table ycsb with start key "user7390" and endkey "user7435" 9 sec: 65045 operations; 8957.04 current ops/sec; [INSERT AverageLatency(us)=1725.7] 2015-04-12 22:49:04,728 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0357fbfe74fc48c4b59f9fdaa7bdb7e2 for table ycsb with start key "user4780" and endkey "user4825" 2015-04-12 22:49:04,876 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-6' to 172.26.0.6 in 15013750ns 2015-04-12 22:49:04,879 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 577153da615f4f58af400d6a68319bc0 for table ycsb with start key "user8560" and endkey "user8605" 2015-04-12 22:49:04,941 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet d55b885b06174162b488c3499e8fe995 for table ycsb with start key "user7255" and endkey "user7300" 2015-04-12 22:49:04,949 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-1-52' to 172.26.1.52 in 4558001ns 2015-04-12 22:49:04,949 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet f1546aa32c6d47cab388c6fda715cb45 for table ycsb with start key "user8425" and endkey "user8470" 2015-04-12 22:49:04,992 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b96d9e8d68794fe7b40f31b055e8fd30 for table ycsb with start key "user2665" and endkey "user2710" 2015-04-12 22:49:05,057 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 643a1fa4b67a40afa1725f00c4e965d8 for table ycsb with start key "user9010" and endkey "user9055" 2015-04-12 22:49:05,253 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-1-35' to 172.26.1.35 in 125813849ns 2015-04-12 22:49:05,259 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-206' to 172.26.3.206 in 3555803ns 10 sec: 83939 operations; 18894 current ops/sec; [INSERT AverageLatency(us)=841.24] 11 sec: 94846 operations; 10202.99 current ops/sec; [INSERT AverageLatency(us)=745.46] 2015-04-12 22:49:06,695 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-176' to 172.26.0.176 in 9105570ns 2015-04-12 22:49:06,755 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-215' to 172.26.0.215 in 4607982ns 2015-04-12 22:49:06,775 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet c4caf51320c1446faf0efe3f5263353f for table ycsb with start key "user5635" and endkey "user5680" 2015-04-12 22:49:06,840 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-44' to 172.26.0.44 in 6944182ns 2015-04-12 22:49:06,845 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e3f1fec6271c4d89b63ac1a6895bbd2a for table ycsb with start key "user2215" and endkey "user2260" 2015-04-12 22:49:06,986 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-171' to 172.26.0.171 in 6791901ns 2015-04-12 22:49:07,000 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-244' to 172.26.3.244 in 4755048ns 2015-04-12 22:49:07,048 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 484a73f28984423c970900be6b6d6bab for table ycsb with start key "user6670" and endkey "user6715" 2015-04-12 22:49:07,064 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 595aeb2406f64c0c9b21daf730b58cff for table ycsb with start key "user4690" and endkey "user4735" 2015-04-12 22:49:07,086 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 7b36c79df5ee4af89d6a5966a9d890b7 for table ycsb with start key "user9145" and endkey "user9190" 2015-04-12 22:49:07,110 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 58c3e978f3a74c81bf9e17d134cb8380 for table ycsb with start key "user5725" and endkey "user5770" 2015-04-12 22:49:07,629 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:49:07,643 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:49:07,650 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:49:07,650 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:49:07,651 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:49:07,652 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:49:07,652 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 12 sec: 121639 operations; 26793 current ops/sec; [INSERT AverageLatency(us)=923.07] 2015-04-12 22:49:07,655 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x03d3c92f, /172.26.3.231:36492 :> /172.26.1.49:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:49:07,680 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,694 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,695 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,696 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,698 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,698 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,699 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,700 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,700 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:07,701 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xa20ac19e, /172.26.3.231:39692 :> /172.26.3.226:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:49:08,043 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 48f4fd70169a4db6a9c546b85a60a9e6 for table ycsb with start key "user3385" and endkey "user3430" 2015-04-12 22:49:08,155 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 55cee3810bd24d6a8ad8638b23491e7b for table ycsb with start key "user3745" and endkey "user3790" 2015-04-12 22:49:08,259 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-225' to 172.26.3.225 in 19567434ns 13 sec: 148567 operations; 26928 current ops/sec; [INSERT AverageLatency(us)=583.34] 14 sec: 151425 operations; 2855.14 current ops/sec; [INSERT AverageLatency(us)=5286.23] 2015-04-12 22:49:09,677 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-138' to 172.26.0.138 in 5706070ns 2015-04-12 22:49:09,803 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet f137ac6a562e45c582883cbc20479f99 for table ycsb with start key "user6805" and endkey "user6850" 2015-04-12 22:49:09,809 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 4533f9d62ad246fea30b8166e1fd7193 for table ycsb with start key "user2575" and endkey "user2620" 2015-04-12 22:49:09,827 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b281c896bfa44701a3da552dd478f496 for table ycsb with start key "user8335" and endkey "user8380" 2015-04-12 22:49:09,884 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 2a971dbdbf25472c8215e9a00277af7f for table ycsb with start key "user2080" and endkey "user2125" 2015-04-12 22:49:09,934 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 75bdc35934e34102ac5d7e86091e9ce1 for table ycsb with start key "user3610" and endkey "user3655" 2015-04-12 22:49:09,941 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-185' to 172.26.3.185 in 3081494ns 2015-04-12 22:49:09,942 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 7664c3a17bfc41089a8280757a106867 for table ycsb with start key "user1405" and endkey "user1450" 2015-04-12 22:49:09,979 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-1-38' to 172.26.1.38 in 6106267ns 2015-04-12 22:49:10,026 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet c1f52273a4e34617a350e55d04aa4b57 for table ycsb with start key "user5860" and endkey "user5905" 2015-04-12 22:49:10,027 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 807fb96d057f4c4c9fbc5c7f24452863 for table ycsb with start key "user4330" and endkey "user4375" 2015-04-12 22:49:10,057 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet b74da296d3ed4510bad0e90a93cfc4ae for table ycsb with start key "user2440" and endkey "user2485" 2015-04-12 22:49:10,085 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-38' to 172.26.0.38 in 5981774ns 2015-04-12 22:49:10,090 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 082bb50b4f554f158317dce06826e944 for table ycsb with start key "user9550" and endkey "user9595" 2015-04-12 22:49:10,130 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet e17b1339ccea4a4a87c956c3f2d039c6 for table ycsb with start key "user8200" and endkey "user8245" 2015-04-12 22:49:10,221 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet d1cfb9fc383e49f68a91f049c544e945 for table ycsb with start key "user9640" and endkey "user9685" 2015-04-12 22:49:10,468 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-1-26' to 172.26.1.26 in 214981216ns 2015-04-12 22:49:10,468 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0b3aa654d164490ab8932553954efe85 for table ycsb with start key "user1045" and endkey "user1090" 2015-04-12 22:49:10,536 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,536 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,569 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,570 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,570 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,570 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,570 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,570 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,571 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,573 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,573 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,576 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,576 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,577 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:49:10,577 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 15 sec: 197411 operations; 45986 current ops/sec; [INSERT AverageLatency(us)=329.77] 16 sec: 221492 operations; 17299.57 current ops/sec; [INSERT AverageLatency(us)=226.39] 17 sec: 273506 operations; 52014 current ops/sec; [INSERT AverageLatency(us)=552.67] 18 sec: 279116 operations; 4542.51 current ops/sec; [INSERT AverageLatency(us)=178.4] 2015-04-12 22:49:14,576 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-247' to 172.26.0.247 in 3274245ns 19 sec: 315560 operations; 36298.8 current ops/sec; [INSERT AverageLatency(us)=759.48] 2015-04-12 22:49:15,542 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-1-49' to 172.26.1.49 in 6036999ns 21 sec: 327006 operations; 7948.61 current ops/sec; [INSERT AverageLatency(us)=581.07] Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:49:17,238 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xa780619b, /172.26.3.231:45533 => /172.26.0.14:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:49:17,466 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 100f480db4fe44b1bbed803d08324229 2015-04-12 22:49:17,467 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 100f480db4fe44b1bbed803d08324229 2015-04-12 22:49:17,467 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 100f480db4fe44b1bbed803d08324229 2015-04-12 22:49:17,467 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 100f480db4fe44b1bbed803d08324229 2015-04-12 22:49:17,468 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x0362c7d6, /172.26.3.231:54551 :> /172.26.0.182:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:49:17,498 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 54e547f98a6d4e569a23ede2a2828a83 2015-04-12 22:49:17,498 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 54e547f98a6d4e569a23ede2a2828a83 2015-04-12 22:49:17,498 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 54e547f98a6d4e569a23ede2a2828a83 2015-04-12 22:49:17,529 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 54e547f98a6d4e569a23ede2a2828a83 2015-04-12 22:49:17,529 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 54e547f98a6d4e569a23ede2a2828a83 2015-04-12 22:49:17,529 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 54e547f98a6d4e569a23ede2a2828a83 2015-04-12 22:49:17,530 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 54e547f98a6d4e569a23ede2a2828a83 2015-04-12 22:49:17,530 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 54e547f98a6d4e569a23ede2a2828a83 2015-04-12 22:49:17,530 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x0d0c17c6, /172.26.3.231:39076 :> /172.26.0.59:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 22 sec: 349392 operations; 22386 current ops/sec; [INSERT AverageLatency(us)=1451.6] 2015-04-12 22:49:18,272 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 43148c4f3062449b856cf35755ed98e4 for table ycsb with start key "user1270" and endkey "user1315" 23 sec: 369244 operations; 19852 current ops/sec; [INSERT AverageLatency(us)=670.65] 2015-04-12 22:49:19,227 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet a8054cc6b9584893841f098b475c9f6a for table ycsb with start key "user9685" and endkey "user9730" 2015-04-12 22:49:19,230 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 4fb17be5ac424707bf9172c52ca080b5 for table ycsb with start key "user9460" and endkey "user9505" 24 sec: 383937 operations; 14678.32 current ops/sec; [INSERT AverageLatency(us)=692.4] 25 sec: 385929 operations; 1374.74 current ops/sec; [INSERT AverageLatency(us)=369.44] 2015-04-12 22:49:21,266 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x021fd450, /172.26.3.231:58745 => /172.26.1.37:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:49:21,370 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:49:21,552 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 0b13aebec0af487395ecd791d4e18df1 for table ycsb with start key "user9235" and endkey "user9280" 2015-04-12 22:49:21,813 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-190' to 172.26.3.190 in 6044747ns 2015-04-12 22:49:21,982 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-171' to 172.26.0.171 in 6711688ns 26 sec: 394822 operations; 8884.12 current ops/sec; [INSERT AverageLatency(us)=1413.15] 2015-04-12 22:49:22,418 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 16c0a1dd4680440aacec12e23b6c2551 for table ycsb with start key "user9280" and endkey "user9325" Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponseWaiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutationGot exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse 2015-04-12 22:49:23,048 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 1c685215d02042e698fd7c1bcadba2d0 for table ycsb with start key "user9325" and endkey "user9370" 27 sec: 407012 operations; 12129.35 current ops/sec; [INSERT AverageLatency(us)=2822.85] Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:49:23,264 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-249' to 172.26.3.249 in 6119469ns 2015-04-12 22:49:23,538 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-47' to 172.26.0.47 in 21198708ns 28 sec: 422887 operations; 13638.32 current ops/sec; [INSERT AverageLatency(us)=287.35] 2015-04-12 22:49:24,472 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 3e0df0a43e634d72888a4e9ebea3b73c for table ycsb with start key "user9595" and endkey "user9640" 2015-04-12 22:49:24,499 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 1b843fd4f2d149c596618c207f16db36 for table ycsb with start key "user9505" and endkey "user9550" 2015-04-12 22:49:24,691 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet 8f965641a17b4fae887d762c83ef7b7b for table ycsb with start key "user9415" and endkey "user9460" 2015-04-12 22:49:24,916 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-100' to 172.26.0.100 in 146324431ns Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutationGot exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponseWaiting more than 10 seconds for an insert or mutation 2015-04-12 22:49:25,120 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-92' to 172.26.0.92 in 6164585ns Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse 30 sec: 438576 operations; 12511.16 current ops/sec; [INSERT AverageLatency(us)=2892.75] Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:49:26,607 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.discoverTablets(KuduClient.java:1149)] Discovering tablet c5fc7ff5909b41979263e2f2ac336eaf for table ycsb with start key "user9370" and endkey "user9415" 31 sec: 469969 operations; 31361.64 current ops/sec; [INSERT AverageLatency(us)=470.26] 33 sec: 488112 operations; 9318.44 current ops/sec; [INSERT AverageLatency(us)=580.92] 2015-04-12 22:49:28,723 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-168' to 172.26.0.168 in 3978158ns 34 sec: 523462 operations; 34154.59 current ops/sec; [INSERT AverageLatency(us)=869.37] 2015-04-12 22:49:30,222 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-225' to 172.26.3.225 in 6559263ns 36 sec: 555133 operations; 16109.36 current ops/sec; [INSERT AverageLatency(us)=404.93] 2015-04-12 22:49:31,810 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-248' to 172.26.0.248 in 5292860ns 2015-04-12 22:49:32,518 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-75' to 172.26.0.75 in 28734716ns 37 sec: 590882 operations; 35749 current ops/sec; [INSERT AverageLatency(us)=556.12] Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation 38 sec: 611504 operations; 20622 current ops/sec; [INSERT AverageLatency(us)=193.21] 2015-04-12 22:49:33,771 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-176' to 172.26.0.176 in 5709250ns 39 sec: 615829 operations; 2813.92 current ops/sec; [INSERT AverageLatency(us)=125.43] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse Got exception java.lang.ClassCastException: java.lang.ClassCastException cannot be cast to kudu.rpc.OperationResponse Waiting more than 10 seconds for an insert or mutation Got exception java.lang.ClassCastException: kudu.rpc.RowsWithErrorException cannot be cast to kudu.rpc.OperationResponse 40 sec: 627212 operations; 11360.28 current ops/sec; [INSERT AverageLatency(us)=605.15] Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:49:36,291 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-195' to 172.26.3.195 in 6024738ns 2015-04-12 22:49:36,629 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-1-71' to 172.26.1.71 in 5034653ns 41 sec: 631356 operations; 4144 current ops/sec; [INSERT AverageLatency(us)=94.62] 2015-04-12 22:49:38,016 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-144' to 172.26.0.144 in 9873550ns 42 sec: 631356 operations; 0 current ops/sec; 2015-04-12 22:49:38,342 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-6' to 172.26.0.6 in 5720247ns 2015-04-12 22:49:38,359 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-64' to 172.26.0.64 in 6304040ns Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:49:38,687 (New I/O worker #1) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-133' to 172.26.0.133 in 5107423ns Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 43 sec: 657416 operations; 22799.65 current ops/sec; [INSERT AverageLatency(us)=3738.81] 44 sec: 714556 operations; 51992.72 current ops/sec; [INSERT AverageLatency(us)=221.08] 46 sec: 755766 operations; 25328.83 current ops/sec; [INSERT AverageLatency(us)=308.4] 47 sec: 818789 operations; 59010.3 current ops/sec; [INSERT AverageLatency(us)=414.36] 48 sec: 854628 operations; 24330.62 current ops/sec; [INSERT AverageLatency(us)=315.14] 49 sec: 898515 operations; 42077.66 current ops/sec; [INSERT AverageLatency(us)=539.47] 51 sec: 925951 operations; 26871.69 current ops/sec; [INSERT AverageLatency(us)=533.34] 52 sec: 931071 operations; 4425.24 current ops/sec; [INSERT AverageLatency(us)=467.14] 53 sec: 961772 operations; 29923 current ops/sec; [INSERT AverageLatency(us)=873.81] 54 sec: 979146 operations; 10568.13 current ops/sec; [INSERT AverageLatency(us)=439.21] 2015-04-12 22:49:51,203 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ~~~masterTableHack~~~ 2015-04-12 22:49:51,204 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ~~~masterTableHack~~~ 2015-04-12 22:49:51,207 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ~~~masterTableHack~~~ 2015-04-12 22:49:51,207 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ~~~masterTableHack~~~ 2015-04-12 22:49:51,207 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ~~~masterTableHack~~~ 2015-04-12 22:49:51,208 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ~~~masterTableHack~~~ 2015-04-12 22:49:51,208 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ~~~masterTableHack~~~ 55 sec: 1024598 operations; 42798.49 current ops/sec; [INSERT AverageLatency(us)=573.77] 57 sec: 1045792 operations; 12518.61 current ops/sec; [INSERT AverageLatency(us)=295.5] 2015-04-12 22:49:53,383 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,512 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,513 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,513 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,513 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,513 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,513 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,513 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,514 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,514 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:49:53,514 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xfb8c0a72, /172.26.3.231:44053 :> /172.26.0.99:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 58 sec: 1093871 operations; 44435.3 current ops/sec; [INSERT AverageLatency(us)=621.58] 59 sec: 1134365 operations; 40494 current ops/sec; [INSERT AverageLatency(us)=366.24] 61 sec: 1142327 operations; 5506.22 current ops/sec; [INSERT AverageLatency(us)=213.54] 62 sec: 1186870 operations; 43414.23 current ops/sec; [INSERT AverageLatency(us)=642.12] 63 sec: 1195160 operations; 5821.63 current ops/sec; [INSERT AverageLatency(us)=303.05] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 64 sec: 1226723 operations; 31563 current ops/sec; [INSERT AverageLatency(us)=903.52] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 65 sec: 1252246 operations; 25523 current ops/sec; [INSERT AverageLatency(us)=640.47] 66 sec: 1252674 operations; 319.4 current ops/sec; [INSERT AverageLatency(us)=239.98] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 68 sec: 1276471 operations; 12413.67 current ops/sec; [INSERT AverageLatency(us)=1286] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 69 sec: 1312716 operations; 36245 current ops/sec; [INSERT AverageLatency(us)=667.62] Waiting more than 10 seconds for an insert or mutation 72 sec: 1349069 operations; 16675.69 current ops/sec; [INSERT AverageLatency(us)=342.86] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 74 sec: 1393144 operations; 21374.88 current ops/sec; [INSERT AverageLatency(us)=648.28] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 75 sec: 1439335 operations; 46191 current ops/sec; [INSERT AverageLatency(us)=628.25] 2015-04-12 22:50:10,692 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x8e40e2aa, /172.26.3.231:54059 => /172.26.3.231:7051] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,701 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x8e40e2aa, /172.26.3.231:54059 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,760 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,762 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,952 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,953 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,953 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,953 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,954 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,954 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,954 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,966 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,966 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,966 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,967 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,967 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,968 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,968 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,971 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,972 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,972 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,972 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,973 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:10,973 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:10,974 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:50:10,999 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,000 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,000 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,001 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,001 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,001 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,002 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,018 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,018 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,018 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,019 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,042 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,042 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,043 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,043 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,044 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,044 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,044 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,045 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,083 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,084 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,084 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,084 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,085 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,085 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,086 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:11,099 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException at kudu.rpc.TabletClient.sendRpc(TabletClient.java:174) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:11,101 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). Waiting more than 10 seconds for an insert or mutation 77 sec: 1467076 operations; 12696.11 current ops/sec; [INSERT AverageLatency(us)=410.87] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 78 sec: 1502135 operations; 35059 current ops/sec; [INSERT AverageLatency(us)=977.02] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 79 sec: 1531947 operations; 29812 current ops/sec; [INSERT AverageLatency(us)=352.2] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 81 sec: 1547947 operations; 9179.58 current ops/sec; [INSERT AverageLatency(us)=472.16] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 82 sec: 1581928 operations; 31907.04 current ops/sec; [INSERT AverageLatency(us)=929.31] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 83 sec: 1611547 operations; 29471.64 current ops/sec; [INSERT AverageLatency(us)=493.49] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 84 sec: 1645283 operations; 33702.3 current ops/sec; [INSERT AverageLatency(us)=449.66] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 86 sec: 1690604 operations; 22347.63 current ops/sec; [INSERT AverageLatency(us)=464.25] Waiting more than 10 seconds for an insert or mutation 87 sec: 1736623 operations; 46019 current ops/sec; [INSERT AverageLatency(us)=554.9] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 88 sec: 1764771 operations; 27488.28 current ops/sec; [INSERT AverageLatency(us)=562.59] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 89 sec: 1796650 operations; 31284.59 current ops/sec; [INSERT AverageLatency(us)=410.37] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 91 sec: 1826266 operations; 15636.75 current ops/sec; [INSERT AverageLatency(us)=328.14] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 92 sec: 1858558 operations; 30783.6 current ops/sec; [INSERT AverageLatency(us)=1160.39] Waiting more than 10 seconds for an insert or mutation 93 sec: 1891715 operations; 30757.88 current ops/sec; [INSERT AverageLatency(us)=426.49] 94 sec: 1924803 operations; 33088 current ops/sec; [INSERT AverageLatency(us)=433.42] 95 sec: 1958654 operations; 19703.73 current ops/sec; [INSERT AverageLatency(us)=368.03] 96 sec: 2008996 operations; 50291.71 current ops/sec; [INSERT AverageLatency(us)=520.96] 97 sec: 2050095 operations; 40894.53 current ops/sec; [INSERT AverageLatency(us)=285.57] 98 sec: 2086691 operations; 36559.44 current ops/sec; [INSERT AverageLatency(us)=359.33] 101 sec: 2124281 operations; 16940.06 current ops/sec; [INSERT AverageLatency(us)=410.36] 102 sec: 2164354 operations; 40032.97 current ops/sec; [INSERT AverageLatency(us)=698.39] 103 sec: 2191813 operations; 27459 current ops/sec; [INSERT AverageLatency(us)=396.73] 2015-04-12 22:50:39,190 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 082bb50b4f554f158317dce06826e944 2015-04-12 22:50:39,190 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 082bb50b4f554f158317dce06826e944 2015-04-12 22:50:39,190 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 082bb50b4f554f158317dce06826e944 2015-04-12 22:50:39,191 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 082bb50b4f554f158317dce06826e944 104 sec: 2220014 operations; 16041.52 current ops/sec; [INSERT AverageLatency(us)=327.33] 105 sec: 2265208 operations; 45148.85 current ops/sec; [INSERT AverageLatency(us)=540.38] 106 sec: 2306876 operations; 39458.33 current ops/sec; [INSERT AverageLatency(us)=294.32] 108 sec: 2323831 operations; 12206.62 current ops/sec; [INSERT AverageLatency(us)=308.28] 2015-04-12 22:50:44,381 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x5eba027f, /172.26.3.231:54169 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:44,381 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:50:44,382 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x5eba027f, /172.26.3.231:54169 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:44,382 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 109 sec: 2367995 operations; 44119.88 current ops/sec; [INSERT AverageLatency(us)=608.15] 110 sec: 2398607 operations; 29923.75 current ops/sec; [INSERT AverageLatency(us)=387.49] 112 sec: 2422642 operations; 13352.78 current ops/sec; [INSERT AverageLatency(us)=347.27] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 113 sec: 2464319 operations; 41635.36 current ops/sec; [INSERT AverageLatency(us)=763.33] 115 sec: 2498959 operations; 16386 current ops/sec; [INSERT AverageLatency(us)=325.86] 116 sec: 2546678 operations; 47671.33 current ops/sec; [INSERT AverageLatency(us)=594.25] 117 sec: 2578369 operations; 31470.71 current ops/sec; [INSERT AverageLatency(us)=358.18] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 118 sec: 2593054 operations; 9693.07 current ops/sec; [INSERT AverageLatency(us)=303.38] Waiting more than 10 seconds for an insert or mutation 119 sec: 2633135 operations; 38465.45 current ops/sec; [INSERT AverageLatency(us)=613.46] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:50:56,201 (New I/O worker #4) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-3-206' to 172.26.3.206 in 15477866ns Waiting more than 10 seconds for an insert or mutation 122 sec: 2666644 operations; 14972.74 current ops/sec; [INSERT AverageLatency(us)=491.76] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 123 sec: 2704906 operations; 34193.03 current ops/sec; [INSERT AverageLatency(us)=862.52] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:50:59,563 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xe67c25c8, /172.26.3.231:34811 => /172.26.3.206:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:50:59,564 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 082bb50b4f554f158317dce06826e944 2015-04-12 22:50:59,564 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 082bb50b4f554f158317dce06826e944 Waiting more than 10 seconds for an insert or mutation 125 sec: 2732583 operations; 12873.02 current ops/sec; [INSERT AverageLatency(us)=436.76] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 126 sec: 2771503 operations; 38920 current ops/sec; [INSERT AverageLatency(us)=790.4] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 127 sec: 2800791 operations; 29229.54 current ops/sec; [INSERT AverageLatency(us)=585.57] 129 sec: 2823431 operations; 12331.15 current ops/sec; [INSERT AverageLatency(us)=287.92] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 130 sec: 2867658 operations; 44227 current ops/sec; [INSERT AverageLatency(us)=766.63] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 131 sec: 2897112 operations; 29018.72 current ops/sec; [INSERT AverageLatency(us)=351.3] 143 sec: 2907640 operations; 887.09 current ops/sec; [INSERT AverageLatency(us)=422.08] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:18,772 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5dd9d72bacf74e498e188514b2e906ff 2015-04-12 22:51:18,781 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xee2654bb, /172.26.3.231:33900 :> /172.26.0.246:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,783 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f843f12e91b74ab39e2587badd5ff13c 2015-04-12 22:51:18,794 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 37ea00dfed7a4694af52000257a344e1 2015-04-12 22:51:18,795 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 39c3319fc9284a1f84318891817eb2f9 2015-04-12 22:51:18,777 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fb55343f179547808b203dee1a9c7c51 2015-04-12 22:51:18,795 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:18,796 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xa2f0b147, /172.26.3.231:54337 :> /172.26.0.113:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,797 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3d05b1860a0549e0bfa403c28d61e229 2015-04-12 22:51:18,797 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3a910f1013c24607b6804b7390f53ef3 2015-04-12 22:51:18,797 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x3dc44ca2, /172.26.3.231:55932 :> /172.26.0.167:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,797 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:51:18,798 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x87794db1, /172.26.3.231:42885 :> /172.26.1.17:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,814 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 25e461781b7d4af5a803e8af41ddc3b0 2015-04-12 22:51:18,815 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e91c5b97259c40b4b3301e52b27b36b2 2015-04-12 22:51:18,815 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e91c5b97259c40b4b3301e52b27b36b2 2015-04-12 22:51:18,815 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xb3f31eff, /172.26.3.231:35210 :> /172.26.3.211:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,816 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ee6f559224624922887eef4ca0ad9ea4 2015-04-12 22:51:18,816 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf599790d, /172.26.3.231:57830 :> /172.26.0.65:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,822 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4ba5a22e14a84329b925dc3e17d20fb8 2015-04-12 22:51:18,823 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x4249cb5d, /172.26.3.231:54229 :> /172.26.0.164:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,824 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 2b221b5a3264493ca51b5fbd9f3a17b7 2015-04-12 22:51:18,832 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5f48d1bdbfa84880ba72d0ca71ecc6e4 2015-04-12 22:51:18,832 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4533f9d62ad246fea30b8166e1fd7193 2015-04-12 22:51:18,835 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 05ac82e5cdcf4d7494929a7191ca76d2 2015-04-12 22:51:18,835 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5f48d1bdbfa84880ba72d0ca71ecc6e4 2015-04-12 22:51:18,835 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x0e8e7dda, /172.26.3.231:35801 :> /172.26.0.158:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,838 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3e0df0a43e634d72888a4e9ebea3b73c 2015-04-12 22:51:18,838 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82cc726f2a1f47a6bd8c7716b2b49729 2015-04-12 22:51:18,838 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3e0df0a43e634d72888a4e9ebea3b73c 2015-04-12 22:51:18,839 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82cc726f2a1f47a6bd8c7716b2b49729 2015-04-12 22:51:18,839 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xd7a14f17, /172.26.3.231:60498 :> /172.26.0.64:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,850 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 292b2c6e55004754ad389b24a8275071 2015-04-12 22:51:18,851 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 99c4e9b15f9a4642932a666528706d1d 2015-04-12 22:51:18,851 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 99c4e9b15f9a4642932a666528706d1d 2015-04-12 22:51:18,851 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x62551ac0, /172.26.3.231:47932 :> /172.26.3.242:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,852 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c1f52273a4e34617a350e55d04aa4b57 2015-04-12 22:51:18,852 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x9fdd58ea, /172.26.3.231:41492 :> /172.26.1.19:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,878 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c5fc7ff5909b41979263e2f2ac336eaf 2015-04-12 22:51:18,879 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x533b5b58, /172.26.3.231:46529 :> /172.26.0.133:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,881 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1c685215d02042e698fd7c1bcadba2d0 2015-04-12 22:51:18,882 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x7d19caa0, /172.26.3.231:56657 :> /172.26.0.28:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,882 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 6cfd594c7fac420090441d55bc7174c7 2015-04-12 22:51:18,882 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c4caf51320c1446faf0efe3f5263353f 2015-04-12 22:51:18,882 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b96d9e8d68794fe7b40f31b055e8fd30 2015-04-12 22:51:18,883 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xc4cf854f, /172.26.3.231:35845 :> /172.26.0.176:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,884 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cb0a1d070bb349d5ad68e1953559de7e 2015-04-12 22:51:18,884 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 794fd028476141bab133ff334c623583 2015-04-12 22:51:18,884 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x3f603e27, /172.26.3.231:35380 :> /172.26.0.53:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,888 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x09e2e211, /172.26.3.231:48713 => /172.26.0.75:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,889 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 48f4fd70169a4db6a9c546b85a60a9e6 2015-04-12 22:51:18,889 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 48f4fd70169a4db6a9c546b85a60a9e6 2015-04-12 22:51:18,889 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 48f4fd70169a4db6a9c546b85a60a9e6 2015-04-12 22:51:18,890 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 48f4fd70169a4db6a9c546b85a60a9e6 2015-04-12 22:51:18,891 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x923da3a5, /172.26.3.231:46835 => /172.26.0.68:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,906 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d1ee46c96b9d4a3187936f69d316fe1c 2015-04-12 22:51:18,906 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d1ee46c96b9d4a3187936f69d316fe1c 2015-04-12 22:51:18,906 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d1ee46c96b9d4a3187936f69d316fe1c 2015-04-12 22:51:18,907 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d1ee46c96b9d4a3187936f69d316fe1c 2015-04-12 22:51:18,907 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d1ee46c96b9d4a3187936f69d316fe1c 2015-04-12 22:51:18,907 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1bab3b084f374671a768eadb8d37b8ce 2015-04-12 22:51:18,907 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1bab3b084f374671a768eadb8d37b8ce 2015-04-12 22:51:18,914 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b281c896bfa44701a3da552dd478f496 2015-04-12 22:51:18,917 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b281c896bfa44701a3da552dd478f496 2015-04-12 22:51:18,917 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 8db6003ef43948a0a806a1ecd312c7eb 2015-04-12 22:51:18,931 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 758e6a5fa0e64ef99390a5c4011c4e5e 2015-04-12 22:51:18,931 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 8db6003ef43948a0a806a1ecd312c7eb 2015-04-12 22:51:18,932 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 8db6003ef43948a0a806a1ecd312c7eb 2015-04-12 22:51:18,932 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x76958c3f, /172.26.3.231:53942 :> /172.26.0.6:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,931 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x7d78db66, /172.26.3.231:45207 => /172.26.0.138:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,937 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 643a1fa4b67a40afa1725f00c4e965d8 2015-04-12 22:51:18,937 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 19ce84717eea4585b9a1696f383447a0 2015-04-12 22:51:18,937 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 19ce84717eea4585b9a1696f383447a0 2015-04-12 22:51:18,938 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 11c8079ea2da4587899ff07de3edaaeb 2015-04-12 22:51:18,938 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 643a1fa4b67a40afa1725f00c4e965d8 2015-04-12 22:51:18,974 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x38fb7b8b, /172.26.3.231:50901 => /172.26.1.71:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,977 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x7d78db66, /172.26.3.231:45207 :> /172.26.0.138:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,982 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x6b3230d1, /172.26.3.231:42461 => /172.26.0.100:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,982 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 8db6003ef43948a0a806a1ecd312c7eb 2015-04-12 22:51:18,982 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 49ca3d948a5649bbbec6000ff1f6a0c1 2015-04-12 22:51:18,983 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 27c587303bec424586cabce85ebab2fe 2015-04-12 22:51:18,983 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 568e7aa39c9d416f95113ae674b2607d 2015-04-12 22:51:18,983 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4f45341b1f3c4092bb73aadb05df5736 2015-04-12 22:51:18,983 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x6b3230d1, /172.26.3.231:42461 :> /172.26.0.100:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,984 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x567208da, /172.26.3.231:58683 => /172.26.1.38:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,984 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 65c9c301df1a4b288a6924b805f8130b 2015-04-12 22:51:18,985 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b0cca8cc157c4957ba8466717c47f7c1 2015-04-12 22:51:18,986 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 65c9c301df1a4b288a6924b805f8130b 2015-04-12 22:51:18,986 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b0cca8cc157c4957ba8466717c47f7c1 2015-04-12 22:51:18,986 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b0cca8cc157c4957ba8466717c47f7c1 2015-04-12 22:51:18,986 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e59a4f9ef47d4e209c98fd73e66073df 2015-04-12 22:51:18,986 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0357fbfe74fc48c4b59f9fdaa7bdb7e2 2015-04-12 22:51:18,990 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0357fbfe74fc48c4b59f9fdaa7bdb7e2 2015-04-12 22:51:18,990 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0357fbfe74fc48c4b59f9fdaa7bdb7e2 2015-04-12 22:51:18,990 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 2b221b5a3264493ca51b5fbd9f3a17b7 2015-04-12 22:51:18,990 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:18,991 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:18,991 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from dad8029d22bd47bfafb1b0168e58fe4c 2015-04-12 22:51:18,991 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0357fbfe74fc48c4b59f9fdaa7bdb7e2 2015-04-12 22:51:18,991 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x38fb7b8b, /172.26.3.231:50901 :> /172.26.1.71:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,992 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xd1e58763, /172.26.3.231:34942 => /172.26.3.190:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:18,992 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 22:51:19,106 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 37ea00dfed7a4694af52000257a344e1 2015-04-12 22:51:19,114 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x567208da, /172.26.3.231:58683 :> /172.26.1.38:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,114 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x567208da, /172.26.3.231:58683 :> /172.26.1.38:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,114 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x567208da, /172.26.3.231:58683 :> /172.26.1.38:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,115 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x567208da, /172.26.3.231:58683 :> /172.26.1.38:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,113 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 22:51:19,117 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 22:51:19,117 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5f48d1bdbfa84880ba72d0ca71ecc6e4 2015-04-12 22:51:19,117 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4533f9d62ad246fea30b8166e1fd7193 2015-04-12 22:51:19,117 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x15894d1d, /172.26.3.231:56766 => /172.26.0.57:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,118 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 68a6dd22e5634e97b27e4656a212cb18 2015-04-12 22:51:19,118 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 68a6dd22e5634e97b27e4656a212cb18 2015-04-12 22:51:19,118 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fac7c03d320440679149f7c3419174de 2015-04-12 22:51:19,118 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fac7c03d320440679149f7c3419174de 2015-04-12 22:51:19,119 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 7823220768a548788bdba28b558352b0 2015-04-12 22:51:19,119 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fac7c03d320440679149f7c3419174de 2015-04-12 22:51:19,119 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 7823220768a548788bdba28b558352b0 2015-04-12 22:51:19,119 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 41207f4c0685405d964530dbdd8a6065 2015-04-12 22:51:19,119 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 7823220768a548788bdba28b558352b0 2015-04-12 22:51:19,120 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x2fa79957, /172.26.3.231:43005 => /172.26.3.239:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,132 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e17b1339ccea4a4a87c956c3f2d039c6 2015-04-12 22:51:19,132 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 7823220768a548788bdba28b558352b0 2015-04-12 22:51:19,133 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 7823220768a548788bdba28b558352b0 2015-04-12 22:51:19,133 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x15894d1d, /172.26.3.231:56766 :> /172.26.0.57:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,134 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x15894d1d, /172.26.3.231:56766 :> /172.26.0.57:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,135 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xffc82bc2, /172.26.3.231:47735 => /172.26.0.49:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,136 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f137ac6a562e45c582883cbc20479f99 2015-04-12 22:51:19,136 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f137ac6a562e45c582883cbc20479f99 2015-04-12 22:51:19,133 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 16c0a1dd4680440aacec12e23b6c2551 2015-04-12 22:51:19,138 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 346cd14c69094045a7a853d9f974800d 2015-04-12 22:51:19,139 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e17b1339ccea4a4a87c956c3f2d039c6 2015-04-12 22:51:19,139 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x2fa79957, /172.26.3.231:43005 :> /172.26.3.239:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,139 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x2fa79957, /172.26.3.231:43005 :> /172.26.3.239:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,140 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf96539ae, /172.26.3.231:60079 => /172.26.0.215:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,140 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 32949ffabbce443fbc16d6390720fade 2015-04-12 22:51:19,154 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 643a1fa4b67a40afa1725f00c4e965d8 2015-04-12 22:51:19,155 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xbe6a7417, /172.26.3.231:44023 => /172.26.3.224:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,156 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c335263a849d4463ab09828f901eced2 2015-04-12 22:51:19,156 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c335263a849d4463ab09828f901eced2 2015-04-12 22:51:19,157 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5ebd6ef592244001b8a568ffedec51d3 2015-04-12 22:51:19,154 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 22c5b8d0fd944b95bf656134e59c6561 2015-04-12 22:51:19,157 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 22c5b8d0fd944b95bf656134e59c6561 2015-04-12 22:51:19,157 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 22c5b8d0fd944b95bf656134e59c6561 2015-04-12 22:51:19,158 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 22c5b8d0fd944b95bf656134e59c6561 2015-04-12 22:51:19,158 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 22c5b8d0fd944b95bf656134e59c6561 2015-04-12 22:51:19,158 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 22c5b8d0fd944b95bf656134e59c6561 2015-04-12 22:51:19,158 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xffc82bc2, /172.26.3.231:47735 :> /172.26.0.49:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,159 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xffc82bc2, /172.26.3.231:47735 :> /172.26.0.49:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,159 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xffc82bc2, /172.26.3.231:47735 :> /172.26.0.49:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,160 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xbf2a075b, /172.26.3.231:37564 => /172.26.1.80:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,160 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fb55343f179547808b203dee1a9c7c51 2015-04-12 22:51:19,176 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e39aa4e04b2c411fa14e66efd4924587 2015-04-12 22:51:19,176 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ea8878afbfc149ff8fb15cf04dde6f6f 2015-04-12 22:51:19,176 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cf096948621a49e6a7a5b55be20eb7c6 2015-04-12 22:51:19,176 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ea8878afbfc149ff8fb15cf04dde6f6f 2015-04-12 22:51:19,177 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ea8878afbfc149ff8fb15cf04dde6f6f 2015-04-12 22:51:19,177 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fb55343f179547808b203dee1a9c7c51 2015-04-12 22:51:19,168 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0896a3f819444888a2e54a61257a68bd 2015-04-12 22:51:19,177 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from afecf7b47f7c42ad8be8ff7b7fcca5e2 2015-04-12 22:51:19,177 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 2b82ac405be945e783d4c493789dc378 2015-04-12 22:51:19,178 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 2b82ac405be945e783d4c493789dc378 2015-04-12 22:51:19,178 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0896a3f819444888a2e54a61257a68bd 2015-04-12 22:51:19,178 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x4c4eb0e8, /172.26.3.231:50803 :> /172.26.0.32:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,183 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e39aa4e04b2c411fa14e66efd4924587 2015-04-12 22:51:19,184 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e815f3695c1a4123a69811ae1f28d8cc 2015-04-12 22:51:19,184 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 778d3039dc464916af771fc0b2c9787a 2015-04-12 22:51:19,184 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xbf2a075b, /172.26.3.231:37564 :> /172.26.1.80:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,184 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xbf2a075b, /172.26.3.231:37564 :> /172.26.1.80:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,185 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xbf2a075b, /172.26.3.231:37564 :> /172.26.1.80:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,185 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x77ca8bfe, /172.26.3.231:44969 => /172.26.0.39:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,187 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 19b2be7173f64f9a90ed886253ec12c5 2015-04-12 22:51:19,187 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 19b2be7173f64f9a90ed886253ec12c5 2015-04-12 22:51:19,187 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 19b2be7173f64f9a90ed886253ec12c5 2015-04-12 22:51:19,201 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 48f4fd70169a4db6a9c546b85a60a9e6 2015-04-12 22:51:19,202 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x77ca8bfe, /172.26.3.231:44969 :> /172.26.0.39:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,203 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xbdfdc38f, /172.26.3.231:52344 => /172.26.0.70:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,204 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 22:51:19,208 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 22:51:19,212 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 22:51:19,212 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 87de27b6d47d4b2b8e9c9f11a827c93a 2015-04-12 22:51:19,212 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 22:51:19,212 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 22:51:19,213 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 22:51:19,213 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xbdfdc38f, /172.26.3.231:52344 :> /172.26.0.70:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,214 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xbdfdc38f, /172.26.3.231:52344 :> /172.26.0.70:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,215 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x018d4147, /172.26.3.231:56507 => /172.26.0.247:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,216 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82ba0cb62280430596448f39a3b52e65 2015-04-12 22:51:19,217 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82ba0cb62280430596448f39a3b52e65 2015-04-12 22:51:19,219 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e3924acd6e48462fb4e8d04c3664e036 2015-04-12 22:51:19,220 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e3924acd6e48462fb4e8d04c3664e036 2015-04-12 22:51:19,220 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82ba0cb62280430596448f39a3b52e65 2015-04-12 22:51:19,220 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e3924acd6e48462fb4e8d04c3664e036 2015-04-12 22:51:19,221 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82ba0cb62280430596448f39a3b52e65 2015-04-12 22:51:19,221 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 05ac82e5cdcf4d7494929a7191ca76d2 2015-04-12 22:51:19,221 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82ba0cb62280430596448f39a3b52e65 2015-04-12 22:51:19,221 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x018d4147, /172.26.3.231:56507 :> /172.26.0.247:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,222 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x018d4147, /172.26.3.231:56507 :> /172.26.0.247:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,222 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x018d4147, /172.26.3.231:56507 :> /172.26.0.247:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,223 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x24eddd6d, /172.26.3.231:43916 => /172.26.1.53:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,223 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 930f13ce130f4333bdba0bc4b4900673 2015-04-12 22:51:19,223 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 930f13ce130f4333bdba0bc4b4900673 2015-04-12 22:51:19,224 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4fb17be5ac424707bf9172c52ca080b5 2015-04-12 22:51:19,224 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4fb17be5ac424707bf9172c52ca080b5 2015-04-12 22:51:19,224 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 76d4ae4f2ac1446d9378448bb47addaa 2015-04-12 22:51:19,225 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf754330e, /172.26.3.231:59086 => /172.26.3.252:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,225 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b7aecd5368774804ab1e16285fca8434 2015-04-12 22:51:19,226 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 58c3e978f3a74c81bf9e17d134cb8380 2015-04-12 22:51:19,226 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 58c3e978f3a74c81bf9e17d134cb8380 2015-04-12 22:51:19,226 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 6e1d8107f384430cbabea9c8711b5d86 2015-04-12 22:51:19,239 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 58c3e978f3a74c81bf9e17d134cb8380 2015-04-12 22:51:19,239 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b7aecd5368774804ab1e16285fca8434 2015-04-12 22:51:19,241 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1481f0b5b9024d70883392336d607dae 2015-04-12 22:51:19,241 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 58c3e978f3a74c81bf9e17d134cb8380 2015-04-12 22:51:19,242 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf754330e, /172.26.3.231:59086 :> /172.26.3.252:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:19,278 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:19,278 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:19,278 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:19,279 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 7cd739b3c5914119a5800a5ec9f83882 2015-04-12 22:51:19,279 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:19,279 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from be7b707c3d8c408b9ca99766bede14cc 2015-04-12 22:51:19,279 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 7cd739b3c5914119a5800a5ec9f83882 2015-04-12 22:51:19,279 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:19,279 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:19,280 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:19,280 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x51fd6f60, /172.26.3.231:39908 :> /172.26.0.33:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:19,342 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4a1614a7cb4f4e4198f1a31cfdf1a844 2015-04-12 22:51:19,342 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4a1614a7cb4f4e4198f1a31cfdf1a844 2015-04-12 22:51:19,342 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0b13aebec0af487395ecd791d4e18df1 2015-04-12 22:51:19,343 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b18c47f8b11c495ea60f2ade6ed10a6a 2015-04-12 22:51:19,343 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 29e50b6b9df249beb72def616de450fe 2015-04-12 22:51:19,343 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 29e50b6b9df249beb72def616de450fe 2015-04-12 22:51:19,343 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4a1614a7cb4f4e4198f1a31cfdf1a844 2015-04-12 22:51:19,344 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b18c47f8b11c495ea60f2ade6ed10a6a 2015-04-12 22:51:19,344 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b18c47f8b11c495ea60f2ade6ed10a6a 2015-04-12 22:51:19,344 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82cc726f2a1f47a6bd8c7716b2b49729 2015-04-12 22:51:19,344 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b18c47f8b11c495ea60f2ade6ed10a6a 2015-04-12 22:51:19,344 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 82cc726f2a1f47a6bd8c7716b2b49729 2015-04-12 22:51:19,345 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x488aa1b1, /172.26.3.231:51813 :> /172.26.3.249:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 144 sec: 2952913 operations; 45273 current ops/sec; [INSERT AverageLatency(us)=3707.62] 2015-04-12 22:51:19,609 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 2b82ac405be945e783d4c493789dc378 2015-04-12 22:51:19,615 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 2b82ac405be945e783d4c493789dc378 2015-04-12 22:51:19,615 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5207814386754341981f5d662f51e2f3 2015-04-12 22:51:19,616 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 794fd028476141bab133ff334c623583 2015-04-12 22:51:19,616 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5207814386754341981f5d662f51e2f3 2015-04-12 22:51:19,616 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 2b82ac405be945e783d4c493789dc378 2015-04-12 22:51:19,616 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 794fd028476141bab133ff334c623583 2015-04-12 22:51:19,618 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x8ab6bd29, /172.26.3.231:43131 :> /172.26.0.237:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 145 sec: 2993893 operations; 40939.06 current ops/sec; [INSERT AverageLatency(us)=631.03] Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:20,681 (New I/O worker #3) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `172.26.3.231' to 172.26.3.231 in 6416151ns 146 sec: 3002389 operations; 5549.31 current ops/sec; [INSERT AverageLatency(us)=139.79] 2015-04-12 22:51:22,471 (New I/O worker #3) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `172.26.3.231' to 172.26.3.231 in 62721191ns Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:22,935 (New I/O worker #3) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-133' to 172.26.0.133 in 8977692ns 147 sec: 3048281 operations; 43458.33 current ops/sec; [INSERT AverageLatency(us)=582.87] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:23,565 (New I/O worker #3) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-33' to 172.26.0.33 in 9091698ns 149 sec: 3065773 operations; 9506.52 current ops/sec; [INSERT AverageLatency(us)=193.99] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:25,560 (New I/O worker #3) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-44' to 172.26.0.44 in 12518279ns 2015-04-12 22:51:25,602 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xc531189a, /172.26.3.231:35519 => /172.26.3.211:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:25,618 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b582c518624547bbaac69b5b860b7606 2015-04-12 22:51:25,665 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x21fea7fe, /172.26.3.231:36081 => /172.26.0.158:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:25,669 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 05ac82e5cdcf4d7494929a7191ca76d2 2015-04-12 22:51:25,753 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x8b2bc18d, /172.26.3.231:54616 => /172.26.0.113:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:25,753 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cf9b1b0aca0744c4a456c74d8df8b31f 2015-04-12 22:51:25,793 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xb6a30a6b, /172.26.3.231:40331 => /172.26.0.159:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:25,795 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1481f0b5b9024d70883392336d607dae 2015-04-12 22:51:25,891 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf500d0ea, /172.26.3.231:56171 => /172.26.0.167:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:25,892 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fc97fd20e21742c2bbab608eae50a048 150 sec: 3098330 operations; 32557 current ops/sec; [INSERT AverageLatency(us)=889.94] Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:26,049 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,049 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,049 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,049 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 22:51:26,049 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,049 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 22:51:26,050 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,053 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 22:51:26,053 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,053 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,053 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 22:51:26,053 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,053 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 22:51:26,055 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:51:26,059 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:51:26,059 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:51:26,063 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 22:51:26,067 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:26,298 (New I/O worker #3) [WARN - kudu.rpc.KuduClient.getIP(KuduClient.java:1604)] Slow DNS lookup! Resolved IP of `ip-172-26-0-160' to 172.26.0.160 in 12024848ns 2015-04-12 22:51:26,629 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xcc8f1c56, /172.26.3.231:42765 => /172.26.0.100:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:26,630 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4f45341b1f3c4092bb73aadb05df5736 2015-04-12 22:51:26,747 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xdcb6a17c, /172.26.3.231:46696 => /172.26.0.80:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:26,748 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 22:51:26,883 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xda5b1309, /172.26.3.231:35859 => /172.26.3.231:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:26,883 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e4fe37d2f1d6457780f2f1d5e11a35ba 2015-04-12 22:51:26,966 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x6f73bef2, /172.26.3.231:45503 => /172.26.0.138:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:26,968 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 643a1fa4b67a40afa1725f00c4e965d8 2015-04-12 22:51:27,028 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,029 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,029 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,029 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,029 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,029 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:27,030 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,030 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:27,030 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,030 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:27,030 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,030 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,031 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,031 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,031 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:27,031 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,031 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,032 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 151 sec: 3119969 operations; 21617.38 current ops/sec; [INSERT AverageLatency(us)=408.1] 2015-04-12 22:51:27,032 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,032 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:27,032 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,032 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,033 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:27,033 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,033 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,033 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,033 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,033 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,034 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,034 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,034 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,038 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,038 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,038 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,038 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e09f57dde36646f6be9f1a093f7f8984 2015-04-12 22:51:27,039 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 50beb22aef7649efb442bd32b9dd698d 2015-04-12 22:51:27,039 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:27,039 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 55537115ebb24a0498f71913f7106518 2015-04-12 22:51:27,039 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3089b9ddffb7460f81168666098577dd 2015-04-12 22:51:27,039 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf2421c44, /172.26.3.231:55697 :> /172.26.0.248:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:27,040 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:27,040 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:27,040 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:27,041 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:27,041 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:27,041 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from be7b707c3d8c408b9ca99766bede14cc 2015-04-12 22:51:27,041 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from be7b707c3d8c408b9ca99766bede14cc 2015-04-12 22:51:27,041 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:27,041 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:27,042 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:27,042 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from be7b707c3d8c408b9ca99766bede14cc 2015-04-12 22:51:27,042 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:27,042 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:27,042 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:27,042 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from be7b707c3d8c408b9ca99766bede14cc 2015-04-12 22:51:27,043 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from be7b707c3d8c408b9ca99766bede14cc 2015-04-12 22:51:27,043 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:27,043 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 22:51:27,043 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 22:51:27,043 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from be7b707c3d8c408b9ca99766bede14cc Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 153 sec: 3134915 operations; 8854.27 current ops/sec; [INSERT AverageLatency(us)=1052.25] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 154 sec: 3169780 operations; 34865 current ops/sec; [INSERT AverageLatency(us)=1275.79] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 155 sec: 3187569 operations; 17389.05 current ops/sec; [INSERT AverageLatency(us)=481.12] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:30,846 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x4ed3396c, /172.26.3.231:36149 => /172.26.0.158:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:51:30,846 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 05ac82e5cdcf4d7494929a7191ca76d2 Waiting more than 10 seconds for an insert or mutation 156 sec: 3191552 operations; 3979.02 current ops/sec; [INSERT AverageLatency(us)=1029.41] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 157 sec: 3192605 operations; 901.54 current ops/sec; [INSERT AverageLatency(us)=2910.98] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 158 sec: 3202237 operations; 9555.56 current ops/sec; [INSERT AverageLatency(us)=2847.01] 159 sec: 3202237 operations; 0 current ops/sec; Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 160 sec: 3220682 operations; 18426.57 current ops/sec; [INSERT AverageLatency(us)=2351.31] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 161 sec: 3268137 operations; 47455 current ops/sec; [INSERT AverageLatency(us)=453.73] 162 sec: 3301132 operations; 27961.86 current ops/sec; [INSERT AverageLatency(us)=221.15] 163 sec: 3331955 operations; 30823 current ops/sec; [INSERT AverageLatency(us)=340.75] Waiting more than 10 seconds for an insert or mutation 165 sec: 3372224 operations; 22272.68 current ops/sec; [INSERT AverageLatency(us)=213.35] 166 sec: 3415022 operations; 42755.24 current ops/sec; [INSERT AverageLatency(us)=374.61] 167 sec: 3451055 operations; 34186.91 current ops/sec; [INSERT AverageLatency(us)=257.52] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 168 sec: 3490215 operations; 39120.88 current ops/sec; [INSERT AverageLatency(us)=1259] Waiting more than 10 seconds for an insert or mutation 170 sec: 3530135 operations; 23075.14 current ops/sec; [INSERT AverageLatency(us)=590.59] 171 sec: 3577091 operations; 46909.09 current ops/sec; [INSERT AverageLatency(us)=519.53] 172 sec: 3629848 operations; 52757 current ops/sec; [INSERT AverageLatency(us)=240.12] Waiting more than 10 seconds for an insert or mutation 174 sec: 3677116 operations; 22444.44 current ops/sec; [INSERT AverageLatency(us)=274.04] Waiting more than 10 seconds for an insert or mutation 175 sec: 3720628 operations; 43468.53 current ops/sec; [INSERT AverageLatency(us)=663.48] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 176 sec: 3753179 operations; 31511.13 current ops/sec; [INSERT AverageLatency(us)=387.66] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 178 sec: 3781490 operations; 13994.56 current ops/sec; [INSERT AverageLatency(us)=545.53] Waiting more than 10 seconds for an insert or mutation 179 sec: 3825719 operations; 44008.96 current ops/sec; [INSERT AverageLatency(us)=605.77] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 181 sec: 3867875 operations; 20179.99 current ops/sec; [INSERT AverageLatency(us)=219.91] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 2015-04-12 22:51:57,284 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:51:57,285 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:51:57,290 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 182 sec: 3916339 operations; 48464 current ops/sec; [INSERT AverageLatency(us)=631.66] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 183 sec: 3954066 operations; 37539.3 current ops/sec; [INSERT AverageLatency(us)=388.45] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 184 sec: 3968672 operations; 9956.37 current ops/sec; [INSERT AverageLatency(us)=280.88] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 185 sec: 4007743 operations; 39071 current ops/sec; [INSERT AverageLatency(us)=519.02] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 186 sec: 4037573 operations; 29800.2 current ops/sec; [INSERT AverageLatency(us)=641.26] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 188 sec: 4063963 operations; 13208.21 current ops/sec; [INSERT AverageLatency(us)=615.79] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 190 sec: 4103533 operations; 34832.75 current ops/sec; [INSERT AverageLatency(us)=688.4] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 191 sec: 4141372 operations; 37801.2 current ops/sec; [INSERT AverageLatency(us)=508.24] Waiting more than 10 seconds for an insert or mutation 192 sec: 4163683 operations; 13546.45 current ops/sec; [INSERT AverageLatency(us)=286.05] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 193 sec: 4209287 operations; 45558.44 current ops/sec; [INSERT AverageLatency(us)=549.71] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 194 sec: 4245770 operations; 36483 current ops/sec; [INSERT AverageLatency(us)=506.34] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 195 sec: 4284346 operations; 38576 current ops/sec; [INSERT AverageLatency(us)=354.33] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 197 sec: 4322628 operations; 18466.96 current ops/sec; [INSERT AverageLatency(us)=382.19] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 198 sec: 4363996 operations; 41368 current ops/sec; [INSERT AverageLatency(us)=535.73] Waiting more than 10 seconds for an insert or mutation 199 sec: 4394170 operations; 30113.77 current ops/sec; [INSERT AverageLatency(us)=332.93] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 200 sec: 4432094 operations; 37924 current ops/sec; [INSERT AverageLatency(us)=418.12] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 202 sec: 4464094 operations; 17947.28 current ops/sec; [INSERT AverageLatency(us)=509.25] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 203 sec: 4514013 operations; 49919 current ops/sec; [INSERT AverageLatency(us)=762.25] 204 sec: 4553131 operations; 39078.92 current ops/sec; [INSERT AverageLatency(us)=344.04] 205 sec: 4592889 operations; 39758 current ops/sec; [INSERT AverageLatency(us)=334.78] 207 sec: 4618016 operations; 14424.23 current ops/sec; [INSERT AverageLatency(us)=304.19] 208 sec: 4663256 operations; 45194.81 current ops/sec; [INSERT AverageLatency(us)=683.84] 209 sec: 4702146 operations; 37215.31 current ops/sec; [INSERT AverageLatency(us)=320.29] 210 sec: 4720962 operations; 11466.18 current ops/sec; [INSERT AverageLatency(us)=310.13] 211 sec: 4766938 operations; 45976 current ops/sec; [INSERT AverageLatency(us)=710.62] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 212 sec: 4809233 operations; 42295 current ops/sec; [INSERT AverageLatency(us)=331.8] 214 sec: 4818428 operations; 6337.01 current ops/sec; [INSERT AverageLatency(us)=235.54] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 215 sec: 4857982 operations; 39554 current ops/sec; [INSERT AverageLatency(us)=471.15] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 217 sec: 4883736 operations; 14181.72 current ops/sec; [INSERT AverageLatency(us)=519.79] 218 sec: 4924176 operations; 40440 current ops/sec; [INSERT AverageLatency(us)=385.49] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 219 sec: 4958235 operations; 34059 current ops/sec; [INSERT AverageLatency(us)=604.56] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 220 sec: 4988256 operations; 28030.81 current ops/sec; [INSERT AverageLatency(us)=472.72] 221 sec: 4998765 operations; 8281.32 current ops/sec; [INSERT AverageLatency(us)=180.04] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 222 sec: 5039768 operations; 41003 current ops/sec; [INSERT AverageLatency(us)=724.58] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 223 sec: 5082857 operations; 43045.95 current ops/sec; [INSERT AverageLatency(us)=383.77] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 224 sec: 5121288 operations; 38431 current ops/sec; [INSERT AverageLatency(us)=423.69] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 226 sec: 5150869 operations; 16845.67 current ops/sec; [INSERT AverageLatency(us)=298.36] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 227 sec: 5198957 operations; 48039.96 current ops/sec; [INSERT AverageLatency(us)=673.33] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 228 sec: 5230126 operations; 29349.34 current ops/sec; [INSERT AverageLatency(us)=530.9] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 229 sec: 5268076 operations; 37574.26 current ops/sec; [INSERT AverageLatency(us)=458.81] 231 sec: 5296673 operations; 17598.15 current ops/sec; [INSERT AverageLatency(us)=294.16] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 232 sec: 5341948 operations; 42874.05 current ops/sec; [INSERT AverageLatency(us)=655.99] 233 sec: 5384679 operations; 42731 current ops/sec; [INSERT AverageLatency(us)=259.77] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 234 sec: 5419766 operations; 35051.95 current ops/sec; [INSERT AverageLatency(us)=290.36] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 235 sec: 5428160 operations; 7047.86 current ops/sec; [INSERT AverageLatency(us)=593.22] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 236 sec: 5470965 operations; 42805 current ops/sec; [INSERT AverageLatency(us)=603.02] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 237 sec: 5513879 operations; 42828.34 current ops/sec; [INSERT AverageLatency(us)=529.46] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 238 sec: 5555449 operations; 39741.87 current ops/sec; [INSERT AverageLatency(us)=397.33] 240 sec: 5595839 operations; 20104.53 current ops/sec; [INSERT AverageLatency(us)=331.33] 241 sec: 5639653 operations; 43770.23 current ops/sec; [INSERT AverageLatency(us)=630.41] 242 sec: 5681123 operations; 41470 current ops/sec; [INSERT AverageLatency(us)=303.08] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 243 sec: 5723643 operations; 42520 current ops/sec; [INSERT AverageLatency(us)=336.07] 244 sec: 5732574 operations; 6079.65 current ops/sec; [INSERT AverageLatency(us)=246.32] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 245 sec: 5774280 operations; 41664.34 current ops/sec; [INSERT AverageLatency(us)=734.29] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 246 sec: 5809469 operations; 35048.8 current ops/sec; [INSERT AverageLatency(us)=270.13] 248 sec: 5842651 operations; 18282.09 current ops/sec; [INSERT AverageLatency(us)=304.27] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 249 sec: 5884412 operations; 41761 current ops/sec; [INSERT AverageLatency(us)=679.45] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 250 sec: 5925480 operations; 41026.97 current ops/sec; [INSERT AverageLatency(us)=542.72] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 251 sec: 5968288 operations; 42808 current ops/sec; [INSERT AverageLatency(us)=320.96] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 252 sec: 6010992 operations; 42661.34 current ops/sec; [INSERT AverageLatency(us)=410.5] 253 sec: 6015379 operations; 3581.22 current ops/sec; [INSERT AverageLatency(us)=208.13] 254 sec: 6059057 operations; 43634.37 current ops/sec; [INSERT AverageLatency(us)=493.09] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 255 sec: 6099745 operations; 40688 current ops/sec; [INSERT AverageLatency(us)=390.42] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 256 sec: 6138033 operations; 38249.75 current ops/sec; [INSERT AverageLatency(us)=387.2] 258 sec: 6149602 operations; 8204.96 current ops/sec; [INSERT AverageLatency(us)=217.6] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 259 sec: 6191961 operations; 42316.68 current ops/sec; [INSERT AverageLatency(us)=668.04] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 260 sec: 6232007 operations; 39966.07 current ops/sec; [INSERT AverageLatency(us)=368.33] 262 sec: 6272195 operations; 20556.52 current ops/sec; [INSERT AverageLatency(us)=237.67] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 263 sec: 6311990 operations; 39755.24 current ops/sec; [INSERT AverageLatency(us)=610.77] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 264 sec: 6350069 operations; 38079 current ops/sec; [INSERT AverageLatency(us)=482.43] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 265 sec: 6390408 operations; 40298.7 current ops/sec; [INSERT AverageLatency(us)=469.39] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 266 sec: 6420847 operations; 17821.43 current ops/sec; [INSERT AverageLatency(us)=443.32] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 267 sec: 6466855 operations; 45962.04 current ops/sec; [INSERT AverageLatency(us)=651.61] Waiting more than 10 seconds for an insert or mutation 268 sec: 6508089 operations; 41234 current ops/sec; [INSERT AverageLatency(us)=331.33] 269 sec: 6548981 operations; 40892 current ops/sec; [INSERT AverageLatency(us)=310.69] 271 sec: 6571312 operations; 13468.64 current ops/sec; [INSERT AverageLatency(us)=246.71] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 272 sec: 6618049 operations; 46737 current ops/sec; [INSERT AverageLatency(us)=672.4] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 273 sec: 6656356 operations; 38268.73 current ops/sec; [INSERT AverageLatency(us)=306.7] 275 sec: 6688599 operations; 17609.5 current ops/sec; [INSERT AverageLatency(us)=201.67] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 276 sec: 6727738 operations; 37815.46 current ops/sec; [INSERT AverageLatency(us)=491.2] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 277 sec: 6769530 operations; 39954.11 current ops/sec; [INSERT AverageLatency(us)=652.04] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 278 sec: 6808925 operations; 39238.05 current ops/sec; [INSERT AverageLatency(us)=341.26] 279 sec: 6826572 operations; 12587.02 current ops/sec; [INSERT AverageLatency(us)=171.77] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 280 sec: 6867760 operations; 41146.85 current ops/sec; [INSERT AverageLatency(us)=901.96] Waiting more than 10 seconds for an insert or mutation 281 sec: 6908631 operations; 40871 current ops/sec; [INSERT AverageLatency(us)=248.52] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 282 sec: 6949559 operations; 40928 current ops/sec; [INSERT AverageLatency(us)=464.04] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 283 sec: 6990365 operations; 40765.23 current ops/sec; [INSERT AverageLatency(us)=320.81] 285 sec: 6999736 operations; 7662.31 current ops/sec; [INSERT AverageLatency(us)=202.95] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 286 sec: 7044952 operations; 45216 current ops/sec; [INSERT AverageLatency(us)=634.95] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 287 sec: 7086365 operations; 41371.63 current ops/sec; [INSERT AverageLatency(us)=318.91] Waiting more than 10 seconds for an insert or mutation 288 sec: 7124269 operations; 37904 current ops/sec; [INSERT AverageLatency(us)=317.96] 289 sec: 7139726 operations; 10458.05 current ops/sec; [INSERT AverageLatency(us)=212.05] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 290 sec: 7179191 operations; 39465 current ops/sec; [INSERT AverageLatency(us)=722.94] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 291 sec: 7215897 operations; 36706 current ops/sec; [INSERT AverageLatency(us)=358.53] 293 sec: 7250243 operations; 18913 current ops/sec; [INSERT AverageLatency(us)=167.88] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 294 sec: 7291354 operations; 41069.93 current ops/sec; [INSERT AverageLatency(us)=697.75] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 295 sec: 7332248 operations; 40894 current ops/sec; [INSERT AverageLatency(us)=497.37] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 296 sec: 7376359 operations; 44111 current ops/sec; [INSERT AverageLatency(us)=455.81] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 297 sec: 7419578 operations; 43175.82 current ops/sec; [INSERT AverageLatency(us)=311.81] 298 sec: 7430270 operations; 7991.03 current ops/sec; [INSERT AverageLatency(us)=234.52] 299 sec: 7472042 operations; 41730.27 current ops/sec; [INSERT AverageLatency(us)=690.87] 300 sec: 7512212 operations; 40170 current ops/sec; [INSERT AverageLatency(us)=308.21] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 301 sec: 7554572 operations; 42360 current ops/sec; [INSERT AverageLatency(us)=344.48] 303 sec: 7587072 operations; 16795.87 current ops/sec; [INSERT AverageLatency(us)=293.3] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 304 sec: 7629139 operations; 42067 current ops/sec; [INSERT AverageLatency(us)=618.35] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 305 sec: 7670779 operations; 38843.28 current ops/sec; [INSERT AverageLatency(us)=274.13] 307 sec: 7690359 operations; 13429.36 current ops/sec; [INSERT AverageLatency(us)=202.71] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 308 sec: 7730245 operations; 37416.51 current ops/sec; [INSERT AverageLatency(us)=709.1] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 309 sec: 7770158 operations; 39873.13 current ops/sec; [INSERT AverageLatency(us)=374.36] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 310 sec: 7810226 operations; 40027.97 current ops/sec; [INSERT AverageLatency(us)=391.15] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 311 sec: 7829516 operations; 14417.04 current ops/sec; [INSERT AverageLatency(us)=572.11] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 312 sec: 7871233 operations; 41675.32 current ops/sec; [INSERT AverageLatency(us)=739.23] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 313 sec: 7911863 operations; 40630 current ops/sec; [INSERT AverageLatency(us)=361.97] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 314 sec: 7952631 operations; 40768 current ops/sec; [INSERT AverageLatency(us)=424.37] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 315 sec: 7994172 operations; 41499.5 current ops/sec; [INSERT AverageLatency(us)=326.66] 317 sec: 8011159 operations; 11377.76 current ops/sec; [INSERT AverageLatency(us)=278.5] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 318 sec: 8052771 operations; 41446.22 current ops/sec; [INSERT AverageLatency(us)=629.81] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 319 sec: 8094419 operations; 41482.07 current ops/sec; [INSERT AverageLatency(us)=348.37] Waiting more than 10 seconds for an insert or mutation 320 sec: 8137532 operations; 39808.86 current ops/sec; [INSERT AverageLatency(us)=287.93] 321 sec: 8156188 operations; 12883.98 current ops/sec; [INSERT AverageLatency(us)=195.21] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 322 sec: 8193664 operations; 36741.18 current ops/sec; [INSERT AverageLatency(us)=527.55] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 323 sec: 8242346 operations; 48633.37 current ops/sec; [INSERT AverageLatency(us)=607.74] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 324 sec: 8282210 operations; 39864 current ops/sec; [INSERT AverageLatency(us)=456.36] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 325 sec: 8321715 operations; 39386.84 current ops/sec; [INSERT AverageLatency(us)=352.04] 326 sec: 8331341 operations; 7731.73 current ops/sec; [INSERT AverageLatency(us)=227.68] 327 sec: 8376360 operations; 44093.05 current ops/sec; [INSERT AverageLatency(us)=552.55] Waiting more than 10 seconds for an insert or mutation 329 sec: 8419741 operations; 40809.97 current ops/sec; [INSERT AverageLatency(us)=336.76] 330 sec: 8462397 operations; 41173.75 current ops/sec; [INSERT AverageLatency(us)=332.94] 331 sec: 8483231 operations; 12438.21 current ops/sec; [INSERT AverageLatency(us)=333.11] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 332 sec: 8527177 operations; 41893.23 current ops/sec; [INSERT AverageLatency(us)=668.2] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 334 sec: 8567425 operations; 18479.34 current ops/sec; [INSERT AverageLatency(us)=280.94] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 335 sec: 8603396 operations; 34957.24 current ops/sec; [INSERT AverageLatency(us)=740.7] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 337 sec: 8641184 operations; 35954.33 current ops/sec; [INSERT AverageLatency(us)=568.77] 338 sec: 8679480 operations; 38105.47 current ops/sec; [INSERT AverageLatency(us)=259.75] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 339 sec: 8698453 operations; 13305.05 current ops/sec; [INSERT AverageLatency(us)=505.63] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 340 sec: 8740787 operations; 42334 current ops/sec; [INSERT AverageLatency(us)=592.6] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 341 sec: 8780249 operations; 39462 current ops/sec; [INSERT AverageLatency(us)=569.92] Waiting more than 10 seconds for an insert or mutation 342 sec: 8820261 operations; 39972.03 current ops/sec; [INSERT AverageLatency(us)=349.02] 344 sec: 8846525 operations; 14922.73 current ops/sec; [INSERT AverageLatency(us)=310.82] 345 sec: 8890680 operations; 44155 current ops/sec; [INSERT AverageLatency(us)=678.29] 346 sec: 8930389 operations; 39669.33 current ops/sec; [INSERT AverageLatency(us)=351.88] Waiting more than 10 seconds for an insert or mutation 347 sec: 8952210 operations; 12342.19 current ops/sec; [INSERT AverageLatency(us)=348.03] 348 sec: 8994133 operations; 41923 current ops/sec; [INSERT AverageLatency(us)=623.31] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 350 sec: 9024568 operations; 16145.89 current ops/sec; [INSERT AverageLatency(us)=383.42] 351 sec: 9066997 operations; 42386.61 current ops/sec; [INSERT AverageLatency(us)=413.94] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 352 sec: 9105258 operations; 38261 current ops/sec; [INSERT AverageLatency(us)=443.97] 354 sec: 9128689 operations; 13567.46 current ops/sec; [INSERT AverageLatency(us)=260.26] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 355 sec: 9165265 operations; 36576 current ops/sec; [INSERT AverageLatency(us)=691.81] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 356 sec: 9188023 operations; 21653.66 current ops/sec; [INSERT AverageLatency(us)=555.4] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 357 sec: 9215869 operations; 24862.5 current ops/sec; [INSERT AverageLatency(us)=619.25] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 359 sec: 9245222 operations; 14297.61 current ops/sec; [INSERT AverageLatency(us)=642.38] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 360 sec: 9281275 operations; 36053 current ops/sec; [INSERT AverageLatency(us)=951.74] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 361 sec: 9318957 operations; 37644.36 current ops/sec; [INSERT AverageLatency(us)=327.98] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 362 sec: 9348220 operations; 29263 current ops/sec; [INSERT AverageLatency(us)=460.07] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 363 sec: 9385422 operations; 37164.84 current ops/sec; [INSERT AverageLatency(us)=544.78] Waiting more than 10 seconds for an insert or mutation 365 sec: 9417682 operations; 17123.14 current ops/sec; [INSERT AverageLatency(us)=268.57] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 366 sec: 9461629 operations; 43903.1 current ops/sec; [INSERT AverageLatency(us)=701.08] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 367 sec: 9503902 operations; 42273 current ops/sec; [INSERT AverageLatency(us)=318.21] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 368 sec: 9546433 operations; 42531 current ops/sec; [INSERT AverageLatency(us)=302.62] Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation 370 sec: 9554632 operations; 5358.82 current ops/sec; [INSERT AverageLatency(us)=315.54] Waiting more than 10 seconds for an insert or mutation 371 sec: 9599663 operations; 44986.01 current ops/sec; [INSERT AverageLatency(us)=791.89] 372 sec: 9643322 operations; 43659 current ops/sec; [INSERT AverageLatency(us)=313.25] Waiting more than 10 seconds for an insert or mutation 373 sec: 9687985 operations; 44663 current ops/sec; [INSERT AverageLatency(us)=324.11] Waiting more than 10 seconds for an insert or mutation 375 sec: 9717745 operations; 16551.72 current ops/sec; [INSERT AverageLatency(us)=308.24] Waiting more than 10 seconds for an insert or mutation 376 sec: 9763435 operations; 45644.36 current ops/sec; [INSERT AverageLatency(us)=666.77] Waiting more than 10 seconds for an insert or mutation 377 sec: 9805182 operations; 41747 current ops/sec; [INSERT AverageLatency(us)=330.66] 378 sec: 9848781 operations; 43599 current ops/sec; [INSERT AverageLatency(us)=316.92] Waiting more than 10 seconds for an insert or mutation 379 sec: 9874562 operations; 15596.49 current ops/sec; [INSERT AverageLatency(us)=295.1] Waiting more than 10 seconds for an insert or mutation 380 sec: 9922435 operations; 47873 current ops/sec; [INSERT AverageLatency(us)=540.6] Waiting more than 10 seconds for an insert or mutation 381 sec: 9966814 operations; 44334.67 current ops/sec; [INSERT AverageLatency(us)=335.75] 382 sec: 10010976 operations; 44162 current ops/sec; [INSERT AverageLatency(us)=291.01] 384 sec: 10023061 operations; 8528.58 current ops/sec; [INSERT AverageLatency(us)=207.92] 385 sec: 10068512 operations; 45451 current ops/sec; [INSERT AverageLatency(us)=545.28] 386 sec: 10109093 operations; 40540.46 current ops/sec; [INSERT AverageLatency(us)=380.1] 388 sec: 10148180 operations; 19436.6 current ops/sec; [INSERT AverageLatency(us)=410.65] 389 sec: 10195417 operations; 47189.81 current ops/sec; [INSERT AverageLatency(us)=618.81] 390 sec: 10238116 operations; 42699 current ops/sec; [INSERT AverageLatency(us)=317.63] 391 sec: 10283212 operations; 45050.95 current ops/sec; [INSERT AverageLatency(us)=296.7] 392 sec: 10290410 operations; 5051.23 current ops/sec; [INSERT AverageLatency(us)=237.58] 393 sec: 10338146 operations; 47736 current ops/sec; [INSERT AverageLatency(us)=725.91] 395 sec: 10367723 operations; 14165.23 current ops/sec; [INSERT AverageLatency(us)=305.51] 396 sec: 10414253 operations; 46530 current ops/sec; [INSERT AverageLatency(us)=743.3] 398 sec: 10439796 operations; 11711.6 current ops/sec; [INSERT AverageLatency(us)=392.22] 400 sec: 10454373 operations; 8287.09 current ops/sec; [INSERT AverageLatency(us)=1760.31] 401 sec: 10471168 operations; 12268.08 current ops/sec; [INSERT AverageLatency(us)=1478.73] 403 sec: 10486599 operations; 9093.11 current ops/sec; [INSERT AverageLatency(us)=1307.92] 405 sec: 10502877 operations; 9442 current ops/sec; [INSERT AverageLatency(us)=1272.33] 407 sec: 10517217 operations; 7823.24 current ops/sec; [INSERT AverageLatency(us)=1350.04] 408 sec: 10531872 operations; 9228.59 current ops/sec; [INSERT AverageLatency(us)=1547.98] 409 sec: 10578888 operations; 42780.71 current ops/sec; [INSERT AverageLatency(us)=674.2] 410 sec: 10616153 operations; 37265 current ops/sec; [INSERT AverageLatency(us)=400.41] 412 sec: 10617024 operations; 653.9 current ops/sec; [INSERT AverageLatency(us)=199.57] 413 sec: 10660563 operations; 39797.99 current ops/sec; [INSERT AverageLatency(us)=692.62] 414 sec: 10691854 operations; 27091.77 current ops/sec; [INSERT AverageLatency(us)=480.88] 415 sec: 10720604 operations; 27459.41 current ops/sec; [INSERT AverageLatency(us)=517.52] 416 sec: 10730450 operations; 6861.32 current ops/sec; [INSERT AverageLatency(us)=421.18] 417 sec: 10772440 operations; 41781.09 current ops/sec; [INSERT AverageLatency(us)=618.44] 418 sec: 10814975 operations; 42535 current ops/sec; [INSERT AverageLatency(us)=293.97] 420 sec: 10858311 operations; 38486.68 current ops/sec; [INSERT AverageLatency(us)=282.22] 421 sec: 10879179 operations; 11776.52 current ops/sec; [INSERT AverageLatency(us)=417.08] 422 sec: 10921741 operations; 42477.05 current ops/sec; [INSERT AverageLatency(us)=752.72] 423 sec: 10961433 operations; 37304.51 current ops/sec; [INSERT AverageLatency(us)=328.59] 424 sec: 10992015 operations; 29405.77 current ops/sec; [INSERT AverageLatency(us)=444.36] 425 sec: 11031459 operations; 39404.6 current ops/sec; [INSERT AverageLatency(us)=373.64] 428 sec: 11063341 operations; 13624.79 current ops/sec; [INSERT AverageLatency(us)=443.4] 429 sec: 11105602 operations; 42008.95 current ops/sec; [INSERT AverageLatency(us)=679.55] 430 sec: 11145796 operations; 40153.85 current ops/sec; [INSERT AverageLatency(us)=347.56] 432 sec: 11160364 operations; 8138.55 current ops/sec; [INSERT AverageLatency(us)=371.46] 433 sec: 11202422 operations; 41112.41 current ops/sec; [INSERT AverageLatency(us)=769] 434 sec: 11222890 operations; 13128.93 current ops/sec; [INSERT AverageLatency(us)=313.38] 435 sec: 11265048 operations; 39547.84 current ops/sec; [INSERT AverageLatency(us)=626.69] 436 sec: 11298310 operations; 29435.4 current ops/sec; [INSERT AverageLatency(us)=397.17] 438 sec: 11327289 operations; 14489.5 current ops/sec; [INSERT AverageLatency(us)=485.62] 439 sec: 11370313 operations; 42981.02 current ops/sec; [INSERT AverageLatency(us)=676.49] 440 sec: 11409184 operations; 38871 current ops/sec; [INSERT AverageLatency(us)=335.15] 441 sec: 11444065 operations; 34881 current ops/sec; [INSERT AverageLatency(us)=339.18] 443 sec: 11459896 operations; 9005.12 current ops/sec; [INSERT AverageLatency(us)=398.16] 444 sec: 11505231 operations; 44014.56 current ops/sec; [INSERT AverageLatency(us)=558.03] 445 sec: 11546168 operations; 40896.1 current ops/sec; [INSERT AverageLatency(us)=320.19] 447 sec: 11560433 operations; 8386.24 current ops/sec; [INSERT AverageLatency(us)=432.82] 448 sec: 11603423 operations; 42990 current ops/sec; [INSERT AverageLatency(us)=631.09] 449 sec: 11629782 operations; 26332.67 current ops/sec; [INSERT AverageLatency(us)=322.18] Waiting more than 10 seconds for an insert or mutation 450 sec: 11645823 operations; 10210.69 current ops/sec; [INSERT AverageLatency(us)=231.03] 451 sec: 11687470 operations; 41605.39 current ops/sec; [INSERT AverageLatency(us)=962.92] 452 sec: 11731456 operations; 42294.23 current ops/sec; [INSERT AverageLatency(us)=441.04] 453 sec: 11777630 operations; 46127.87 current ops/sec; [INSERT AverageLatency(us)=308.41] 455 sec: 11801001 operations; 13870.03 current ops/sec; [INSERT AverageLatency(us)=274.21] 456 sec: 11845532 operations; 41347.26 current ops/sec; [INSERT AverageLatency(us)=664.31] 457 sec: 11883921 operations; 38350.65 current ops/sec; [INSERT AverageLatency(us)=380.7] 458 sec: 11920858 operations; 36900.1 current ops/sec; [INSERT AverageLatency(us)=377.04] 460 sec: 11935472 operations; 8561.22 current ops/sec; [INSERT AverageLatency(us)=331.25] 461 sec: 11978299 operations; 42784.22 current ops/sec; [INSERT AverageLatency(us)=696.47] 463 sec: 12015979 operations; 15348.27 current ops/sec; [INSERT AverageLatency(us)=321.89] 465 sec: 12039424 operations; 14418.82 current ops/sec; [INSERT AverageLatency(us)=1144.3] 466 sec: 12082346 operations; 42922 current ops/sec; [INSERT AverageLatency(us)=643.81] 467 sec: 12112140 operations; 28107.55 current ops/sec; [INSERT AverageLatency(us)=379.23] 469 sec: 12131531 operations; 11067.92 current ops/sec; [INSERT AverageLatency(us)=493.44] 470 sec: 12176148 operations; 44617 current ops/sec; [INSERT AverageLatency(us)=730.15] 471 sec: 12214968 operations; 35977.76 current ops/sec; [INSERT AverageLatency(us)=342.98] 472 sec: 12245655 operations; 29563.58 current ops/sec; [INSERT AverageLatency(us)=369.04] 473 sec: 12255981 operations; 6977.03 current ops/sec; [INSERT AverageLatency(us)=467.75] 474 sec: 12299270 operations; 43245.75 current ops/sec; [INSERT AverageLatency(us)=669.56] 475 sec: 12329939 operations; 29292.26 current ops/sec; [INSERT AverageLatency(us)=374.09] 477 sec: 12353248 operations; 12039.77 current ops/sec; [INSERT AverageLatency(us)=539.92] 478 sec: 12398817 operations; 45523.48 current ops/sec; [INSERT AverageLatency(us)=616.72] 479 sec: 12433773 operations; 34782.09 current ops/sec; [INSERT AverageLatency(us)=349.29] 480 sec: 12466812 operations; 33005.99 current ops/sec; [INSERT AverageLatency(us)=451.35] 482 sec: 12470623 operations; 2356.83 current ops/sec; [INSERT AverageLatency(us)=235.23] 483 sec: 12507522 operations; 35754.84 current ops/sec; [INSERT AverageLatency(us)=927.83] 484 sec: 12547123 operations; 33818.1 current ops/sec; [INSERT AverageLatency(us)=384.37] 487 sec: 12577477 operations; 12200.16 current ops/sec; [INSERT AverageLatency(us)=394.03] 488 sec: 12619001 operations; 41482.52 current ops/sec; [INSERT AverageLatency(us)=979.75] 489 sec: 12645929 operations; 26928 current ops/sec; [INSERT AverageLatency(us)=373.3] 491 sec: 12667973 operations; 12052.49 current ops/sec; [INSERT AverageLatency(us)=367.28] 492 sec: 12711095 operations; 43122 current ops/sec; [INSERT AverageLatency(us)=764.22] 493 sec: 12739169 operations; 28074 current ops/sec; [INSERT AverageLatency(us)=485.05] 495 sec: 12773394 operations; 15837.58 current ops/sec; [INSERT AverageLatency(us)=342.2] 496 sec: 12821299 operations; 42772.32 current ops/sec; [INSERT AverageLatency(us)=586.62] 497 sec: 12860284 operations; 38946.05 current ops/sec; [INSERT AverageLatency(us)=352.59] 498 sec: 12863569 operations; 2438.75 current ops/sec; [INSERT AverageLatency(us)=265.36] 499 sec: 12906294 operations; 42682.32 current ops/sec; [INSERT AverageLatency(us)=667.61] 500 sec: 12946110 operations; 39816 current ops/sec; [INSERT AverageLatency(us)=347.04] 501 sec: 12946148 operations; 31.85 current ops/sec; [INSERT AverageLatency(us)=338.21] 502 sec: 12991525 operations; 45377 current ops/sec; [INSERT AverageLatency(us)=617.14] 503 sec: 13027982 operations; 36457 current ops/sec; [INSERT AverageLatency(us)=342.41] 504 sec: 13063383 operations; 35365.63 current ops/sec; [INSERT AverageLatency(us)=345.16] 505 sec: 13105409 operations; 42026 current ops/sec; [INSERT AverageLatency(us)=295.53] 507 sec: 13109981 operations; 3279.77 current ops/sec; [INSERT AverageLatency(us)=219.98] 508 sec: 13157167 operations; 42548.24 current ops/sec; [INSERT AverageLatency(us)=786.72] 509 sec: 13202256 operations; 45089 current ops/sec; [INSERT AverageLatency(us)=252.29] 510 sec: 13208766 operations; 4909.5 current ops/sec; [INSERT AverageLatency(us)=169.63] 511 sec: 13256987 operations; 48172.83 current ops/sec; [INSERT AverageLatency(us)=701.82] 512 sec: 13289704 operations; 30662.61 current ops/sec; [INSERT AverageLatency(us)=382.36] 513 sec: 13333173 operations; 43469 current ops/sec; [INSERT AverageLatency(us)=371.14] 515 sec: 13333850 operations; 501.85 current ops/sec; [INSERT AverageLatency(us)=512.44] 516 sec: 13382416 operations; 44637.87 current ops/sec; [INSERT AverageLatency(us)=494.83] 517 sec: 13428638 operations; 46222 current ops/sec; [INSERT AverageLatency(us)=332.92] 518 sec: 13430068 operations; 974.11 current ops/sec; [INSERT AverageLatency(us)=3852.46] 519 sec: 13475768 operations; 43732.06 current ops/sec; [INSERT AverageLatency(us)=296.32] 522 sec: 13512327 operations; 14729.65 current ops/sec; [INSERT AverageLatency(us)=370] 523 sec: 13557638 operations; 42426.03 current ops/sec; [INSERT AverageLatency(us)=665.19] 524 sec: 13592825 operations; 35151.85 current ops/sec; [INSERT AverageLatency(us)=423.47] 526 sec: 13596485 operations; 2077.19 current ops/sec; [INSERT AverageLatency(us)=240.51] 527 sec: 13627048 operations; 17122.13 current ops/sec; [INSERT AverageLatency(us)=954.91] 528 sec: 13670652 operations; 43560.44 current ops/sec; [INSERT AverageLatency(us)=657.43] 529 sec: 13711397 operations; 36089.46 current ops/sec; [INSERT AverageLatency(us)=344.89] 530 sec: 13744415 operations; 33018 current ops/sec; [INSERT AverageLatency(us)=445.27] 532 sec: 13752558 operations; 6556.36 current ops/sec; [INSERT AverageLatency(us)=214.08] 533 sec: 13798392 operations; 41069.89 current ops/sec; [INSERT AverageLatency(us)=600.96] 534 sec: 13830447 operations; 30042.17 current ops/sec; [INSERT AverageLatency(us)=471.79] 535 sec: 13870009 operations; 39562 current ops/sec; [INSERT AverageLatency(us)=391.75] 537 sec: 13904754 operations; 17663.96 current ops/sec; [INSERT AverageLatency(us)=373] 538 sec: 13948470 operations; 43716 current ops/sec; [INSERT AverageLatency(us)=697.6] 539 sec: 13976808 operations; 28225.1 current ops/sec; [INSERT AverageLatency(us)=498.43] 540 sec: 14012610 operations; 35766.23 current ops/sec; [INSERT AverageLatency(us)=389.81] 541 sec: 14015198 operations; 2176.62 current ops/sec; [INSERT AverageLatency(us)=228.91] 542 sec: 14058626 operations; 43428 current ops/sec; [INSERT AverageLatency(us)=590.91] 543 sec: 14104137 operations; 45465.53 current ops/sec; [INSERT AverageLatency(us)=310.75] 545 sec: 14119485 operations; 9800.77 current ops/sec; [INSERT AverageLatency(us)=323.55] 546 sec: 14164743 operations; 42980.06 current ops/sec; [INSERT AverageLatency(us)=679.62] 547 sec: 14203054 operations; 38311 current ops/sec; [INSERT AverageLatency(us)=420.3] 548 sec: 14220910 operations; 10429.91 current ops/sec; [INSERT AverageLatency(us)=382] 549 sec: 14263955 operations; 41710.27 current ops/sec; [INSERT AverageLatency(us)=671.65] 551 sec: 14300033 operations; 31814.81 current ops/sec; [INSERT AverageLatency(us)=440.42] 552 sec: 14322656 operations; 12208.85 current ops/sec; [INSERT AverageLatency(us)=400.83] 553 sec: 14365774 operations; 43074.93 current ops/sec; [INSERT AverageLatency(us)=784.96] 555 sec: 14371902 operations; 3716.19 current ops/sec; [INSERT AverageLatency(us)=210.31] 557 sec: 14385264 operations; 7768.6 current ops/sec; [INSERT AverageLatency(us)=1974.28] 559 sec: 14397739 operations; 7186.06 current ops/sec; [INSERT AverageLatency(us)=1906.46] 560 sec: 14410713 operations; 9821.35 current ops/sec; [INSERT AverageLatency(us)=1739.3] 561 sec: 14423249 operations; 9348.25 current ops/sec; [INSERT AverageLatency(us)=1389.74] 563 sec: 14436012 operations; 7758.66 current ops/sec; [INSERT AverageLatency(us)=1148.63] 565 sec: 14448036 operations; 6863.01 current ops/sec; [INSERT AverageLatency(us)=1366.69] 566 sec: 14460075 operations; 8819.78 current ops/sec; [INSERT AverageLatency(us)=1878.52] 2015-04-12 22:58:22,103 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:58:22,104 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b843fd4f2d149c596618c207f16db36 2015-04-12 22:58:22,104 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x4fa7d7f5, /172.26.3.231:44302 :> /172.26.0.99:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 567 sec: 14472047 operations; 8881.31 current ops/sec; [INSERT AverageLatency(us)=1474.88] 569 sec: 14483050 operations; 8162.46 current ops/sec; [INSERT AverageLatency(us)=1236.33] Waiting more than 10 seconds for an insert or mutation 570 sec: 14491163 operations; 4524.82 current ops/sec; [INSERT AverageLatency(us)=3367.77] 572 sec: 14501064 operations; 7253.48 current ops/sec; [INSERT AverageLatency(us)=2689.98] 573 sec: 14508822 operations; 5601.44 current ops/sec; [INSERT AverageLatency(us)=2416.45] 575 sec: 14514819 operations; 4468.7 current ops/sec; [INSERT AverageLatency(us)=3154.71] 576 sec: 14520669 operations; 3257.24 current ops/sec; [INSERT AverageLatency(us)=2977.73] 578 sec: 14527476 operations; 4943.36 current ops/sec; [INSERT AverageLatency(us)=3774.5] 579 sec: 14533584 operations; 4541.26 current ops/sec; [INSERT AverageLatency(us)=3007.3] 580 sec: 14538438 operations; 3685.65 current ops/sec; [INSERT AverageLatency(us)=4417.48] 582 sec: 14542018 operations; 2010.11 current ops/sec; [INSERT AverageLatency(us)=4684.89] 2015-04-12 22:58:38,229 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c335263a849d4463ab09828f901eced2 583 sec: 14547570 operations; 4121.75 current ops/sec; [INSERT AverageLatency(us)=4887.98] 585 sec: 14551492 operations; 2996.18 current ops/sec; [INSERT AverageLatency(us)=4013.07] 2015-04-12 22:58:40,871 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 6cfd594c7fac420090441d55bc7174c7 2015-04-12 22:58:40,871 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c4caf51320c1446faf0efe3f5263353f 2015-04-12 22:58:40,885 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 75bdc35934e34102ac5d7e86091e9ce1 586 sec: 14554654 operations; 2325 current ops/sec; [INSERT AverageLatency(us)=7413.38] 588 sec: 14557295 operations; 1449.51 current ops/sec; [INSERT AverageLatency(us)=3123.47] 2015-04-12 22:58:44,035 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cc79343ea6ca409d99587e843d24054d 2015-04-12 22:58:44,036 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cc79343ea6ca409d99587e843d24054d 2015-04-12 22:58:44,036 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cc79343ea6ca409d99587e843d24054d 2015-04-12 22:58:44,036 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cc79343ea6ca409d99587e843d24054d 2015-04-12 22:58:44,036 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 063b13cae2cc465288520e3d5179c718 2015-04-12 22:58:44,037 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x3d1fb113, /172.26.3.231:55077 :> /172.26.3.230:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:44,081 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xc0d5ebaf, /172.26.3.231:54295 => /172.26.3.231:7051] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 589 sec: 14561150 operations; 2853.44 current ops/sec; [INSERT AverageLatency(us)=6647.14] 2015-04-12 22:58:45,402 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc0d5ebaf, /172.26.3.231:54295 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:45,409 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:58:45,410 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc0d5ebaf, /172.26.3.231:54295 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:45,410 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:58:45,411 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc0d5ebaf, /172.26.3.231:54295 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:45,411 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 591 sec: 14563759 operations; 2005.38 current ops/sec; [INSERT AverageLatency(us)=8136.71] 2015-04-12 22:58:45,411 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc0d5ebaf, /172.26.3.231:54295 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:46,699 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:58:46,699 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc0d5ebaf, /172.26.3.231:54295 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:46,700 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 22:58:46,700 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc0d5ebaf, /172.26.3.231:54295 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:46,701 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 592 sec: 14566064 operations; 1756.86 current ops/sec; [INSERT AverageLatency(us)=5620.46] 2015-04-12 22:58:46,701 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xe926a8b2, /172.26.3.231:49068 => /172.26.1.51:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:47,997 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 568e7aa39c9d416f95113ae674b2607d 2015-04-12 22:58:47,997 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 568e7aa39c9d416f95113ae674b2607d 2015-04-12 22:58:47,997 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 568e7aa39c9d416f95113ae674b2607d 2015-04-12 22:58:47,997 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 568e7aa39c9d416f95113ae674b2607d 2015-04-12 22:58:47,997 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 568e7aa39c9d416f95113ae674b2607d 2015-04-12 22:58:47,998 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 568e7aa39c9d416f95113ae674b2607d 2015-04-12 22:58:47,998 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5207814386754341981f5d662f51e2f3 2015-04-12 22:58:47,998 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 794fd028476141bab133ff334c623583 2015-04-12 22:58:47,998 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 794fd028476141bab133ff334c623583 2015-04-12 22:58:47,999 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5207814386754341981f5d662f51e2f3 2015-04-12 22:58:47,999 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5207814386754341981f5d662f51e2f3 2015-04-12 22:58:47,999 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5207814386754341981f5d662f51e2f3 594 sec: 14568343 operations; 1256.34 current ops/sec; [INSERT AverageLatency(us)=6908.59] 2015-04-12 22:58:47,999 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x820f2f6a, /172.26.3.231:43535 :> /172.26.0.237:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:49,838 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f836a48e19fe49f99776c7b5fe5e5c12 595 sec: 14571315 operations; 2273.91 current ops/sec; [INSERT AverageLatency(us)=7381.2] 597 sec: 14573021 operations; 1134.31 current ops/sec; [INSERT AverageLatency(us)=9856.72] 598 sec: 14574153 operations; 771.64 current ops/sec; [INSERT AverageLatency(us)=11921.57] 600 sec: 14575105 operations; 498.43 current ops/sec; [INSERT AverageLatency(us)=15093.99] 2015-04-12 22:58:56,002 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf6b57593, /172.26.3.231:35906 => /172.26.3.231:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:56,002 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e4fe37d2f1d6457780f2f1d5e11a35ba 2015-04-12 22:58:56,003 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e4fe37d2f1d6457780f2f1d5e11a35ba 2015-04-12 22:58:56,004 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a33230b6b6e7450484f9bcb05f6724c9 2015-04-12 22:58:56,020 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xa998450b, /172.26.3.231:48056 => /172.26.0.49:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 601 sec: 14577087 operations; 1406.67 current ops/sec; [INSERT AverageLatency(us)=20492.06] 2015-04-12 22:58:57,399 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f137ac6a562e45c582883cbc20479f99 2015-04-12 22:58:57,400 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 22c5b8d0fd944b95bf656134e59c6561 2015-04-12 22:58:57,400 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 22c5b8d0fd944b95bf656134e59c6561 2015-04-12 22:58:57,400 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f137ac6a562e45c582883cbc20479f99 2015-04-12 22:58:57,400 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f137ac6a562e45c582883cbc20479f99 2015-04-12 22:58:57,400 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xa998450b, /172.26.3.231:48056 :> /172.26.0.49:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:57,401 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xa998450b, /172.26.3.231:48056 :> /172.26.0.49:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:57,401 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xa998450b, /172.26.3.231:48056 :> /172.26.0.49:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:57,406 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x38f0414a, /172.26.3.231:39267 => /172.26.1.35:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:58:57,411 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1b4c1836283542a5aad5da8e27f160b8 2015-04-12 22:58:57,412 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d2b34753240441f9b88483645427781c 2015-04-12 22:58:57,412 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e6f33f2dbad147f185f9ba6551e9bdbd 2015-04-12 22:58:57,412 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f843f12e91b74ab39e2587badd5ff13c 2015-04-12 22:58:57,412 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e6f33f2dbad147f185f9ba6551e9bdbd 603 sec: 14577717 operations; 440.25 current ops/sec; [INSERT AverageLatency(us)=14194] 604 sec: 14578565 operations; 548.51 current ops/sec; [INSERT AverageLatency(us)=16675.83] 606 sec: 14579336 operations; 365.23 current ops/sec; [INSERT AverageLatency(us)=21002.5] 608 sec: 14580782 operations; 895.36 current ops/sec; [INSERT AverageLatency(us)=21970.63] 609 sec: 14581069 operations; 208.12 current ops/sec; [INSERT AverageLatency(us)=26570.62] 611 sec: 14581530 operations; 334.79 current ops/sec; [INSERT AverageLatency(us)=23794.27] 613 sec: 14581669 operations; 72.32 current ops/sec; [INSERT AverageLatency(us)=68043.7] 2015-04-12 22:58:57,413 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c0ea31aaabc944bba27b4fe6a3a31eae 2015-04-12 22:59:08,784 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f843f12e91b74ab39e2587badd5ff13c 2015-04-12 22:59:08,784 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x38f0414a, /172.26.3.231:39267 :> /172.26.1.35:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:08,784 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x38f0414a, /172.26.3.231:39267 :> /172.26.1.35:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:08,793 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5c20b2b2cbfd41149d11068755ce6d7d 2015-04-12 22:59:08,794 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5f48d1bdbfa84880ba72d0ca71ecc6e4 2015-04-12 22:59:08,797 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 05ac82e5cdcf4d7494929a7191ca76d2 2015-04-12 22:59:08,797 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 79d622bf67e5444abb8d13e0ae463cc6 2015-04-12 22:59:08,797 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b1ba53af715c493aa175e83d5e674bfd 2015-04-12 22:59:08,798 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 6486d35136cc4bb4a279e71bb8ba4978 2015-04-12 22:59:08,798 (New I/O worker #3) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 8c4058addb27446a9a79719ab1a87412 614 sec: 14582830 operations; 820.49 current ops/sec; [INSERT AverageLatency(us)=33562.79] 615 sec: 14582969 operations; 108.26 current ops/sec; [INSERT AverageLatency(us)=10068.1] 617 sec: 14583175 operations; 159.07 current ops/sec; [INSERT AverageLatency(us)=49824.71] 618 sec: 14583638 operations; 260.11 current ops/sec; [INSERT AverageLatency(us)=54123.63] 2015-04-12 22:59:25,597 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x52abaee8, /172.26.3.231:48168 => /172.26.3.242:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:25,597 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 346cd14c69094045a7a853d9f974800d 2015-04-12 22:59:25,598 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 346cd14c69094045a7a853d9f974800d 2015-04-12 22:59:25,598 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 99c4e9b15f9a4642932a666528706d1d 2015-04-12 22:59:25,598 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 346cd14c69094045a7a853d9f974800d 2015-04-12 22:59:25,598 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 99c4e9b15f9a4642932a666528706d1d 2015-04-12 22:59:25,598 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d55b885b06174162b488c3499e8fe995 2015-04-12 22:59:25,598 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 99c4e9b15f9a4642932a666528706d1d 2015-04-12 22:59:25,599 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d55b885b06174162b488c3499e8fe995 2015-04-12 22:59:25,599 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d55b885b06174162b488c3499e8fe995 2015-04-12 22:59:25,599 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 99c4e9b15f9a4642932a666528706d1d 2015-04-12 22:59:25,599 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x52abaee8, /172.26.3.231:48168 :> /172.26.3.242:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:36,670 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x52abaee8, /172.26.3.231:48168 :> /172.26.3.242:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:36,671 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x52abaee8, /172.26.3.231:48168 :> /172.26.3.242:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:36,671 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x52abaee8, /172.26.3.231:48168 :> /172.26.3.242:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:36,672 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x52abaee8, /172.26.3.231:48168 :> /172.26.3.242:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:31,129 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 778d3039dc464916af771fc0b2c9787a 2015-04-12 22:59:36,672 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e39aa4e04b2c411fa14e66efd4924587 2015-04-12 22:59:36,673 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xc3d8578d, /172.26.3.231:52410 :> /172.26.3.196:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 22:59:36,674 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x0d57122f, /172.26.3.231:54296 => /172.26.3.231:7051] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to java_pid21852.hprof ... Heap dump file created [1763491720 bytes in 23.410 secs] Exception in thread "Thread-14" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.StringCoding.encode(StringCoding.java:277) at java.lang.String.getBytes(String.java:969) at kudu.rpc.Bytes.UTF8(Bytes.java:566) at kudu.rpc.Bytes.fromString(Bytes.java:914) at kudu.rpc.Operation.createAndFillWriteRequestPB(Operation.java:394) at kudu.rpc.Batch.serialize(Batch.java:32) at kudu.rpc.TabletClient.encode(TabletClient.java:203) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:146) at kudu.rpc.KuduClient.sendRpcToTablet(KuduClient.java:605) at kudu.rpc.KuduSession.flushTablet(KuduSession.java:535) at kudu.rpc.KuduSession.addToBuffer(KuduSession.java:389) at kudu.rpc.KuduSession.apply(KuduSession.java:293) at com.yahoo.ycsb.db.KuduYCSBClient.applyOperation(KuduYCSBClient.java:330) at com.yahoo.ycsb.db.KuduYCSBClient.insert(KuduYCSBClient.java:320) at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148) at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461) at com.yahoo.ycsb.ClientThread.run(Client.java:269) 682 sec: 14588006 operations; 68.48 current ops/sec; [INSERT AverageLatency(us)=85772.44] Exception in thread "Thread-16" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.StringCoding.encode(StringCoding.java:277) at java.lang.String.getBytes(String.java:969) at kudu.rpc.Bytes.UTF8(Bytes.java:566) at kudu.rpc.Bytes.fromString(Bytes.java:914) at kudu.rpc.Operation.createAndFillWriteRequestPB(Operation.java:394) at kudu.rpc.Batch.serialize(Batch.java:32) at kudu.rpc.TabletClient.encode(TabletClient.java:203) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:146) at kudu.rpc.KuduClient.sendRpcToTablet(KuduClient.java:605) at kudu.rpc.KuduSession.flushTablet(KuduSession.java:535) at kudu.rpc.KuduSession.addToBuffer(KuduSession.java:389) at kudu.rpc.KuduSession.apply(KuduSession.java:293) at com.yahoo.ycsb.db.KuduYCSBClient.applyOperation(KuduYCSBClient.java:330) at com.yahoo.ycsb.db.KuduYCSBClient.insert(KuduYCSBClient.java:320) at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148) at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461) at com.yahoo.ycsb.ClientThread.run(Client.java:269) Exception in thread "Thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232) at java.lang.StringCoding.encode(StringCoding.java:278) at java.lang.String.getBytes(String.java:969) at kudu.rpc.Bytes.UTF8(Bytes.java:566) at kudu.rpc.Bytes.fromString(Bytes.java:914) at kudu.rpc.Operation.createAndFillWriteRequestPB(Operation.java:394) at kudu.rpc.Batch.serialize(Batch.java:32) at kudu.rpc.TabletClient.encode(TabletClient.java:203) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:146) at kudu.rpc.KuduClient.sendRpcToTablet(KuduClient.java:605) at kudu.rpc.KuduSession.flushTablet(KuduSession.java:535) at kudu.rpc.KuduSession.addToBuffer(KuduSession.java:389) at kudu.rpc.KuduSession.apply(KuduSession.java:293) at com.yahoo.ycsb.db.KuduYCSBClient.applyOperation(KuduYCSBClient.java:330) at com.yahoo.ycsb.db.KuduYCSBClient.insert(KuduYCSBClient.java:320) at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148) at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461) at com.yahoo.ycsb.ClientThread.run(Client.java:269) Exception in thread "Thread-4" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.nio.CharBuffer.wrap(CharBuffer.java:350) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:238) at java.lang.StringCoding.encode(StringCoding.java:278) at java.lang.String.getBytes(String.java:969) at kudu.rpc.Bytes.UTF8(Bytes.java:566) at kudu.rpc.Bytes.fromString(Bytes.java:914) at kudu.rpc.Operation.createAndFillWriteRequestPB(Operation.java:394) at kudu.rpc.Batch.serialize(Batch.java:32) at kudu.rpc.TabletClient.encode(TabletClient.java:203) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:146) at kudu.rpc.KuduClient.sendRpcToTablet(KuduClient.java:605) at kudu.rpc.KuduSession.flushTablet(KuduSession.java:535) at kudu.rpc.KuduSession.addToBuffer(KuduSession.java:389) at kudu.rpc.KuduSession.apply(KuduSession.java:293) at com.yahoo.ycsb.db.KuduYCSBClient.applyOperation(KuduYCSBClient.java:330) at com.yahoo.ycsb.db.KuduYCSBClient.insert(KuduYCSBClient.java:320) at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148) at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461) at com.yahoo.ycsb.ClientThread.run(Client.java:269) 2015-04-12 23:00:38,984 (New I/O worker #3) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x3c85fe43, /172.26.3.231:54297 => /172.26.3.231:7051] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:00:38,985 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x3c85fe43, /172.26.3.231:54297 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:00:38,985 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:00:38,986 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x3c85fe43, /172.26.3.231:54297 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:00:38,986 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). Exception in thread "Thread-10" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "Thread-13" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "Thread-12" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "Thread-11" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "Thread-15" java.lang.OutOfMemoryError: GC overhead limit exceeded 743 sec: 14588713 operations; 11.55 current ops/sec; [INSERT AverageLatency(us)=1216677.76] 747 sec: 14588740 operations; 8.89 current ops/sec; [INSERT AverageLatency(us)=18.86] Exception in thread "Thread-17" java.lang.OutOfMemoryError: GC overhead limit exceeded 2015-04-12 23:00:38,986 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x3c85fe43, /172.26.3.231:54297 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:28,114 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:28,114 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x3c85fe43, /172.26.3.231:54297 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:28,115 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:28,115 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x3c85fe43, /172.26.3.231:54297 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:28,116 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:28,116 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x3c85fe43, /172.26.3.231:54297 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Exception in thread "Thread-9" java.lang.OutOfMemoryError: GC overhead limit exceeded 2015-04-12 23:01:47,503 (New I/O worker #3) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:13,955 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e59a4f9ef47d4e209c98fd73e66073df 2015-04-12 23:01:47,504 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xeb8feb48, /172.26.3.231:43013 :> /172.26.0.111:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:47,506 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b7aecd5368774804ab1e16285fca8434 2015-04-12 23:01:47,506 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c4caf51320c1446faf0efe3f5263353f 2015-04-12 23:01:47,506 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x65e99bf0, /172.26.3.231:59373 :> /172.26.3.252:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Exception in thread "Thread-6" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "Thread-7" java.lang.OutOfMemoryError: GC overhead limit exceeded 2015-04-12 23:00:38,987 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x0d57122f, /172.26.3.231:54296 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:56,671 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:56,671 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x0d57122f, /172.26.3.231:54296 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:59,731 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:59,731 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x0d57122f, /172.26.3.231:54296 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:59,732 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:59,732 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x0d57122f, /172.26.3.231:54296 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:59,732 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:59,733 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x0d57122f, /172.26.3.231:54296 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:59,733 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:59,733 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x0d57122f, /172.26.3.231:54296 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:59,734 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:59,734 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0x0d57122f, /172.26.3.231:54296 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:01:59,735 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:01:59,735 (New I/O worker #2) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 2015-04-12 23:02:02,835 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x0d57122f, /172.26.3.231:54296 :> /172.26.3.231:7051] java.lang.OutOfMemoryError: GC overhead limit exceeded 2015-04-12 23:02:02,837 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xed1d5256, /172.26.3.231:57123 => /172.26.0.57:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:05,898 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 68a6dd22e5634e97b27e4656a212cb18 2015-04-12 23:02:05,898 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 484a73f28984423c970900be6b6d6bab 2015-04-12 23:02:05,898 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 41207f4c0685405d964530dbdd8a6065 2015-04-12 23:02:05,898 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fac7c03d320440679149f7c3419174de 2015-04-12 23:02:05,899 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xed1d5256, /172.26.3.231:57123 :> /172.26.0.57:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:05,899 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x705188cf, /172.26.3.231:56963 => /172.26.0.28:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:05,900 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 941a3f6fd55144c88da7462b1fb999d4 2015-04-12 23:02:05,900 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 23:02:05,900 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fdc1d51f50a24eb0be3e00eaa3c31de3 2015-04-12 23:02:05,900 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from fdc1d51f50a24eb0be3e00eaa3c31de3 2015-04-12 23:02:08,929 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 941a3f6fd55144c88da7462b1fb999d4 2015-04-12 23:02:08,929 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 364f71ae824140fca856dee2e90429bd 2015-04-12 23:02:08,930 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x705188cf, /172.26.3.231:56963 :> /172.26.0.28:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:08,930 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x705188cf, /172.26.3.231:56963 :> /172.26.0.28:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:08,930 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x705188cf, /172.26.3.231:56963 :> /172.26.0.28:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:08,931 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xfef48e6f, /172.26.3.231:35295 => /172.26.3.190:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:08,931 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 23:02:08,932 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 Exception in thread "Thread-5" java.lang.OutOfMemoryError: GC overhead limit exceeded 2015-04-12 23:02:08,932 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 23:02:14,543 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 23:02:14,543 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 23:02:14,544 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xfef48e6f, /172.26.3.231:35295 :> /172.26.3.190:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:14,545 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x974fd30c, /172.26.3.231:59351 => /172.26.0.171:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:14,545 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5207814386754341981f5d662f51e2f3 2015-04-12 23:02:17,605 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from afecf7b47f7c42ad8be8ff7b7fcca5e2 2015-04-12 23:02:17,605 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from afecf7b47f7c42ad8be8ff7b7fcca5e2 2015-04-12 23:02:17,605 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from afecf7b47f7c42ad8be8ff7b7fcca5e2 2015-04-12 23:02:17,606 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x4d58d47a, /172.26.3.231:46558 => /172.26.1.64:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:17,609 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e5b5258c3f8444d58d99e1bc59ce5f3d 2015-04-12 23:02:17,609 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 6a1e42c0bc924e5ca9af3e9b0a695e61 2015-04-12 23:02:17,610 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e5b5258c3f8444d58d99e1bc59ce5f3d 2015-04-12 23:02:17,610 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 182c18e3050349e391a3ebb9634f805e 2015-04-12 23:02:17,610 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 182c18e3050349e391a3ebb9634f805e 2015-04-12 23:02:17,610 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b281c896bfa44701a3da552dd478f496 2015-04-12 23:02:17,610 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d1ee46c96b9d4a3187936f69d316fe1c 2015-04-12 23:02:17,611 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1bab3b084f374671a768eadb8d37b8ce 2015-04-12 23:02:17,611 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1bab3b084f374671a768eadb8d37b8ce 2015-04-12 23:02:17,611 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 01864d7c02dc40ff818da7436686990c 2015-04-12 23:02:17,611 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 01864d7c02dc40ff818da7436686990c 2015-04-12 23:02:17,611 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d1ee46c96b9d4a3187936f69d316fe1c 2015-04-12 23:02:08,932 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b18c47f8b11c495ea60f2ade6ed10a6a 2015-04-12 23:02:17,611 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b18c47f8b11c495ea60f2ade6ed10a6a 2015-04-12 23:02:17,612 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4a1614a7cb4f4e4198f1a31cfdf1a844 2015-04-12 23:02:17,612 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf9b07807, /172.26.3.231:52118 :> /172.26.3.249:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:17,612 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 23:02:17,613 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f051f354d76e4582bd1714515ae72c29 2015-04-12 23:02:17,613 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 59445b81fb98424ba7fff0dde2f10788 2015-04-12 23:02:17,613 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from be7b707c3d8c408b9ca99766bede14cc 2015-04-12 23:02:17,613 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x60e709bc, /172.26.3.231:40265 :> /172.26.0.33:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Apr 12, 2015 11:01:34 PM org.jboss.netty.channel.socket.nio.AbstractNioSelector WARNING: Unexpected exception in the selector loop. java.lang.OutOfMemoryError: GC overhead limit exceeded 2015-04-12 23:02:17,619 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x23e9b65a, /172.26.3.231:42798 => /172.26.0.100:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:17,620 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 27c587303bec424586cabce85ebab2fe 2015-04-12 23:02:17,620 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4f45341b1f3c4092bb73aadb05df5736 2015-04-12 23:02:17,620 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x02c10c09, /172.26.3.231:47118 :> /172.26.0.68:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,660 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cb0a1d070bb349d5ad68e1953559de7e 2015-04-12 23:02:20,661 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4f45341b1f3c4092bb73aadb05df5736 2015-04-12 23:02:20,662 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 32949ffabbce443fbc16d6390720fade 2015-04-12 23:02:20,662 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 32949ffabbce443fbc16d6390720fade 2015-04-12 23:02:20,662 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 6a23c56677d446bdbb6a4f4bc0800fc8 2015-04-12 23:02:20,662 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b06a40fa59ad4038bf280da0dcb5dba9 2015-04-12 23:02:20,662 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b06a40fa59ad4038bf280da0dcb5dba9 2015-04-12 23:02:20,663 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cb0a1d070bb349d5ad68e1953559de7e 2015-04-12 23:02:20,663 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 6a23c56677d446bdbb6a4f4bc0800fc8 2015-04-12 23:02:20,663 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xc1b3b7cb, /172.26.3.231:33993 :> /172.26.0.157:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,663 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5dd9d72bacf74e498e188514b2e906ff 2015-04-12 23:02:20,663 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 5dd9d72bacf74e498e188514b2e906ff 2015-04-12 23:02:20,664 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c1f52273a4e34617a350e55d04aa4b57 2015-04-12 23:02:20,664 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xcc564b25, /172.26.3.231:34220 :> /172.26.0.246:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,664 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 76d4ae4f2ac1446d9378448bb47addaa 2015-04-12 23:02:20,664 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 930f13ce130f4333bdba0bc4b4900673 2015-04-12 23:02:20,665 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 7b36c79df5ee4af89d6a5966a9d890b7 2015-04-12 23:02:20,665 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 27c587303bec424586cabce85ebab2fe 2015-04-12 23:02:20,666 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xe03e2a43, /172.26.3.231:54298 => /172.26.3.231:7051] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,666 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,667 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,667 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,668 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,668 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,669 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,669 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,671 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,672 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,668 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 76d4ae4f2ac1446d9378448bb47addaa 2015-04-12 23:02:20,673 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4fb17be5ac424707bf9172c52ca080b5 2015-04-12 23:02:20,673 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 76d4ae4f2ac1446d9378448bb47addaa 2015-04-12 23:02:20,673 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xae43b2ad, /172.26.3.231:44253 :> /172.26.1.53:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,674 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cc79343ea6ca409d99587e843d24054d 2015-04-12 23:02:20,674 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0ed3afbf6d304ddb8d463ce89d0ceff8 2015-04-12 23:02:20,674 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 577153da615f4f58af400d6a68319bc0 2015-04-12 23:02:20,675 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cc79343ea6ca409d99587e843d24054d 2015-04-12 23:02:20,675 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0ed3afbf6d304ddb8d463ce89d0ceff8 2015-04-12 23:02:20,675 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x65d80f91, /172.26.3.231:38604 :> /172.26.0.44:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,676 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b0cca8cc157c4957ba8466717c47f7c1 2015-04-12 23:02:20,676 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 65c9c301df1a4b288a6924b805f8130b 2015-04-12 23:02:20,676 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b0cca8cc157c4957ba8466717c47f7c1 2015-04-12 23:02:20,677 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 65c9c301df1a4b288a6924b805f8130b 2015-04-12 23:02:20,677 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 65c9c301df1a4b288a6924b805f8130b 2015-04-12 23:02:20,677 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x7e159af9, /172.26.3.231:59054 :> /172.26.1.38:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,678 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 23:02:20,678 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from f836a48e19fe49f99776c7b5fe5e5c12 2015-04-12 23:02:20,678 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 23:02:20,678 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 23:02:20,678 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,679 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,680 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,680 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,681 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,681 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,682 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,682 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,684 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,685 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xe03e2a43, /172.26.3.231:54298 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,682 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 23:02:20,686 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0d7a95cfe5794ec886d2cfad7ff3c9f3 2015-04-12 23:02:20,686 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xff4ff08a, /172.26.3.231:46775 :> /172.26.0.80:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,686 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ae51f0105b264219821a479498695c9f 2015-04-12 23:02:20,687 (New I/O worker #4) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:20,687 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x1f5a5800, /172.26.3.231:54693 => /172.26.0.113:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:20,688 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cf9b1b0aca0744c4a456c74d8df8b31f 2015-04-12 23:02:20,688 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3bfab595ced947fda4a6355f7de42b66 2015-04-12 23:02:20,688 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cf9b1b0aca0744c4a456c74d8df8b31f 2015-04-12 23:02:20,688 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3bfab595ced947fda4a6355f7de42b66 2015-04-12 23:02:20,689 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 3bfab595ced947fda4a6355f7de42b66 2015-04-12 23:02:20,689 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from b582c518624547bbaac69b5b860b7606 2015-04-12 23:02:21,956 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 0b3aa654d164490ab8932553954efe85 2015-04-12 23:02:23,220 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 23:02:23,221 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 23:02:23,221 (New I/O worker #4) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from ab40d29079234dad9eea3ed93b9bc9d7 2015-04-12 23:02:23,221 (New I/O worker #4) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf3855c8f, /172.26.3.231:52699 :> /172.26.0.70:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,277 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xc3d4bbd3, /172.26.3.231:54299 => /172.26.3.231:7051] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,278 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,278 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,279 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,279 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,280 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,280 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,280 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,281 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,281 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,281 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,282 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,282 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,282 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,284 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,284 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,285 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,285 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,286 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,286 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,287 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:26,287 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived$GetMasterRegistrationErrCB.call(GetMasterRegistrationReceived.java:171)] Error receiving a response from: 172.26.3.231:7051 kudu.rpc.ConnectionResetException: Connection reset on [id: 0xc3d4bbd3, /172.26.3.231:54299 :> /172.26.3.231:7051] at kudu.rpc.TabletClient.cleanup(TabletClient.java:598) at kudu.rpc.TabletClient.channelDisconnected(TabletClient.java:577) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.close(Channels.java:812) at kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:648) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:100) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:26,288 (New I/O worker #1) [WARN - kudu.rpc.GetMasterRegistrationReceived.incrementCountAndCheckExhausted(GetMasterRegistrationReceived.java:92)] Unable to find the leader of the master quorum(172.26.3.231:7051). 2015-04-12 23:02:29,424 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 16c0a1dd4680440aacec12e23b6c2551 2015-04-12 23:02:29,431 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 99c4e9b15f9a4642932a666528706d1d 2015-04-12 23:02:29,431 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 99c4e9b15f9a4642932a666528706d1d 2015-04-12 23:02:29,431 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from e17b1339ccea4a4a87c956c3f2d039c6 2015-04-12 23:02:29,431 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from a8054cc6b9584893841f098b475c9f6a 2015-04-12 23:02:29,431 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xc7bb0731, /172.26.3.231:43171 :> /172.26.3.239:7050] java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:29,432 (New I/O worker #2) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0x55d2d366, /172.26.3.231:49161 => /172.26.1.51:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.TabletClient.sendRpc(TabletClient.java:153) at kudu.rpc.TabletClient.sendQueuedRpcs(TabletClient.java:678) at kudu.rpc.TabletClient.becomeReady(TabletClient.java:661) at kudu.rpc.TabletClient.sendContext(TabletClient.java:685) at kudu.rpc.SecureRpcHelper.handleSuccessResponse(SecureRpcHelper.java:213) at kudu.rpc.SecureRpcHelper.handleResponse(SecureRpcHelper.java:108) at kudu.rpc.TabletClient.decode(TabletClient.java:305) at kudu.rpc.TabletClient.decode(TabletClient.java:82) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:29,433 (New I/O worker #2) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 568e7aa39c9d416f95113ae674b2607d 2015-04-12 23:02:31,235 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xf530d3cf, /172.26.3.231:47197 => /172.26.0.68:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:31,236 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 1bab3b084f374671a768eadb8d37b8ce 2015-04-12 23:02:31,241 (New I/O worker #1) [ERROR - kudu.rpc.TabletClient.exceptionCaught(TabletClient.java:645)] Unexpected exception from downstream on [id: 0xac4fa7e1, /172.26.3.231:39460 => /172.26.0.59:7050] java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:26) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at kudu.rpc.KuduClient$TabletClientPipeline.sendDownstream(KuduClient.java:1526) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at kudu.rpc.SecureRpcHelper.sendSaslMessage(SecureRpcHelper.java:94) at kudu.rpc.SecureRpcHelper.sendNegotiateMessage(SecureRpcHelper.java:85) at kudu.rpc.SecureRpcHelper.sendHello(SecureRpcHelper.java:79) at kudu.rpc.TabletClient.channelConnected(TabletClient.java:560) at kudu.rpc.TabletClient.handleUpstream(TabletClient.java:569) at kudu.rpc.KuduClient$TabletClientPipeline.sendUpstream(KuduClient.java:1534) at org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:227) at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:162) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-04-12 23:02:32,617 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 4cdb522ac5144879bfe5e56f0c04c9d8 2015-04-12 23:02:32,619 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 8c4058addb27446a9a79719ab1a87412 2015-04-12 23:02:32,619 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 6486d35136cc4bb4a279e71bb8ba4978 2015-04-12 23:02:32,639 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cf9b1b0aca0744c4a456c74d8df8b31f 2015-04-12 23:02:32,639 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cf9b1b0aca0744c4a456c74d8df8b31f 2015-04-12 23:02:32,648 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from cc79343ea6ca409d99587e843d24054d 2015-04-12 23:02:32,648 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from 063b13cae2cc465288520e3d5179c718 2015-04-12 23:02:32,661 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from d55b885b06174162b488c3499e8fe995 2015-04-12 23:02:32,684 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 23:02:32,685 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 2015-04-12 23:02:32,685 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 23:02:32,685 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from c6e4c091611e4eb7a6fdcb1a3bde69f2 2015-04-12 23:02:32,685 (New I/O worker #1) [INFO - kudu.rpc.KuduClient.invalidateTabletCache(KuduClient.java:1066)] Removing a tablet server location from df31ca08502f4d58adc30bd0c27ec214 Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Couldn't cleanup properly because: com.stumbleupon.async.TimeoutException: Timed out after 10000ms when joining Deferred@1451181907(state=PENDING, result=null, callback=kudu.rpc.KuduSession$OperationsInLookupDoneCB@3e23c682 -> wakeup thread Thread-2, errback=passthrough -> wakeup thread Thread-2) Couldn't cleanup properly because: com.stumbleupon.async.TimeoutException: Timed out after 10000ms when joining Deferred@515743917(state=PENDING, result=null, callback=kudu.rpc.KuduSession$OperationsInLookupDoneCB@337fcd9f -> wakeup thread Thread-3, errback=passthrough -> wakeup thread Thread-3) Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation Waiting more than 10 seconds for an insert or mutation