2012-10-08 14:05:50,726 - INFO - ================================================= 2012-10-08 14:05:50,726 - INFO - System Regression Test Framework 2012-10-08 14:05:50,726 - INFO - ================================================= 2012-10-08 14:05:50,727 - INFO - ================================================= 2012-10-08 14:05:50,727 - INFO - setting up remote hosts ... 2012-10-08 14:05:50,728 - INFO - ================================================= 2012-10-08 14:05:55,041 - INFO - ================================================= 2012-10-08 14:05:55,042 - INFO - looking up test suites ... 2012-10-08 14:05:55,042 - INFO - ================================================= 2012-10-08 14:05:55,042 - INFO - found a testsuite : replication_testsuite (system_test_runner) 2012-10-08 14:05:55,043 - INFO - ================================================= 2012-10-08 14:05:55,043 - INFO - setting up remote hosts ... 2012-10-08 14:05:55,043 - INFO - ================================================= 2012-10-08 14:05:59,253 - INFO - found a test module file : replica_basic_test.py (system_test_runner) 2012-10-08 14:05:59,258 - INFO - ================================================= 2012-10-08 14:05:59,258 - INFO - Running Test for : 2012-10-08 14:05:59,258 - INFO - suite : replication_testsuite 2012-10-08 14:05:59,258 - INFO - module : replica_basic_test 2012-10-08 14:05:59,258 - INFO - class : ReplicaBasicTest 2012-10-08 14:05:59,258 - INFO - ================================================= 2012-10-08 14:05:59,442 - INFO - ====================================================== 2012-10-08 14:05:59,442 - INFO - Skipping : testcase_0001 2012-10-08 14:05:59,442 - INFO - ====================================================== 2012-10-08 14:05:59,443 - INFO - ====================================================== 2012-10-08 14:05:59,443 - INFO - Skipping : testcase_0002 2012-10-08 14:05:59,443 - INFO - ====================================================== 2012-10-08 14:05:59,445 - INFO - ====================================================== 2012-10-08 14:05:59,445 - INFO - Skipping : testcase_0003 2012-10-08 14:05:59,445 - INFO - ====================================================== 2012-10-08 14:05:59,446 - INFO - ====================================================== 2012-10-08 14:05:59,446 - INFO - Skipping : testcase_0004 2012-10-08 14:05:59,446 - INFO - ====================================================== 2012-10-08 14:05:59,447 - INFO - ====================================================== 2012-10-08 14:05:59,447 - INFO - Skipping : testcase_0005 2012-10-08 14:05:59,448 - INFO - ====================================================== 2012-10-08 14:05:59,449 - INFO - ====================================================== 2012-10-08 14:05:59,449 - INFO - Skipping : testcase_0006 2012-10-08 14:05:59,449 - INFO - ====================================================== 2012-10-08 14:05:59,450 - INFO - ====================================================== 2012-10-08 14:05:59,450 - INFO - Skipping : testcase_0007 2012-10-08 14:05:59,450 - INFO - ====================================================== 2012-10-08 14:05:59,452 - INFO - ====================================================== 2012-10-08 14:05:59,452 - INFO - Skipping : testcase_0008 2012-10-08 14:05:59,452 - INFO - ====================================================== 2012-10-08 14:05:59,453 - INFO - ====================================================== 2012-10-08 14:05:59,453 - INFO - Skipping : testcase_0009 2012-10-08 14:05:59,453 - INFO - ====================================================== 2012-10-08 14:05:59,454 - INFO - ====================================================== 2012-10-08 14:05:59,454 - INFO - Skipping : testcase_0010 2012-10-08 14:05:59,454 - INFO - ====================================================== 2012-10-08 14:05:59,456 - INFO - ====================================================== 2012-10-08 14:05:59,456 - INFO - Skipping : testcase_0021 2012-10-08 14:05:59,456 - INFO - ====================================================== 2012-10-08 14:05:59,458 - INFO - ====================================================== 2012-10-08 14:05:59,458 - INFO - Skipping : testcase_0022 2012-10-08 14:05:59,458 - INFO - ====================================================== 2012-10-08 14:05:59,460 - INFO - ====================================================== 2012-10-08 14:05:59,460 - INFO - Skipping : testcase_0023 2012-10-08 14:05:59,460 - INFO - ====================================================== 2012-10-08 14:05:59,462 - INFO - ====================================================== 2012-10-08 14:05:59,462 - INFO - Skipping : testcase_0101 2012-10-08 14:05:59,462 - INFO - ====================================================== 2012-10-08 14:05:59,463 - INFO - ====================================================== 2012-10-08 14:05:59,463 - INFO - Skipping : testcase_0102 2012-10-08 14:05:59,463 - INFO - ====================================================== 2012-10-08 14:05:59,465 - INFO - ====================================================== 2012-10-08 14:05:59,465 - INFO - Skipping : testcase_0103 2012-10-08 14:05:59,465 - INFO - ====================================================== 2012-10-08 14:05:59,466 - INFO - ====================================================== 2012-10-08 14:05:59,466 - INFO - Skipping : testcase_0104 2012-10-08 14:05:59,466 - INFO - ====================================================== 2012-10-08 14:05:59,467 - INFO - ====================================================== 2012-10-08 14:05:59,467 - INFO - Skipping : testcase_0105 2012-10-08 14:05:59,467 - INFO - ====================================================== 2012-10-08 14:05:59,469 - INFO - ====================================================== 2012-10-08 14:05:59,469 - INFO - Skipping : testcase_0106 2012-10-08 14:05:59,469 - INFO - ====================================================== 2012-10-08 14:05:59,470 - INFO - ====================================================== 2012-10-08 14:05:59,470 - INFO - Skipping : testcase_0107 2012-10-08 14:05:59,470 - INFO - ====================================================== 2012-10-08 14:05:59,472 - INFO - ====================================================== 2012-10-08 14:05:59,472 - INFO - Skipping : testcase_0108 2012-10-08 14:05:59,472 - INFO - ====================================================== 2012-10-08 14:05:59,473 - INFO - ====================================================== 2012-10-08 14:05:59,473 - INFO - Skipping : testcase_0109 2012-10-08 14:05:59,473 - INFO - ====================================================== 2012-10-08 14:05:59,474 - INFO - ====================================================== 2012-10-08 14:05:59,474 - INFO - Skipping : testcase_0110 2012-10-08 14:05:59,475 - INFO - ====================================================== 2012-10-08 14:05:59,476 - INFO - ====================================================== 2012-10-08 14:05:59,476 - INFO - Skipping : testcase_0111 2012-10-08 14:05:59,476 - INFO - ====================================================== 2012-10-08 14:05:59,477 - INFO - ====================================================== 2012-10-08 14:05:59,477 - INFO - Skipping : testcase_0112 2012-10-08 14:05:59,477 - INFO - ====================================================== 2012-10-08 14:05:59,479 - INFO - ====================================================== 2012-10-08 14:05:59,479 - INFO - Skipping : testcase_0113 2012-10-08 14:05:59,479 - INFO - ====================================================== 2012-10-08 14:05:59,480 - INFO - ====================================================== 2012-10-08 14:05:59,480 - INFO - Skipping : testcase_0114 2012-10-08 14:05:59,480 - INFO - ====================================================== 2012-10-08 14:05:59,481 - INFO - ====================================================== 2012-10-08 14:05:59,482 - INFO - Skipping : testcase_0115 2012-10-08 14:05:59,482 - INFO - ====================================================== 2012-10-08 14:05:59,483 - INFO - ====================================================== 2012-10-08 14:05:59,483 - INFO - Skipping : testcase_0116 2012-10-08 14:05:59,483 - INFO - ====================================================== 2012-10-08 14:05:59,484 - INFO - ====================================================== 2012-10-08 14:05:59,484 - INFO - Skipping : testcase_0117 2012-10-08 14:05:59,484 - INFO - ====================================================== 2012-10-08 14:05:59,486 - INFO - ====================================================== 2012-10-08 14:05:59,486 - INFO - Skipping : testcase_0118 2012-10-08 14:05:59,486 - INFO - ====================================================== ======================================================================================= Test Case Name : testcase_0121 ======================================================================================= Description : 01 : Leader Failure in Replication with multi topics & partitions : Base Test 02 : Produce and consume messages to 2 topics - 3 partitions 03 : This test sends messages to 3 replicas 04 : To trigger leader election: find the leader and terminate by controlled failure (kill -15) 05 : Restart the terminated broker 06 : Lookup brokers' log4j messages and verify that leader is re-elected successfully 07 : At the end it verifies the log size and contents 08 : Use a consumer to verify no message loss. 09 : Producer dimensions : mode:sync, acks:-1, comp:0 10 : Log segment size : 102400 ======================================================================================= Test Case Args : bounce_leader : true sleep_seconds_between_producer_calls : 1 num_iteration : 3 num_partition : 3 message_producing_free_time_sec : 15 replica_factor : 3 ======================================================================================= 2012-10-08 14:05:59,487 - INFO - found a new cluster_config : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/cluster_config.json (system_test_utils) 2012-10-08 14:05:59,489 - INFO - ================================================= 2012-10-08 14:05:59,489 - INFO - setting up remote hosts ... 2012-10-08 14:05:59,489 - INFO - ================================================= 2012-10-08 14:06:05,348 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:06:07,893 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:06:10,382 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:06:12,928 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:06:15,476 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:06:15,476 - INFO - skipping role [producer_performance] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:06:15,476 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:06:15,476 - INFO - skipping role [producer_performance] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:06:15,476 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:06:15,477 - INFO - skipping role [console_consumer] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:06:15,477 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:06:15,477 - INFO - skipping role [console_consumer] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:06:18,602 - INFO - calling generate_properties_files (kafka_system_test_utils) 2012-10-08 14:06:18,602 - INFO - config template (source) pathname : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/config (kafka_system_test_utils) 2012-10-08 14:06:18,603 - INFO - testcase config (dest) pathname : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/config (kafka_system_test_utils) 2012-10-08 14:06:21,715 - INFO - ====================================================== 2012-10-08 14:06:21,715 - INFO - starting zookeepers 2012-10-08 14:06:21,716 - INFO - ====================================================== 2012-10-08 14:06:24,154 - INFO - starting zookeeper in host [localhost] on client port [2188] (kafka_system_test_utils) 2012-10-08 14:06:30,593 - INFO - starting metrics collection on jmx port : 9990 (metrics) 2012-10-08 14:06:32,059 - INFO - sleeping for 2s 2012-10-08 14:06:34,062 - INFO - ====================================================== 2012-10-08 14:06:34,062 - INFO - starting brokers 2012-10-08 14:06:34,062 - INFO - ====================================================== 2012-10-08 14:06:34,063 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:06:40,526 - INFO - starting metrics collection on jmx port : 9991 (metrics) 2012-10-08 14:07:07,102 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:07:13,553 - INFO - starting metrics collection on jmx port : 9992 (metrics) 2012-10-08 14:07:41,275 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:07:47,739 - INFO - starting metrics collection on jmx port : 9993 (metrics) 2012-10-08 14:08:16,378 - INFO - sleeping for 5s 2012-10-08 14:08:21,380 - INFO - ====================================================== 2012-10-08 14:08:21,380 - INFO - creating topics 2012-10-08 14:08:21,380 - INFO - ====================================================== 2012-10-08 14:08:21,381 - INFO - creating topic: [test_1] at: [localhost:2188] (kafka_system_test_utils) 2012-10-08 14:08:21,385 - INFO - creating topic: [test_2] at: [localhost:2188] (kafka_system_test_utils) 2012-10-08 14:08:21,388 - INFO - sleeping for 5s 2012-10-08 14:08:26,388 - INFO - ====================================================== 2012-10-08 14:08:26,388 - INFO - starting producer in the background 2012-10-08 14:08:26,389 - INFO - ====================================================== 2012-10-08 14:08:26,390 - INFO - starting producer preformance (kafka_system_test_utils) 2012-10-08 14:08:27,390 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [0] (kafka_system_test_utils) 2012-10-08 14:08:28,391 - INFO - starting metrics collection on jmx port : 9997 (metrics) 2012-10-08 14:08:31,393 - INFO - starting producer preformance (kafka_system_test_utils) 2012-10-08 14:08:38,534 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [0] (kafka_system_test_utils) 2012-10-08 14:08:39,534 - INFO - starting metrics collection on jmx port : 9998 (metrics) 2012-10-08 14:08:42,461 - INFO - sleeping for 15 sec to produce some messages 2012-10-08 14:08:49,084 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [100] (kafka_system_test_utils) 2012-10-08 14:08:57,466 - INFO - ====================================================== 2012-10-08 14:08:57,466 - INFO - Iteration 1 of 3 2012-10-08 14:08:57,466 - INFO - ====================================================== 2012-10-08 14:08:57,466 - INFO - ====================================================== 2012-10-08 14:08:57,466 - INFO - looking up leader 2012-10-08 14:08:57,467 - INFO - ====================================================== 2012-10-08 14:08:58,997 - INFO - ====================================================== 2012-10-08 14:08:58,997 - INFO - validating leader election 2012-10-08 14:08:58,997 - INFO - ====================================================== 2012-10-08 14:08:58,998 - INFO - found leader in entity [3] with brokerid [3] for partition [2] (kafka_system_test_utils) 2012-10-08 14:08:58,998 - INFO - ====================================================== 2012-10-08 14:08:58,998 - INFO - bounce_leader flag : true 2012-10-08 14:08:58,998 - INFO - ====================================================== 2012-10-08 14:08:58,998 - INFO - stopping leader in entity 3 with pid 20889 (kafka_system_test_utils) 2012-10-08 14:08:59,446 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:08:59,697 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [100] (kafka_system_test_utils) 2012-10-08 14:09:09,456 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:09:10,901 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:09:11,276 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [200] (kafka_system_test_utils) 2012-10-08 14:09:12,503 - INFO - leader Re-election Latency: 0.351999998093 sec (kafka_system_test_utils) 2012-10-08 14:09:12,503 - INFO - ====================================================== 2012-10-08 14:09:12,504 - INFO - starting the previously terminated broker 2012-10-08 14:09:12,504 - INFO - ====================================================== 2012-10-08 14:09:12,505 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:09:19,074 - INFO - starting metrics collection on jmx port : 9993 (metrics) 2012-10-08 14:09:21,891 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [200] (kafka_system_test_utils) 2012-10-08 14:09:32,631 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [300] (kafka_system_test_utils) 2012-10-08 14:09:43,325 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [300] (kafka_system_test_utils) 2012-10-08 14:09:47,804 - INFO - sleeping for 15s 2012-10-08 14:09:53,666 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [400] (kafka_system_test_utils) 2012-10-08 14:10:02,805 - INFO - ====================================================== 2012-10-08 14:10:02,805 - INFO - Iteration 2 of 3 2012-10-08 14:10:02,805 - INFO - ====================================================== 2012-10-08 14:10:02,805 - INFO - ====================================================== 2012-10-08 14:10:02,805 - INFO - looking up leader 2012-10-08 14:10:02,806 - INFO - ====================================================== 2012-10-08 14:10:04,018 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [400] (kafka_system_test_utils) 2012-10-08 14:10:04,046 - INFO - ====================================================== 2012-10-08 14:10:04,047 - INFO - validating leader election 2012-10-08 14:10:04,047 - INFO - ====================================================== 2012-10-08 14:10:04,047 - INFO - found leader in entity [1] with brokerid [1] for partition [0] (kafka_system_test_utils) 2012-10-08 14:10:04,047 - INFO - ====================================================== 2012-10-08 14:10:04,047 - INFO - bounce_leader flag : true 2012-10-08 14:10:04,047 - INFO - ====================================================== 2012-10-08 14:10:04,048 - INFO - stopping leader in entity 1 with pid 14691 (kafka_system_test_utils) 2012-10-08 14:10:05,168 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:10:15,174 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:10:16,404 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:10:17,636 - INFO - leader Re-election Latency: 0.325999975204 sec (kafka_system_test_utils) 2012-10-08 14:10:17,636 - INFO - ====================================================== 2012-10-08 14:10:17,637 - INFO - starting the previously terminated broker 2012-10-08 14:10:17,637 - INFO - ====================================================== 2012-10-08 14:10:17,638 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:10:24,197 - INFO - starting metrics collection on jmx port : 9991 (metrics) 2012-10-08 14:10:28,466 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [500] (kafka_system_test_utils) 2012-10-08 14:10:40,084 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [500] (kafka_system_test_utils) 2012-10-08 14:10:52,122 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [600] (kafka_system_test_utils) 2012-10-08 14:10:53,998 - INFO - sleeping for 15s 2012-10-08 14:11:04,341 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [600] (kafka_system_test_utils) 2012-10-08 14:11:09,002 - INFO - ====================================================== 2012-10-08 14:11:09,002 - INFO - Iteration 3 of 3 2012-10-08 14:11:09,002 - INFO - ====================================================== 2012-10-08 14:11:09,002 - INFO - ====================================================== 2012-10-08 14:11:09,003 - INFO - looking up leader 2012-10-08 14:11:09,003 - INFO - ====================================================== 2012-10-08 14:11:10,546 - INFO - ====================================================== 2012-10-08 14:11:10,546 - INFO - validating leader election 2012-10-08 14:11:10,547 - INFO - ====================================================== 2012-10-08 14:11:10,547 - INFO - found leader in entity [2] with brokerid [2] for partition [2] (kafka_system_test_utils) 2012-10-08 14:11:10,547 - INFO - ====================================================== 2012-10-08 14:11:10,547 - INFO - bounce_leader flag : true 2012-10-08 14:11:10,547 - INFO - ====================================================== 2012-10-08 14:11:10,547 - INFO - stopping leader in entity 2 with pid 17783 (kafka_system_test_utils) 2012-10-08 14:11:11,216 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:11:17,109 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [700] (kafka_system_test_utils) 2012-10-08 14:11:21,217 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:11:22,524 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:11:23,963 - INFO - leader Re-election Latency: 0.334000110626 sec (kafka_system_test_utils) 2012-10-08 14:11:23,963 - INFO - ====================================================== 2012-10-08 14:11:23,963 - INFO - starting the previously terminated broker 2012-10-08 14:11:23,963 - INFO - ====================================================== 2012-10-08 14:11:23,964 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:11:27,657 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [700] (kafka_system_test_utils) 2012-10-08 14:11:30,443 - INFO - starting metrics collection on jmx port : 9992 (metrics) 2012-10-08 14:11:38,962 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [800] (kafka_system_test_utils) 2012-10-08 14:11:48,820 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [800] (kafka_system_test_utils) 2012-10-08 14:11:59,814 - INFO - sleeping for 15s 2012-10-08 14:12:00,694 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [900] (kafka_system_test_utils) 2012-10-08 14:12:10,769 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [900] (kafka_system_test_utils) 2012-10-08 14:12:22,248 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1000] (kafka_system_test_utils) 2012-10-08 14:12:35,806 - INFO - status of backgroundProducerStopped : [True] (ReplicationUtils) 2012-10-08 14:12:36,808 - INFO - all producer threads completed (ReplicationUtils) 2012-10-08 14:12:36,808 - INFO - ====================================================== 2012-10-08 14:12:36,808 - INFO - starting consumer in the background 2012-10-08 14:12:36,808 - INFO - ====================================================== 2012-10-08 14:12:36,808 - INFO - starting console consumer (kafka_system_test_utils) 2012-10-08 14:12:38,815 - INFO - starting metrics collection on jmx port : 9999 (metrics) 2012-10-08 14:12:44,045 - INFO - starting console consumer (kafka_system_test_utils) 2012-10-08 14:12:46,051 - INFO - starting metrics collection on jmx port : 9099 (metrics) 2012-10-08 14:12:51,987 - INFO - sleeping for 10s 2012-10-08 14:13:01,996 - INFO - ====================================================== 2012-10-08 14:13:01,996 - INFO - stopping all entities 2012-10-08 14:13:01,996 - INFO - ====================================================== 2012-10-08 14:13:10,507 - INFO - ================================================ 2012-10-08 14:13:10,507 - INFO - collecting logs from remote machines 2012-10-08 14:13:10,507 - INFO - ================================================ 2012-10-08 14:13:17,385 - INFO - ====================================================== 2012-10-08 14:13:17,385 - INFO - validating data matched 2012-10-08 14:13:17,385 - INFO - ====================================================== 2012-10-08 14:13:17,407 - INFO - no. of unique messages on topic [test_1] sent from publisher : 1100 (kafka_system_test_utils) 2012-10-08 14:13:17,407 - INFO - no. of unique messages on topic [test_1] received by consumer : 1100 (kafka_system_test_utils) 2012-10-08 14:13:17,460 - INFO - no. of unique messages on topic [test_2] sent from publisher : 1000 (kafka_system_test_utils) 2012-10-08 14:13:17,460 - INFO - no. of unique messages on topic [test_2] received by consumer : 1000 (kafka_system_test_utils) 2012-10-08 14:13:17,460 - INFO - ================================================ 2012-10-08 14:13:17,460 - INFO - validating broker log segment checksums 2012-10-08 14:13:17,460 - INFO - ================================================ 2012-10-08 14:13:31,942 - INFO - log segment files checksum : {u'broker-1': {'test_1-0/00000000000000000000.kafka': '2470608590', 'test_1-0/00000000000000102510.kafka': '2165908748', 'test_1-1/00000000000000000000.kafka': '4164745606', 'test_1-1/00000000000000102510.kafka': '3502300135', 'test_1-2/00000000000000000000.kafka': '597352581', 'test_1-2/00000000000000102510.kafka': '1819399430', 'test_2-0/00000000000000000000.kafka': '2094120552', 'test_2-0/00000000000000102510.kafka': '4020495642', 'test_2-1/00000000000000000000.kafka': '3767172923', 'test_2-1/00000000000000102510.kafka': '3373903885', 'test_2-2/00000000000000000000.kafka': '1225350130', 'test_2-2/00000000000000102510.kafka': '1440333614'}, u'broker-2': {'test_1-0/00000000000000000000.kafka': '2470608590', 'test_1-0/00000000000000102510.kafka': '2165908748', 'test_1-1/00000000000000000000.kafka': '4164745606', 'test_1-1/00000000000000102510.kafka': '3502300135', 'test_1-2/00000000000000000000.kafka': '597352581', 'test_1-2/00000000000000102510.kafka': '1819399430', 'test_2-0/00000000000000000000.kafka': '2094120552', 'test_2-0/00000000000000102510.kafka': '4020495642', 'test_2-1/00000000000000000000.kafka': '3767172923', 'test_2-1/00000000000000102510.kafka': '3373903885', 'test_2-2/00000000000000000000.kafka': '1225350130', 'test_2-2/00000000000000102510.kafka': '1440333614'}, u'broker-3': {'test_1-0/00000000000000000000.kafka': '2470608590', 'test_1-0/00000000000000102510.kafka': '2165908748', 'test_1-1/00000000000000000000.kafka': '4164745606', 'test_1-1/00000000000000102510.kafka': '3502300135', 'test_1-2/00000000000000000000.kafka': '597352581', 'test_1-2/00000000000000102510.kafka': '1819399430', 'test_2-0/00000000000000000000.kafka': '2094120552', 'test_2-0/00000000000000102510.kafka': '4020495642', 'test_2-1/00000000000000000000.kafka': '3767172923', 'test_2-1/00000000000000102510.kafka': '3373903885', 'test_2-2/00000000000000000000.kafka': '1225350130', 'test_2-2/00000000000000102510.kafka': '1440333614'}} 2012-10-08 14:13:32,175 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,176 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,177 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,179 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:13:32,216 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,216 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,217 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:13:32,252 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,253 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,253 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:13:32,289 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,290 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,291 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,293 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:13:32,327 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,328 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,329 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs.csv: There should be exactly one column that matches attribute: kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Mean in headers: time,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:50thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:75thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:95thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:98thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:999thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:99thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Count,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Max1349730565049 (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 111, in plot_graphs attributeColumnIndex = ensure_valid_headers(headers, attribute) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 86, in ensure_valid_headers " headers: {0}".format(",".join(headers))) Exception: There should be exactly one column that matches attribute: kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Mean in headers: time,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:50thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:75thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:95thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:98thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:999thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:99thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Count,kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Max1349730565049 2012-10-08 14:13:32,330 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Mean.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:13:32,365 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,366 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,366 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:13:32,403 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,404 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,404 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,406 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:13:32,441 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,442 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,442 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs.csv: There should be exactly one column that matches attribute: kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Mean in headers: time,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:50thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:75thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:95thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:98thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:999thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:99thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Count,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Max1349730578900 (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 111, in plot_graphs attributeColumnIndex = ensure_valid_headers(headers, attribute) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 86, in ensure_valid_headers " headers: {0}".format(",".join(headers))) Exception: There should be exactly one column that matches attribute: kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Mean in headers: time,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:50thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:75thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:95thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:98thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:999thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:99thPercentile,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Count,kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Max1349730578900 2012-10-08 14:13:32,444 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Mean.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:13:32,480 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,481 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:32,481 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:13:33,463 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.server:type=ControllerStat,name=LeaderElectionRateAndTimeMs:OneMinuteRate.svg: line contains NULL byte (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: line contains NULL byte 2012-10-08 14:13:33,497 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,498 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,499 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:OneMinuteRate.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:13:33,533 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,534 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,534 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-3/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,536 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:13:33,572 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,573 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,574 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-3/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,576 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:13:33,610 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-1/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,610 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-2/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,611 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/logs/broker-3/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:13:33,613 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0121/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:13:35,399 - INFO - ====================================================== 2012-10-08 14:13:35,399 - INFO - stopping all entities - please wait ... 2012-10-08 14:13:35,400 - INFO - ====================================================== ======================================================================================= Test Case Name : testcase_0122 ======================================================================================= Description : 01 : Leader Failure in Replication with multi topics & partitions : 1. acks => 1 02 : Produce and consume messages to 2 topics - 3 partitions 03 : This test sends messages to 3 replicas 04 : To trigger leader election: find the leader and terminate by controlled failure (kill -15) 05 : Restart the terminated broker 06 : Lookup brokers' log4j messages and verify that leader is re-elected successfully 07 : At the end it verifies the log size and contents 08 : Use a consumer to verify no message loss. 09 : Producer dimensions : mode:sync, acks:1, comp:0 10 : Log segment size : 512000 ======================================================================================= Test Case Args : bounce_leader : true sleep_seconds_between_producer_calls : 1 num_iteration : 3 num_partition : 3 message_producing_free_time_sec : 15 replica_factor : 3 ======================================================================================= 2012-10-08 14:14:46,876 - INFO - found a new cluster_config : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/cluster_config.json (system_test_utils) 2012-10-08 14:14:46,879 - INFO - ================================================= 2012-10-08 14:14:46,879 - INFO - setting up remote hosts ... 2012-10-08 14:14:46,879 - INFO - ================================================= 2012-10-08 14:14:53,374 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:14:56,069 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:14:58,675 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:15:01,114 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:15:03,573 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:15:03,573 - INFO - skipping role [producer_performance] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:15:03,573 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:15:03,573 - INFO - skipping role [producer_performance] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:15:03,573 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:15:03,574 - INFO - skipping role [console_consumer] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:15:03,574 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:15:03,574 - INFO - skipping role [console_consumer] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:15:06,506 - INFO - calling generate_properties_files (kafka_system_test_utils) 2012-10-08 14:15:06,506 - INFO - config template (source) pathname : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/config (kafka_system_test_utils) 2012-10-08 14:15:06,506 - INFO - testcase config (dest) pathname : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/config (kafka_system_test_utils) 2012-10-08 14:15:09,693 - INFO - ====================================================== 2012-10-08 14:15:09,693 - INFO - starting zookeepers 2012-10-08 14:15:09,694 - INFO - ====================================================== 2012-10-08 14:15:11,978 - INFO - starting zookeeper in host [localhost] on client port [2188] (kafka_system_test_utils) 2012-10-08 14:15:18,302 - INFO - starting metrics collection on jmx port : 9990 (metrics) 2012-10-08 14:15:19,617 - INFO - sleeping for 2s 2012-10-08 14:15:21,619 - INFO - ====================================================== 2012-10-08 14:15:21,619 - INFO - starting brokers 2012-10-08 14:15:21,619 - INFO - ====================================================== 2012-10-08 14:15:21,620 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:15:28,252 - INFO - starting metrics collection on jmx port : 9991 (metrics) 2012-10-08 14:15:54,557 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:16:00,919 - INFO - starting metrics collection on jmx port : 9992 (metrics) 2012-10-08 14:16:29,047 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:16:35,501 - INFO - starting metrics collection on jmx port : 9993 (metrics) 2012-10-08 14:17:03,373 - INFO - sleeping for 5s 2012-10-08 14:17:08,374 - INFO - ====================================================== 2012-10-08 14:17:08,374 - INFO - creating topics 2012-10-08 14:17:08,374 - INFO - ====================================================== 2012-10-08 14:17:08,375 - INFO - creating topic: [test_1] at: [localhost:2188] (kafka_system_test_utils) 2012-10-08 14:17:08,382 - INFO - creating topic: [test_2] at: [localhost:2188] (kafka_system_test_utils) 2012-10-08 14:17:08,387 - INFO - sleeping for 5s 2012-10-08 14:17:13,387 - INFO - ====================================================== 2012-10-08 14:17:13,388 - INFO - starting producer in the background 2012-10-08 14:17:13,388 - INFO - ====================================================== 2012-10-08 14:17:13,389 - INFO - starting producer preformance (kafka_system_test_utils) 2012-10-08 14:17:14,389 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [0] (kafka_system_test_utils) 2012-10-08 14:17:15,389 - INFO - starting metrics collection on jmx port : 9997 (metrics) 2012-10-08 14:17:16,968 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [100] (kafka_system_test_utils) 2012-10-08 14:17:18,444 - INFO - starting producer preformance (kafka_system_test_utils) 2012-10-08 14:17:19,445 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [0] (kafka_system_test_utils) 2012-10-08 14:17:20,445 - INFO - starting metrics collection on jmx port : 9998 (metrics) 2012-10-08 14:17:21,062 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [200] (kafka_system_test_utils) 2012-10-08 14:17:22,571 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [100] (kafka_system_test_utils) 2012-10-08 14:17:23,322 - INFO - sleeping for 15 sec to produce some messages 2012-10-08 14:17:23,968 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [300] (kafka_system_test_utils) 2012-10-08 14:17:25,370 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [200] (kafka_system_test_utils) 2012-10-08 14:17:26,912 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [400] (kafka_system_test_utils) 2012-10-08 14:17:28,361 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [300] (kafka_system_test_utils) 2012-10-08 14:17:29,774 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [500] (kafka_system_test_utils) 2012-10-08 14:17:31,455 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [400] (kafka_system_test_utils) 2012-10-08 14:17:33,017 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [600] (kafka_system_test_utils) 2012-10-08 14:17:34,438 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [500] (kafka_system_test_utils) 2012-10-08 14:17:36,031 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [700] (kafka_system_test_utils) 2012-10-08 14:17:37,404 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [600] (kafka_system_test_utils) 2012-10-08 14:17:38,338 - INFO - ====================================================== 2012-10-08 14:17:38,338 - INFO - Iteration 1 of 3 2012-10-08 14:17:38,338 - INFO - ====================================================== 2012-10-08 14:17:38,338 - INFO - ====================================================== 2012-10-08 14:17:38,338 - INFO - looking up leader 2012-10-08 14:17:38,338 - INFO - ====================================================== 2012-10-08 14:17:39,044 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [800] (kafka_system_test_utils) 2012-10-08 14:17:39,855 - INFO - ====================================================== 2012-10-08 14:17:39,855 - INFO - validating leader election 2012-10-08 14:17:39,855 - INFO - ====================================================== 2012-10-08 14:17:39,855 - INFO - found leader in entity [3] with brokerid [3] for partition [1] (kafka_system_test_utils) 2012-10-08 14:17:39,855 - INFO - ====================================================== 2012-10-08 14:17:39,855 - INFO - bounce_leader flag : true 2012-10-08 14:17:39,855 - INFO - ====================================================== 2012-10-08 14:17:39,856 - INFO - stopping leader in entity 3 with pid 9814 (kafka_system_test_utils) 2012-10-08 14:17:40,351 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:17:42,612 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [700] (kafka_system_test_utils) 2012-10-08 14:17:44,452 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [900] (kafka_system_test_utils) 2012-10-08 14:17:45,903 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [800] (kafka_system_test_utils) 2012-10-08 14:17:47,326 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1000] (kafka_system_test_utils) 2012-10-08 14:17:48,982 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [900] (kafka_system_test_utils) 2012-10-08 14:17:50,357 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1100] (kafka_system_test_utils) 2012-10-08 14:17:50,361 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:17:51,562 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:17:52,021 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1000] (kafka_system_test_utils) 2012-10-08 14:17:53,556 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1200] (kafka_system_test_utils) 2012-10-08 14:17:53,773 - INFO - leader Re-election Latency: 0.325000047684 sec (kafka_system_test_utils) 2012-10-08 14:17:53,773 - INFO - ====================================================== 2012-10-08 14:17:53,773 - INFO - starting the previously terminated broker 2012-10-08 14:17:53,773 - INFO - ====================================================== 2012-10-08 14:17:53,774 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:17:54,947 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1100] (kafka_system_test_utils) 2012-10-08 14:17:56,361 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1300] (kafka_system_test_utils) 2012-10-08 14:17:57,932 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1200] (kafka_system_test_utils) 2012-10-08 14:17:59,329 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1400] (kafka_system_test_utils) 2012-10-08 14:18:00,445 - INFO - starting metrics collection on jmx port : 9993 (metrics) 2012-10-08 14:18:00,685 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1300] (kafka_system_test_utils) 2012-10-08 14:18:02,685 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1500] (kafka_system_test_utils) 2012-10-08 14:18:04,272 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1400] (kafka_system_test_utils) 2012-10-08 14:18:05,916 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1600] (kafka_system_test_utils) 2012-10-08 14:18:07,428 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1500] (kafka_system_test_utils) 2012-10-08 14:18:08,847 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1700] (kafka_system_test_utils) 2012-10-08 14:18:10,368 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1600] (kafka_system_test_utils) 2012-10-08 14:18:11,829 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1800] (kafka_system_test_utils) 2012-10-08 14:18:13,466 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1700] (kafka_system_test_utils) 2012-10-08 14:18:14,892 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1900] (kafka_system_test_utils) 2012-10-08 14:18:16,308 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1800] (kafka_system_test_utils) 2012-10-08 14:18:17,902 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2000] (kafka_system_test_utils) 2012-10-08 14:18:19,571 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1900] (kafka_system_test_utils) 2012-10-08 14:18:21,079 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2100] (kafka_system_test_utils) 2012-10-08 14:18:22,601 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2000] (kafka_system_test_utils) 2012-10-08 14:18:24,136 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2200] (kafka_system_test_utils) 2012-10-08 14:18:25,554 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2100] (kafka_system_test_utils) 2012-10-08 14:18:27,059 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2300] (kafka_system_test_utils) 2012-10-08 14:18:28,601 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2200] (kafka_system_test_utils) 2012-10-08 14:18:29,315 - INFO - sleeping for 15s 2012-10-08 14:18:29,967 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2400] (kafka_system_test_utils) 2012-10-08 14:18:31,739 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2300] (kafka_system_test_utils) 2012-10-08 14:18:33,292 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2500] (kafka_system_test_utils) 2012-10-08 14:18:34,679 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2400] (kafka_system_test_utils) 2012-10-08 14:18:36,107 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2600] (kafka_system_test_utils) 2012-10-08 14:18:37,720 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2500] (kafka_system_test_utils) 2012-10-08 14:18:39,149 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2700] (kafka_system_test_utils) 2012-10-08 14:18:40,550 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2600] (kafka_system_test_utils) 2012-10-08 14:18:41,965 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2800] (kafka_system_test_utils) 2012-10-08 14:18:43,519 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2700] (kafka_system_test_utils) 2012-10-08 14:18:44,316 - INFO - ====================================================== 2012-10-08 14:18:44,317 - INFO - Iteration 2 of 3 2012-10-08 14:18:44,317 - INFO - ====================================================== 2012-10-08 14:18:44,317 - INFO - ====================================================== 2012-10-08 14:18:44,317 - INFO - looking up leader 2012-10-08 14:18:44,317 - INFO - ====================================================== 2012-10-08 14:18:45,205 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2900] (kafka_system_test_utils) 2012-10-08 14:18:45,680 - INFO - ====================================================== 2012-10-08 14:18:45,680 - INFO - validating leader election 2012-10-08 14:18:45,680 - INFO - ====================================================== 2012-10-08 14:18:45,680 - INFO - found leader in entity [1] with brokerid [1] for partition [2] (kafka_system_test_utils) 2012-10-08 14:18:45,680 - INFO - ====================================================== 2012-10-08 14:18:45,680 - INFO - bounce_leader flag : true 2012-10-08 14:18:45,680 - INFO - ====================================================== 2012-10-08 14:18:45,681 - INFO - stopping leader in entity 1 with pid 3557 (kafka_system_test_utils) 2012-10-08 14:18:46,332 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:18:48,376 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2800] (kafka_system_test_utils) 2012-10-08 14:18:51,084 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3000] (kafka_system_test_utils) 2012-10-08 14:18:53,821 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2900] (kafka_system_test_utils) 2012-10-08 14:18:56,338 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:18:56,538 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3100] (kafka_system_test_utils) 2012-10-08 14:18:58,442 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:18:59,090 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3000] (kafka_system_test_utils) 2012-10-08 14:18:59,905 - INFO - leader Re-election Latency: 0.322999954224 sec (kafka_system_test_utils) 2012-10-08 14:18:59,905 - INFO - ====================================================== 2012-10-08 14:18:59,905 - INFO - starting the previously terminated broker 2012-10-08 14:18:59,906 - INFO - ====================================================== 2012-10-08 14:18:59,906 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:19:01,944 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3200] (kafka_system_test_utils) 2012-10-08 14:19:03,491 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3100] (kafka_system_test_utils) 2012-10-08 14:19:05,230 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3300] (kafka_system_test_utils) 2012-10-08 14:19:06,459 - INFO - starting metrics collection on jmx port : 9991 (metrics) 2012-10-08 14:19:06,789 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3200] (kafka_system_test_utils) 2012-10-08 14:19:08,400 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3400] (kafka_system_test_utils) 2012-10-08 14:19:09,791 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3300] (kafka_system_test_utils) 2012-10-08 14:19:11,388 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3500] (kafka_system_test_utils) 2012-10-08 14:19:12,876 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3400] (kafka_system_test_utils) 2012-10-08 14:19:14,432 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3600] (kafka_system_test_utils) 2012-10-08 14:19:15,798 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3500] (kafka_system_test_utils) 2012-10-08 14:19:17,322 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3700] (kafka_system_test_utils) 2012-10-08 14:19:18,955 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3600] (kafka_system_test_utils) 2012-10-08 14:19:20,437 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3800] (kafka_system_test_utils) 2012-10-08 14:19:21,835 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3700] (kafka_system_test_utils) 2012-10-08 14:19:23,256 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3900] (kafka_system_test_utils) 2012-10-08 14:19:24,819 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3800] (kafka_system_test_utils) 2012-10-08 14:19:26,336 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4000] (kafka_system_test_utils) 2012-10-08 14:19:27,789 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3900] (kafka_system_test_utils) 2012-10-08 14:19:29,345 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4100] (kafka_system_test_utils) 2012-10-08 14:19:30,770 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4000] (kafka_system_test_utils) 2012-10-08 14:19:32,222 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4200] (kafka_system_test_utils) 2012-10-08 14:19:33,785 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4100] (kafka_system_test_utils) 2012-10-08 14:19:34,617 - INFO - sleeping for 15s 2012-10-08 14:19:35,403 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4300] (kafka_system_test_utils) 2012-10-08 14:19:36,820 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4200] (kafka_system_test_utils) 2012-10-08 14:19:38,530 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4400] (kafka_system_test_utils) 2012-10-08 14:19:40,049 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4300] (kafka_system_test_utils) 2012-10-08 14:19:41,584 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4500] (kafka_system_test_utils) 2012-10-08 14:19:43,027 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4400] (kafka_system_test_utils) 2012-10-08 14:19:44,762 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4600] (kafka_system_test_utils) 2012-10-08 14:19:46,460 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4500] (kafka_system_test_utils) 2012-10-08 14:19:48,166 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4700] (kafka_system_test_utils) 2012-10-08 14:19:49,587 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4600] (kafka_system_test_utils) 2012-10-08 14:19:49,631 - INFO - ====================================================== 2012-10-08 14:19:49,631 - INFO - Iteration 3 of 3 2012-10-08 14:19:49,632 - INFO - ====================================================== 2012-10-08 14:19:49,632 - INFO - ====================================================== 2012-10-08 14:19:49,632 - INFO - looking up leader 2012-10-08 14:19:49,632 - INFO - ====================================================== 2012-10-08 14:19:51,031 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4800] (kafka_system_test_utils) 2012-10-08 14:19:52,128 - INFO - ====================================================== 2012-10-08 14:19:52,128 - INFO - validating leader election 2012-10-08 14:19:52,128 - INFO - ====================================================== 2012-10-08 14:19:52,128 - INFO - found leader in entity [2] with brokerid [2] for partition [2] (kafka_system_test_utils) 2012-10-08 14:19:52,129 - INFO - ====================================================== 2012-10-08 14:19:52,129 - INFO - bounce_leader flag : true 2012-10-08 14:19:52,129 - INFO - ====================================================== 2012-10-08 14:19:52,129 - INFO - stopping leader in entity 2 with pid 6710 (kafka_system_test_utils) 2012-10-08 14:19:52,464 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4700] (kafka_system_test_utils) 2012-10-08 14:19:52,946 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:19:55,045 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4900] (kafka_system_test_utils) 2012-10-08 14:19:56,693 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4800] (kafka_system_test_utils) 2012-10-08 14:19:58,173 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5000] (kafka_system_test_utils) 2012-10-08 14:19:59,885 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4900] (kafka_system_test_utils) 2012-10-08 14:20:01,279 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5100] (kafka_system_test_utils) 2012-10-08 14:20:02,691 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5000] (kafka_system_test_utils) 2012-10-08 14:20:02,955 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:20:04,304 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5200] (kafka_system_test_utils) 2012-10-08 14:20:05,149 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:20:05,913 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5100] (kafka_system_test_utils) 2012-10-08 14:20:06,623 - INFO - leader Re-election Latency: 0.325000047684 sec (kafka_system_test_utils) 2012-10-08 14:20:06,624 - INFO - ====================================================== 2012-10-08 14:20:06,624 - INFO - starting the previously terminated broker 2012-10-08 14:20:06,624 - INFO - ====================================================== 2012-10-08 14:20:06,624 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:20:08,680 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5300] (kafka_system_test_utils) 2012-10-08 14:20:10,361 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5200] (kafka_system_test_utils) 2012-10-08 14:20:12,114 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5400] (kafka_system_test_utils) 2012-10-08 14:20:13,071 - INFO - starting metrics collection on jmx port : 9992 (metrics) 2012-10-08 14:20:13,659 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5300] (kafka_system_test_utils) 2012-10-08 14:20:15,130 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5500] (kafka_system_test_utils) 2012-10-08 14:20:16,585 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5400] (kafka_system_test_utils) 2012-10-08 14:20:18,128 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5600] (kafka_system_test_utils) 2012-10-08 14:20:19,544 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5500] (kafka_system_test_utils) 2012-10-08 14:20:21,273 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5700] (kafka_system_test_utils) 2012-10-08 14:20:22,868 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5600] (kafka_system_test_utils) 2012-10-08 14:20:24,581 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5800] (kafka_system_test_utils) 2012-10-08 14:20:26,304 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5700] (kafka_system_test_utils) 2012-10-08 14:20:27,819 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5900] (kafka_system_test_utils) 2012-10-08 14:20:29,287 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5800] (kafka_system_test_utils) 2012-10-08 14:20:30,702 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6000] (kafka_system_test_utils) 2012-10-08 14:20:32,326 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5900] (kafka_system_test_utils) 2012-10-08 14:20:33,820 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6100] (kafka_system_test_utils) 2012-10-08 14:20:35,362 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6000] (kafka_system_test_utils) 2012-10-08 14:20:36,967 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6200] (kafka_system_test_utils) 2012-10-08 14:20:38,466 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6100] (kafka_system_test_utils) 2012-10-08 14:20:39,981 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6300] (kafka_system_test_utils) 2012-10-08 14:20:41,569 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6200] (kafka_system_test_utils) 2012-10-08 14:20:42,582 - INFO - sleeping for 15s 2012-10-08 14:20:43,043 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6400] (kafka_system_test_utils) 2012-10-08 14:20:44,757 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6300] (kafka_system_test_utils) 2012-10-08 14:20:46,304 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6500] (kafka_system_test_utils) 2012-10-08 14:20:47,800 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6400] (kafka_system_test_utils) 2012-10-08 14:20:49,325 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6600] (kafka_system_test_utils) 2012-10-08 14:20:50,822 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6500] (kafka_system_test_utils) 2012-10-08 14:20:52,401 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6700] (kafka_system_test_utils) 2012-10-08 14:20:53,839 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6600] (kafka_system_test_utils) 2012-10-08 14:20:55,254 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6800] (kafka_system_test_utils) 2012-10-08 14:20:56,675 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6700] (kafka_system_test_utils) 2012-10-08 14:21:01,278 - INFO - status of backgroundProducerStopped : [True] (ReplicationUtils) 2012-10-08 14:21:02,279 - INFO - all producer threads completed (ReplicationUtils) 2012-10-08 14:21:02,279 - INFO - ====================================================== 2012-10-08 14:21:02,279 - INFO - starting consumer in the background 2012-10-08 14:21:02,280 - INFO - ====================================================== 2012-10-08 14:21:02,280 - INFO - starting console consumer (kafka_system_test_utils) 2012-10-08 14:21:04,289 - INFO - starting metrics collection on jmx port : 9999 (metrics) 2012-10-08 14:21:09,583 - INFO - starting console consumer (kafka_system_test_utils) 2012-10-08 14:21:11,591 - INFO - starting metrics collection on jmx port : 9099 (metrics) 2012-10-08 14:21:17,399 - INFO - sleeping for 10s 2012-10-08 14:21:27,405 - INFO - ====================================================== 2012-10-08 14:21:27,405 - INFO - stopping all entities 2012-10-08 14:21:27,405 - INFO - ====================================================== 2012-10-08 14:21:37,386 - INFO - ================================================ 2012-10-08 14:21:37,386 - INFO - collecting logs from remote machines 2012-10-08 14:21:37,386 - INFO - ================================================ 2012-10-08 14:21:43,945 - INFO - ====================================================== 2012-10-08 14:21:43,945 - INFO - validating data matched 2012-10-08 14:21:43,946 - INFO - ====================================================== 2012-10-08 14:21:44,027 - INFO - no. of unique messages on topic [test_1] sent from publisher : 6900 (kafka_system_test_utils) 2012-10-08 14:21:44,027 - INFO - no. of unique messages on topic [test_1] received by consumer : 6899 (kafka_system_test_utils) 2012-10-08 14:21:44,028 - INFO - See /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/console_consumer-6/msg_id_missing_in_consumer.log for missing MessageID (kafka_system_test_utils) 2012-10-08 14:21:44,105 - INFO - no. of unique messages on topic [test_2] sent from publisher : 6800 (kafka_system_test_utils) 2012-10-08 14:21:44,105 - INFO - no. of unique messages on topic [test_2] received by consumer : 6800 (kafka_system_test_utils) 2012-10-08 14:21:44,106 - INFO - ================================================ 2012-10-08 14:21:44,106 - INFO - validating broker log segment checksums 2012-10-08 14:21:44,106 - INFO - ================================================ 2012-10-08 14:22:06,289 - INFO - log segment files checksum : {u'broker-1': {'test_1-0/00000000000000000000.kafka': '4049600389', 'test_1-0/00000000000000512040.kafka': '34914238', 'test_1-0/00000000000001024080.kafka': '367122128', 'test_1-1/00000000000000000000.kafka': '631002584', 'test_1-1/00000000000000512040.kafka': '1232167070', 'test_1-1/00000000000001024080.kafka': '2986223701', 'test_1-2/00000000000000000000.kafka': '1635415488', 'test_1-2/00000000000000512040.kafka': '118696972', 'test_1-2/00000000000001024080.kafka': '2635972356', 'test_2-0/00000000000000000000.kafka': '3659299603', 'test_2-0/00000000000000512040.kafka': '849686671', 'test_2-0/00000000000001024080.kafka': '3792772397', 'test_2-1/00000000000000000000.kafka': '2642867470', 'test_2-1/00000000000000512040.kafka': '34687406', 'test_2-1/00000000000001024080.kafka': '3754640895', 'test_2-2/00000000000000000000.kafka': '3379369987', 'test_2-2/00000000000000512040.kafka': '3294467716', 'test_2-2/00000000000001024080.kafka': '651773651'}, u'broker-2': {'test_1-0/00000000000000000000.kafka': '4049600389', 'test_1-0/00000000000000512040.kafka': '34914238', 'test_1-0/00000000000001024080.kafka': '367122128', 'test_1-1/00000000000000000000.kafka': '631002584', 'test_1-1/00000000000000512040.kafka': '1232167070', 'test_1-1/00000000000001024080.kafka': '2986223701', 'test_1-2/00000000000000000000.kafka': '1635415488', 'test_1-2/00000000000000512040.kafka': '118696972', 'test_1-2/00000000000001024080.kafka': '2635972356', 'test_2-0/00000000000000000000.kafka': '3659299603', 'test_2-0/00000000000000512040.kafka': '849686671', 'test_2-0/00000000000001024080.kafka': '3792772397', 'test_2-1/00000000000000000000.kafka': '2642867470', 'test_2-1/00000000000000512040.kafka': '34687406', 'test_2-1/00000000000001024080.kafka': '3754640895', 'test_2-2/00000000000000000000.kafka': '3379369987', 'test_2-2/00000000000000512040.kafka': '3294467716', 'test_2-2/00000000000001024080.kafka': '651773651'}, u'broker-3': {'test_1-0/00000000000000000000.kafka': '4049600389', 'test_1-0/00000000000000512040.kafka': '34914238', 'test_1-0/00000000000001024080.kafka': '367122128', 'test_1-1/00000000000000000000.kafka': '631002584', 'test_1-1/00000000000000512040.kafka': '1232167070', 'test_1-1/00000000000001024080.kafka': '2986223701', 'test_1-2/00000000000000000000.kafka': '1635415488', 'test_1-2/00000000000000512040.kafka': '118696972', 'test_1-2/00000000000001024080.kafka': '2635972356', 'test_2-0/00000000000000000000.kafka': '3659299603', 'test_2-0/00000000000000512040.kafka': '849686671', 'test_2-0/00000000000001024080.kafka': '3792772397', 'test_2-1/00000000000000000000.kafka': '2642867470', 'test_2-1/00000000000000512040.kafka': '34687406', 'test_2-1/00000000000001024080.kafka': '3754640895', 'test_2-2/00000000000000000000.kafka': '3379369987', 'test_2-2/00000000000000512040.kafka': '3294467716', 'test_2-2/00000000000001024080.kafka': '651773651'}} 2012-10-08 14:22:06,497 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,497 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,498 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,500 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:22:06,619 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,620 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,621 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:22:06,655 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,656 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,656 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:22:06,693 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,693 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,694 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,696 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:22:06,730 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,731 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,732 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:22:06,767 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,768 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,768 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:22:06,805 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,806 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,806 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,808 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:22:06,842 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,843 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,844 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:22:06,878 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,879 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:06,880 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:22:07,822 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.server:type=ControllerStat,name=LeaderElectionRateAndTimeMs:OneMinuteRate.svg: line contains NULL byte (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: line contains NULL byte 2012-10-08 14:22:07,859 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,860 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,861 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-3/metrics/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs.csv: There should be exactly one column that matches attribute: kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:OneMinuteRate in headers: time,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:50thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:75thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:95thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:98thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:999thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:99thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:Count,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:EventType1349731104336 (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 111, in plot_graphs attributeColumnIndex = ensure_valid_headers(headers, attribute) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 86, in ensure_valid_headers " headers: {0}".format(",".join(headers))) Exception: There should be exactly one column that matches attribute: kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:OneMinuteRate in headers: time,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:50thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:75thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:95thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:98thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:999thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:99thPercentile,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:Count,kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:EventType1349731104336 2012-10-08 14:22:07,863 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:22:07,898 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,899 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,899 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-3/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,901 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:22:07,937 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,938 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,939 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-3/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,941 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:22:07,975 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-1/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,975 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-2/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,976 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/broker-3/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:22:07,978 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:22:09,000 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/producer_performance-4/metrics/kafka.producer:type=ProducerRequestStat,name=ProduceRequestRateAndTimeMs.csv: pop from empty list (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 110, in plot_graphs headers = csv_reader.pop(0) IndexError: pop from empty list 2012-10-08 14:22:09,000 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/producer_performance-5/metrics/kafka.producer:type=ProducerRequestStat,name=ProduceRequestRateAndTimeMs.csv: pop from empty list (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 110, in plot_graphs headers = csv_reader.pop(0) IndexError: pop from empty list 2012-10-08 14:22:09,000 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/producer_performance/kafka.producer:type=ProducerRequestStat,name=ProduceRequestRateAndTimeMs:OneMinuteRate.svg: min() arg is an empty sequence (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 128, in plot_graphs xmin = min(map(lambda coord: coord.x, coordinates)) ValueError: min() arg is an empty sequence 2012-10-08 14:22:09,153 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/producer_performance-4/metrics/kafka.producer:type=ProducerRequestStat,name=ProducerRequestSize.csv: There should be exactly one column that matches attribute: kafka.producer:type=ProducerRequestStat,name=ProducerRequestSize:Mean in headers: time (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 111, in plot_graphs attributeColumnIndex = ensure_valid_headers(headers, attribute) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 86, in ensure_valid_headers " headers: {0}".format(",".join(headers))) Exception: There should be exactly one column that matches attribute: kafka.producer:type=ProducerRequestStat,name=ProducerRequestSize:Mean in headers: time 2012-10-08 14:22:09,153 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/logs/producer_performance-5/metrics/kafka.producer:type=ProducerRequestStat,name=ProducerRequestSize.csv: pop from empty list (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 110, in plot_graphs headers = csv_reader.pop(0) IndexError: pop from empty list 2012-10-08 14:22:09,153 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0122/dashboards/producer_performance/kafka.producer:type=ProducerRequestStat,name=ProducerRequestSize:Mean.svg: min() arg is an empty sequence (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 128, in plot_graphs xmin = min(map(lambda coord: coord.x, coordinates)) ValueError: min() arg is an empty sequence 2012-10-08 14:22:09,166 - INFO - ====================================================== 2012-10-08 14:22:09,166 - INFO - stopping all entities - please wait ... 2012-10-08 14:22:09,166 - INFO - ====================================================== ======================================================================================= Test Case Name : testcase_0123 ======================================================================================= Description : 01 : Leader Failure in Replication with multi topics & partitions : 1. mode => async; 2. comp => 0 02 : Produce and consume messages to 2 topics - 3 partitions 03 : This test sends messages to 3 replicas 04 : To trigger leader election: find the leader and terminate by controlled failure (kill -15) 05 : Restart the terminated broker 06 : Lookup brokers' log4j messages and verify that leader is re-elected successfully 07 : At the end it verifies the log size and contents 08 : Use a consumer to verify no message loss. 09 : Producer dimensions : mode:async, acks:1, comp:1 10 : Log segment size : 512000 ======================================================================================= Test Case Args : bounce_leader : true sleep_seconds_between_producer_calls : 1 num_iteration : 3 num_partition : 3 message_producing_free_time_sec : 15 replica_factor : 3 ======================================================================================= 2012-10-08 14:24:32,209 - INFO - found a new cluster_config : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/cluster_config.json (system_test_utils) 2012-10-08 14:24:32,211 - INFO - ================================================= 2012-10-08 14:24:32,211 - INFO - setting up remote hosts ... 2012-10-08 14:24:32,211 - INFO - ================================================= 2012-10-08 14:24:38,477 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:24:41,152 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:24:44,220 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:24:46,586 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:24:48,954 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:24:48,954 - INFO - skipping role [producer_performance] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:24:48,954 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:24:48,955 - INFO - skipping role [producer_performance] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:24:48,955 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:24:48,955 - INFO - skipping role [console_consumer] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:24:48,955 - INFO - cleaning up data dir on host: [localhost] (kafka_system_test_utils) 2012-10-08 14:24:48,955 - INFO - skipping role [console_consumer] on host : [localhost] (kafka_system_test_utils) 2012-10-08 14:24:51,962 - INFO - calling generate_properties_files (kafka_system_test_utils) 2012-10-08 14:24:51,963 - INFO - config template (source) pathname : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/config (kafka_system_test_utils) 2012-10-08 14:24:51,963 - INFO - testcase config (dest) pathname : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/config (kafka_system_test_utils) 2012-10-08 14:24:55,101 - INFO - ====================================================== 2012-10-08 14:24:55,101 - INFO - starting zookeepers 2012-10-08 14:24:55,101 - INFO - ====================================================== 2012-10-08 14:24:57,729 - INFO - starting zookeeper in host [localhost] on client port [2188] (kafka_system_test_utils) 2012-10-08 14:25:04,036 - INFO - starting metrics collection on jmx port : 9990 (metrics) 2012-10-08 14:25:05,449 - INFO - sleeping for 2s 2012-10-08 14:25:07,451 - INFO - ====================================================== 2012-10-08 14:25:07,451 - INFO - starting brokers 2012-10-08 14:25:07,451 - INFO - ====================================================== 2012-10-08 14:25:07,452 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:25:14,051 - INFO - starting metrics collection on jmx port : 9991 (metrics) 2012-10-08 14:25:41,073 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:25:47,456 - INFO - starting metrics collection on jmx port : 9992 (metrics) 2012-10-08 14:26:14,584 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:26:20,987 - INFO - starting metrics collection on jmx port : 9993 (metrics) 2012-10-08 14:26:48,788 - INFO - sleeping for 5s 2012-10-08 14:26:53,790 - INFO - ====================================================== 2012-10-08 14:26:53,790 - INFO - creating topics 2012-10-08 14:26:53,790 - INFO - ====================================================== 2012-10-08 14:26:53,791 - INFO - creating topic: [test_1] at: [localhost:2188] (kafka_system_test_utils) 2012-10-08 14:26:53,800 - INFO - creating topic: [test_2] at: [localhost:2188] (kafka_system_test_utils) 2012-10-08 14:26:53,807 - INFO - sleeping for 5s 2012-10-08 14:26:58,807 - INFO - ====================================================== 2012-10-08 14:26:58,808 - INFO - starting producer in the background 2012-10-08 14:26:58,808 - INFO - ====================================================== 2012-10-08 14:26:58,809 - INFO - starting producer preformance (kafka_system_test_utils) 2012-10-08 14:26:59,809 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [0] (kafka_system_test_utils) 2012-10-08 14:27:00,810 - INFO - starting metrics collection on jmx port : 9997 (metrics) 2012-10-08 14:27:02,579 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [100] (kafka_system_test_utils) 2012-10-08 14:27:03,581 - INFO - starting producer preformance (kafka_system_test_utils) 2012-10-08 14:27:05,229 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [0] (kafka_system_test_utils) 2012-10-08 14:27:06,230 - INFO - starting metrics collection on jmx port : 9998 (metrics) 2012-10-08 14:27:06,915 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [200] (kafka_system_test_utils) 2012-10-08 14:27:08,557 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [100] (kafka_system_test_utils) 2012-10-08 14:27:09,013 - INFO - sleeping for 15 sec to produce some messages 2012-10-08 14:27:10,210 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [300] (kafka_system_test_utils) 2012-10-08 14:27:11,891 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [200] (kafka_system_test_utils) 2012-10-08 14:27:13,403 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [400] (kafka_system_test_utils) 2012-10-08 14:27:14,914 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [300] (kafka_system_test_utils) 2012-10-08 14:27:16,405 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [500] (kafka_system_test_utils) 2012-10-08 14:27:18,144 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [400] (kafka_system_test_utils) 2012-10-08 14:27:19,726 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [600] (kafka_system_test_utils) 2012-10-08 14:27:21,137 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [500] (kafka_system_test_utils) 2012-10-08 14:27:22,812 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [700] (kafka_system_test_utils) 2012-10-08 14:27:24,014 - INFO - ====================================================== 2012-10-08 14:27:24,014 - INFO - Iteration 1 of 3 2012-10-08 14:27:24,014 - INFO - ====================================================== 2012-10-08 14:27:24,015 - INFO - ====================================================== 2012-10-08 14:27:24,015 - INFO - looking up leader 2012-10-08 14:27:24,015 - INFO - ====================================================== 2012-10-08 14:27:24,558 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [600] (kafka_system_test_utils) 2012-10-08 14:27:25,368 - INFO - ====================================================== 2012-10-08 14:27:25,368 - INFO - validating leader election 2012-10-08 14:27:25,368 - INFO - ====================================================== 2012-10-08 14:27:25,368 - INFO - found leader in entity [3] with brokerid [3] for partition [2] (kafka_system_test_utils) 2012-10-08 14:27:25,369 - INFO - ====================================================== 2012-10-08 14:27:25,369 - INFO - bounce_leader flag : true 2012-10-08 14:27:25,369 - INFO - ====================================================== 2012-10-08 14:27:25,369 - INFO - stopping leader in entity 3 with pid 28736 (kafka_system_test_utils) 2012-10-08 14:27:25,904 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:27:26,189 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [800] (kafka_system_test_utils) 2012-10-08 14:27:27,745 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [700] (kafka_system_test_utils) 2012-10-08 14:27:29,269 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [900] (kafka_system_test_utils) 2012-10-08 14:27:30,660 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [800] (kafka_system_test_utils) 2012-10-08 14:27:32,009 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1000] (kafka_system_test_utils) 2012-10-08 14:27:33,390 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [900] (kafka_system_test_utils) 2012-10-08 14:27:34,768 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1100] (kafka_system_test_utils) 2012-10-08 14:27:35,915 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:27:36,187 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1000] (kafka_system_test_utils) 2012-10-08 14:27:37,300 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:27:38,653 - INFO - leader Re-election Latency: 0.324000120163 sec (kafka_system_test_utils) 2012-10-08 14:27:38,654 - INFO - ====================================================== 2012-10-08 14:27:38,654 - INFO - starting the previously terminated broker 2012-10-08 14:27:38,654 - INFO - ====================================================== 2012-10-08 14:27:38,655 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:27:39,885 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1200] (kafka_system_test_utils) 2012-10-08 14:27:41,404 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1100] (kafka_system_test_utils) 2012-10-08 14:27:42,764 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1300] (kafka_system_test_utils) 2012-10-08 14:27:44,216 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1200] (kafka_system_test_utils) 2012-10-08 14:27:45,290 - INFO - starting metrics collection on jmx port : 9993 (metrics) 2012-10-08 14:27:45,775 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1400] (kafka_system_test_utils) 2012-10-08 14:27:47,443 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1300] (kafka_system_test_utils) 2012-10-08 14:27:48,932 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1500] (kafka_system_test_utils) 2012-10-08 14:27:50,303 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1400] (kafka_system_test_utils) 2012-10-08 14:27:51,855 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1600] (kafka_system_test_utils) 2012-10-08 14:27:53,449 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1500] (kafka_system_test_utils) 2012-10-08 14:27:54,945 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1700] (kafka_system_test_utils) 2012-10-08 14:27:56,414 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1600] (kafka_system_test_utils) 2012-10-08 14:27:57,819 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1800] (kafka_system_test_utils) 2012-10-08 14:27:59,269 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1700] (kafka_system_test_utils) 2012-10-08 14:28:00,860 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1900] (kafka_system_test_utils) 2012-10-08 14:28:02,477 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1800] (kafka_system_test_utils) 2012-10-08 14:28:04,174 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2000] (kafka_system_test_utils) 2012-10-08 14:28:05,777 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [1900] (kafka_system_test_utils) 2012-10-08 14:28:07,433 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2100] (kafka_system_test_utils) 2012-10-08 14:28:08,902 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2000] (kafka_system_test_utils) 2012-10-08 14:28:10,581 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2200] (kafka_system_test_utils) 2012-10-08 14:28:12,189 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2100] (kafka_system_test_utils) 2012-10-08 14:28:13,771 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2300] (kafka_system_test_utils) 2012-10-08 14:28:14,601 - INFO - sleeping for 15s 2012-10-08 14:28:15,433 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2200] (kafka_system_test_utils) 2012-10-08 14:28:17,026 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2400] (kafka_system_test_utils) 2012-10-08 14:28:18,537 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2300] (kafka_system_test_utils) 2012-10-08 14:28:19,977 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2500] (kafka_system_test_utils) 2012-10-08 14:28:21,434 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2400] (kafka_system_test_utils) 2012-10-08 14:28:23,210 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2600] (kafka_system_test_utils) 2012-10-08 14:28:24,652 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2500] (kafka_system_test_utils) 2012-10-08 14:28:26,214 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2700] (kafka_system_test_utils) 2012-10-08 14:28:27,650 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2600] (kafka_system_test_utils) 2012-10-08 14:28:29,225 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2800] (kafka_system_test_utils) 2012-10-08 14:28:29,609 - INFO - ====================================================== 2012-10-08 14:28:29,610 - INFO - Iteration 2 of 3 2012-10-08 14:28:29,610 - INFO - ====================================================== 2012-10-08 14:28:29,610 - INFO - ====================================================== 2012-10-08 14:28:29,610 - INFO - looking up leader 2012-10-08 14:28:29,610 - INFO - ====================================================== 2012-10-08 14:28:30,836 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2700] (kafka_system_test_utils) 2012-10-08 14:28:31,279 - INFO - ====================================================== 2012-10-08 14:28:31,279 - INFO - validating leader election 2012-10-08 14:28:31,280 - INFO - ====================================================== 2012-10-08 14:28:31,280 - INFO - found leader in entity [2] with brokerid [2] for partition [1] (kafka_system_test_utils) 2012-10-08 14:28:31,280 - INFO - ====================================================== 2012-10-08 14:28:31,280 - INFO - bounce_leader flag : true 2012-10-08 14:28:31,280 - INFO - ====================================================== 2012-10-08 14:28:31,280 - INFO - stopping leader in entity 2 with pid 25630 (kafka_system_test_utils) 2012-10-08 14:28:31,767 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:28:33,372 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2900] (kafka_system_test_utils) 2012-10-08 14:28:35,974 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2800] (kafka_system_test_utils) 2012-10-08 14:28:37,357 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3000] (kafka_system_test_utils) 2012-10-08 14:28:39,076 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [2900] (kafka_system_test_utils) 2012-10-08 14:28:40,768 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3100] (kafka_system_test_utils) 2012-10-08 14:28:41,771 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:28:42,158 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3000] (kafka_system_test_utils) 2012-10-08 14:28:43,160 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:28:43,586 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3200] (kafka_system_test_utils) 2012-10-08 14:28:44,736 - INFO - leader Re-election Latency: 0.323999881744 sec (kafka_system_test_utils) 2012-10-08 14:28:44,736 - INFO - ====================================================== 2012-10-08 14:28:44,736 - INFO - starting the previously terminated broker 2012-10-08 14:28:44,736 - INFO - ====================================================== 2012-10-08 14:28:44,737 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:28:45,509 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3100] (kafka_system_test_utils) 2012-10-08 14:28:47,165 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3300] (kafka_system_test_utils) 2012-10-08 14:28:48,622 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3200] (kafka_system_test_utils) 2012-10-08 14:28:50,245 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3400] (kafka_system_test_utils) 2012-10-08 14:28:51,299 - INFO - starting metrics collection on jmx port : 9992 (metrics) 2012-10-08 14:28:51,980 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3300] (kafka_system_test_utils) 2012-10-08 14:28:53,407 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3500] (kafka_system_test_utils) 2012-10-08 14:28:54,908 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3400] (kafka_system_test_utils) 2012-10-08 14:28:56,536 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3600] (kafka_system_test_utils) 2012-10-08 14:28:58,095 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3500] (kafka_system_test_utils) 2012-10-08 14:28:59,734 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3700] (kafka_system_test_utils) 2012-10-08 14:29:01,319 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3600] (kafka_system_test_utils) 2012-10-08 14:29:03,011 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3800] (kafka_system_test_utils) 2012-10-08 14:29:04,741 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3700] (kafka_system_test_utils) 2012-10-08 14:29:06,318 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3900] (kafka_system_test_utils) 2012-10-08 14:29:07,773 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3800] (kafka_system_test_utils) 2012-10-08 14:29:09,148 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4000] (kafka_system_test_utils) 2012-10-08 14:29:10,898 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [3900] (kafka_system_test_utils) 2012-10-08 14:29:12,532 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4100] (kafka_system_test_utils) 2012-10-08 14:29:13,950 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4000] (kafka_system_test_utils) 2012-10-08 14:29:15,433 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4200] (kafka_system_test_utils) 2012-10-08 14:29:17,529 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4100] (kafka_system_test_utils) 2012-10-08 14:29:19,009 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4300] (kafka_system_test_utils) 2012-10-08 14:29:20,515 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4200] (kafka_system_test_utils) 2012-10-08 14:29:21,517 - INFO - sleeping for 15s 2012-10-08 14:29:22,095 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4400] (kafka_system_test_utils) 2012-10-08 14:29:23,807 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4300] (kafka_system_test_utils) 2012-10-08 14:29:25,212 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4500] (kafka_system_test_utils) 2012-10-08 14:29:26,876 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4400] (kafka_system_test_utils) 2012-10-08 14:29:28,439 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4600] (kafka_system_test_utils) 2012-10-08 14:29:29,862 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4500] (kafka_system_test_utils) 2012-10-08 14:29:31,293 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4700] (kafka_system_test_utils) 2012-10-08 14:29:32,826 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4600] (kafka_system_test_utils) 2012-10-08 14:29:34,450 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4800] (kafka_system_test_utils) 2012-10-08 14:29:35,842 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4700] (kafka_system_test_utils) 2012-10-08 14:29:36,523 - INFO - ====================================================== 2012-10-08 14:29:36,523 - INFO - Iteration 3 of 3 2012-10-08 14:29:36,523 - INFO - ====================================================== 2012-10-08 14:29:36,524 - INFO - ====================================================== 2012-10-08 14:29:36,524 - INFO - looking up leader 2012-10-08 14:29:36,524 - INFO - ====================================================== 2012-10-08 14:29:37,422 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4900] (kafka_system_test_utils) 2012-10-08 14:29:37,937 - INFO - ====================================================== 2012-10-08 14:29:37,937 - INFO - validating leader election 2012-10-08 14:29:37,937 - INFO - ====================================================== 2012-10-08 14:29:37,937 - INFO - found leader in entity [1] with brokerid [1] for partition [0] (kafka_system_test_utils) 2012-10-08 14:29:37,937 - INFO - ====================================================== 2012-10-08 14:29:37,937 - INFO - bounce_leader flag : true 2012-10-08 14:29:37,938 - INFO - ====================================================== 2012-10-08 14:29:37,938 - INFO - stopping leader in entity 1 with pid 22522 (kafka_system_test_utils) 2012-10-08 14:29:39,242 - INFO - sleeping for 10s for leader re-election to complete (kafka_system_test_utils) 2012-10-08 14:29:39,759 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4800] (kafka_system_test_utils) 2012-10-08 14:29:43,033 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5000] (kafka_system_test_utils) 2012-10-08 14:29:45,665 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [4900] (kafka_system_test_utils) 2012-10-08 14:29:48,402 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5100] (kafka_system_test_utils) 2012-10-08 14:29:49,247 - INFO - looking up broker shutdown... (kafka_system_test_utils) 2012-10-08 14:29:50,964 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5000] (kafka_system_test_utils) 2012-10-08 14:29:51,056 - INFO - looking up new leader (kafka_system_test_utils) 2012-10-08 14:29:53,046 - INFO - leader Re-election Latency: 0.328000068665 sec (kafka_system_test_utils) 2012-10-08 14:29:53,047 - INFO - ====================================================== 2012-10-08 14:29:53,047 - INFO - starting the previously terminated broker 2012-10-08 14:29:53,047 - INFO - ====================================================== 2012-10-08 14:29:53,047 - INFO - starting broker in host [localhost] on client port [] (kafka_system_test_utils) 2012-10-08 14:29:53,515 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5200] (kafka_system_test_utils) 2012-10-08 14:29:55,089 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5100] (kafka_system_test_utils) 2012-10-08 14:29:56,706 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5300] (kafka_system_test_utils) 2012-10-08 14:29:58,269 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5200] (kafka_system_test_utils) 2012-10-08 14:29:59,784 - INFO - starting metrics collection on jmx port : 9991 (metrics) 2012-10-08 14:30:00,524 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5400] (kafka_system_test_utils) 2012-10-08 14:30:02,154 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5300] (kafka_system_test_utils) 2012-10-08 14:30:03,741 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5500] (kafka_system_test_utils) 2012-10-08 14:30:05,213 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5400] (kafka_system_test_utils) 2012-10-08 14:30:07,164 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5600] (kafka_system_test_utils) 2012-10-08 14:30:08,678 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5500] (kafka_system_test_utils) 2012-10-08 14:30:10,293 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5700] (kafka_system_test_utils) 2012-10-08 14:30:11,892 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5600] (kafka_system_test_utils) 2012-10-08 14:30:13,543 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5800] (kafka_system_test_utils) 2012-10-08 14:30:15,218 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5700] (kafka_system_test_utils) 2012-10-08 14:30:16,630 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5900] (kafka_system_test_utils) 2012-10-08 14:30:18,214 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5800] (kafka_system_test_utils) 2012-10-08 14:30:19,660 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6000] (kafka_system_test_utils) 2012-10-08 14:30:21,169 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [5900] (kafka_system_test_utils) 2012-10-08 14:30:22,673 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6100] (kafka_system_test_utils) 2012-10-08 14:30:25,162 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6000] (kafka_system_test_utils) 2012-10-08 14:30:26,587 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6200] (kafka_system_test_utils) 2012-10-08 14:30:28,815 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6100] (kafka_system_test_utils) 2012-10-08 14:30:30,129 - INFO - sleeping for 15s 2012-10-08 14:30:30,619 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6300] (kafka_system_test_utils) 2012-10-08 14:30:32,044 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6200] (kafka_system_test_utils) 2012-10-08 14:30:33,741 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6400] (kafka_system_test_utils) 2012-10-08 14:30:35,237 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6300] (kafka_system_test_utils) 2012-10-08 14:30:36,750 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6500] (kafka_system_test_utils) 2012-10-08 14:30:38,199 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6400] (kafka_system_test_utils) 2012-10-08 14:30:40,042 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6600] (kafka_system_test_utils) 2012-10-08 14:30:41,500 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6500] (kafka_system_test_utils) 2012-10-08 14:30:42,997 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6700] (kafka_system_test_utils) 2012-10-08 14:30:44,643 - INFO - #### [producer thread] status of stopBackgroundProducer : [False] => producing [100] messages with starting message id : [6600] (kafka_system_test_utils) 2012-10-08 14:30:49,057 - INFO - status of backgroundProducerStopped : [True] (ReplicationUtils) 2012-10-08 14:30:50,059 - INFO - all producer threads completed (ReplicationUtils) 2012-10-08 14:30:50,059 - INFO - ====================================================== 2012-10-08 14:30:50,059 - INFO - starting consumer in the background 2012-10-08 14:30:50,059 - INFO - ====================================================== 2012-10-08 14:30:50,059 - INFO - starting console consumer (kafka_system_test_utils) 2012-10-08 14:30:52,070 - INFO - starting metrics collection on jmx port : 9999 (metrics) 2012-10-08 14:30:57,102 - INFO - starting console consumer (kafka_system_test_utils) 2012-10-08 14:30:59,115 - INFO - starting metrics collection on jmx port : 9099 (metrics) 2012-10-08 14:31:04,608 - INFO - sleeping for 10s 2012-10-08 14:31:14,615 - INFO - ====================================================== 2012-10-08 14:31:14,615 - INFO - stopping all entities 2012-10-08 14:31:14,615 - INFO - ====================================================== 2012-10-08 14:31:24,230 - INFO - ================================================ 2012-10-08 14:31:24,230 - INFO - collecting logs from remote machines 2012-10-08 14:31:24,230 - INFO - ================================================ 2012-10-08 14:31:31,156 - INFO - ====================================================== 2012-10-08 14:31:31,156 - INFO - validating data matched 2012-10-08 14:31:31,156 - INFO - ====================================================== 2012-10-08 14:31:31,241 - INFO - no. of unique messages on topic [test_1] sent from publisher : 6800 (kafka_system_test_utils) 2012-10-08 14:31:31,241 - INFO - no. of unique messages on topic [test_1] received by consumer : 6800 (kafka_system_test_utils) 2012-10-08 14:31:31,315 - INFO - no. of unique messages on topic [test_2] sent from publisher : 6700 (kafka_system_test_utils) 2012-10-08 14:31:31,315 - INFO - no. of unique messages on topic [test_2] received by consumer : 6695 (kafka_system_test_utils) 2012-10-08 14:31:31,315 - INFO - See /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/console_consumer-7/msg_id_missing_in_consumer.log for missing MessageID (kafka_system_test_utils) 2012-10-08 14:31:31,316 - INFO - ================================================ 2012-10-08 14:31:31,317 - INFO - validating broker log segment checksums 2012-10-08 14:31:31,317 - INFO - ================================================ 2012-10-08 14:31:38,996 - INFO - log segment files checksum : {u'broker-1': {'test_1-0/00000000000000000000.kafka': '691684159', 'test_1-1/00000000000000000000.kafka': '3681205309', 'test_1-2/00000000000000000000.kafka': '3571211770', 'test_2-0/00000000000000000000.kafka': '85299972', 'test_2-1/00000000000000000000.kafka': '4115697331', 'test_2-2/00000000000000000000.kafka': '3974179431'}, u'broker-2': {'test_1-0/00000000000000000000.kafka': '691684159', 'test_1-1/00000000000000000000.kafka': '3681205309', 'test_1-2/00000000000000000000.kafka': '3571211770', 'test_2-0/00000000000000000000.kafka': '85299972', 'test_2-1/00000000000000000000.kafka': '4115697331', 'test_2-2/00000000000000000000.kafka': '3974179431'}, u'broker-3': {'test_1-0/00000000000000000000.kafka': '691684159', 'test_1-1/00000000000000000000.kafka': '3681205309', 'test_1-2/00000000000000000000.kafka': '3571211770', 'test_2-0/00000000000000000000.kafka': '85299972', 'test_2-1/00000000000000000000.kafka': '4115697331', 'test_2-2/00000000000000000000.kafka': '3974179431'}} 2012-10-08 14:31:39,205 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-RequestsPerSec:OneMinuteRate.svg: line contains NULL byte (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: line contains NULL byte 2012-10-08 14:31:39,240 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,241 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,241 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-TotalTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:31:39,277 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,278 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,278 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Produce-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:31:39,315 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,316 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,317 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,319 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RequestsPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:31:39,355 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,356 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,356 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-TotalTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:31:39,391 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,392 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,392 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Consumer-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:31:39,430 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,431 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,432 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-3/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,434 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-RequestsPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:31:39,469 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,470 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,471 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-TotalTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:31:39,508 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,509 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:39,510 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.network:type=RequestMetrics,name=Fetch-Follower-RemoteTimeNs:Mean.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:31:40,463 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.server:type=ControllerStat,name=LeaderElectionRateAndTimeMs:OneMinuteRate.svg: line contains NULL byte (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: line contains NULL byte 2012-10-08 14:31:40,498 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:40,498 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.message:type=LogFlushStats,name=LogFlushRateAndTimeMs:OneMinuteRate.svg: newline inside string (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: newline inside string 2012-10-08 14:31:40,533 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec:OneMinuteRate.svg: line contains NULL byte (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 105, in plot_graphs csv_reader = list(csv.reader(open(inputCsvFile, "rb"))) Error: line contains NULL byte 2012-10-08 14:31:40,570 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:40,571 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:40,572 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-3/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:40,574 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:31:40,609 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-1/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:40,610 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-2/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:40,610 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/broker-3/metrics/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec.csv: list index out of range (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 115, in plot_graphs yVal = float(line[attributeColumnIndex]) IndexError: list index out of range 2012-10-08 14:31:40,612 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/broker/kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec:OneMinuteRate.svg: integer division or modulo by zero (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 140, in plot_graphs plt.legend(plots,labels, loc=2) File "/usr/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 2437, in legend ret = gca().legend(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/axes.py", line 4044, in legend self.legend_ = mlegend.Legend(self, handles, labels, **kwargs) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 304, in __init__ self._init_legend_box(handles, labels) File "/usr/lib64/python2.6/site-packages/matplotlib/legend.py", line 592, in _init_legend_box nrows, num_largecol = divmod(len(handleboxes), ncol) ZeroDivisionError: integer division or modulo by zero 2012-10-08 14:31:41,845 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/producer_performance-4/metrics/kafka.producer:type=ProducerRequestStat,name=ProduceRequestRateAndTimeMs.csv: pop from empty list (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 110, in plot_graphs headers = csv_reader.pop(0) IndexError: pop from empty list 2012-10-08 14:31:41,845 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/producer_performance-5/metrics/kafka.producer:type=ProducerRequestStat,name=ProduceRequestRateAndTimeMs.csv: pop from empty list (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 110, in plot_graphs headers = csv_reader.pop(0) IndexError: pop from empty list 2012-10-08 14:31:41,845 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/producer_performance/kafka.producer:type=ProducerRequestStat,name=ProduceRequestRateAndTimeMs:OneMinuteRate.svg: min() arg is an empty sequence (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 128, in plot_graphs xmin = min(map(lambda coord: coord.x, coordinates)) ValueError: min() arg is an empty sequence 2012-10-08 14:31:41,882 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/producer_performance-4/metrics/kafka.producer:type=ProducerRequestStat,name=ProducerRequestSize.csv: pop from empty list (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 110, in plot_graphs headers = csv_reader.pop(0) IndexError: pop from empty list 2012-10-08 14:31:41,882 - ERROR - ERROR while plotting data for /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/logs/producer_performance-5/metrics/kafka.producer:type=ProducerRequestStat,name=ProducerRequestSize.csv: pop from empty list (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 110, in plot_graphs headers = csv_reader.pop(0) IndexError: pop from empty list 2012-10-08 14:31:41,883 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_0123/dashboards/producer_performance/kafka.producer:type=ProducerRequestStat,name=ProducerRequestSize:Mean.svg: min() arg is an empty sequence (metrics) Traceback (most recent call last): File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role "time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile) File "/home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/utils/metrics.py", line 128, in plot_graphs xmin = min(map(lambda coord: coord.x, coordinates)) ValueError: min() arg is an empty sequence 2012-10-08 14:31:41,895 - INFO - ====================================================== 2012-10-08 14:31:41,895 - INFO - stopping all entities - please wait ... 2012-10-08 14:31:41,895 - INFO - ====================================================== 2012-10-08 14:35:26,261 - INFO - ====================================================== 2012-10-08 14:35:26,262 - INFO - Skipping : testcase_1 2012-10-08 14:35:26,262 - INFO - ====================================================== 2012-10-08 14:35:26,262 - INFO - found a testsuite : mirror_maker_testsuite (system_test_runner) 2012-10-08 14:35:26,262 - INFO - found a new cluster_config : /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/mirror_maker_testsuite/cluster_config.json (system_test_utils) 2012-10-08 14:35:26,266 - INFO - ================================================= 2012-10-08 14:35:26,266 - INFO - setting up remote hosts ... 2012-10-08 14:35:26,266 - INFO - ================================================= 2012-10-08 14:35:36,148 - INFO - found a test module file : mirror_maker_test.py (system_test_runner) 2012-10-08 14:35:36,163 - INFO - ================================================= 2012-10-08 14:35:36,163 - INFO - Running Test for : 2012-10-08 14:35:36,163 - INFO - suite : mirror_maker_testsuite 2012-10-08 14:35:36,164 - INFO - module : mirror_maker_test 2012-10-08 14:35:36,164 - INFO - class : MirrorMakerTest 2012-10-08 14:35:36,164 - INFO - ================================================= 2012-10-08 14:35:36,171 - INFO - ====================================================== 2012-10-08 14:35:36,172 - INFO - Skipping : testcase_5001 2012-10-08 14:35:36,172 - INFO - ====================================================== ======================================================== TEST REPORTS ======================================================== _test_case_name : testcase_0001 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0002 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0003 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0004 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0005 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0006 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0007 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0008 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0009 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0010 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0021 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0022 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0023 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0101 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0102 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0103 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0104 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0105 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0106 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0107 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0108 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0109 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0110 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0111 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0112 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0113 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0114 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0115 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0116 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0117 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0118 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_0121 _test_class_name : ReplicaBasicTest arg : bounce_leader : true arg : message_producing_free_time_sec : 15 arg : num_iteration : 3 arg : num_partition : 3 arg : replica_factor : 3 arg : sleep_seconds_between_producer_calls : 1 validation_status : Leader Election Latency - iter 1 brokerid 3 : 352.00 ms Leader Election Latency - iter 2 brokerid 1 : 326.00 ms Leader Election Latency - iter 3 brokerid 2 : 334.00 ms Leader Election Latency MAX : 352.00 Leader Election Latency MIN : 326.00 Log segment checksum matching across all replicas : PASSED Unique messages from consumer on [test_1] : 1100 Unique messages from consumer on [test_2] : 1000 Unique messages from producer on [test_1] : 1100 Unique messages from producer on [test_2] : 1000 Validate for data matched on topic [test_1] : PASSED Validate for data matched on topic [test_2] : PASSED Validate leader election successful : PASSED ======================================================== _test_case_name : testcase_0122 _test_class_name : ReplicaBasicTest arg : bounce_leader : true arg : message_producing_free_time_sec : 15 arg : num_iteration : 3 arg : num_partition : 3 arg : replica_factor : 3 arg : sleep_seconds_between_producer_calls : 1 validation_status : Leader Election Latency - iter 1 brokerid 3 : 325.00 ms Leader Election Latency - iter 2 brokerid 1 : 323.00 ms Leader Election Latency - iter 3 brokerid 2 : 325.00 ms Leader Election Latency MAX : 325.00 Leader Election Latency MIN : 323.00 Log segment checksum matching across all replicas : PASSED Unique messages from consumer on [test_1] : 6899 Unique messages from consumer on [test_2] : 6800 Unique messages from producer on [test_1] : 6900 Unique messages from producer on [test_2] : 6800 Validate for data matched on topic [test_1] : FAILED Validate for data matched on topic [test_2] : PASSED Validate leader election successful : PASSED ======================================================== _test_case_name : testcase_0123 _test_class_name : ReplicaBasicTest arg : bounce_leader : true arg : message_producing_free_time_sec : 15 arg : num_iteration : 3 arg : num_partition : 3 arg : replica_factor : 3 arg : sleep_seconds_between_producer_calls : 1 validation_status : Leader Election Latency - iter 1 brokerid 3 : 324.00 ms Leader Election Latency - iter 2 brokerid 2 : 324.00 ms Leader Election Latency - iter 3 brokerid 1 : 328.00 ms Leader Election Latency MAX : 328.00 Leader Election Latency MIN : 324.00 Log segment checksum matching across all replicas : PASSED Unique messages from consumer on [test_1] : 6800 Unique messages from consumer on [test_2] : 6695 Unique messages from producer on [test_1] : 6800 Unique messages from producer on [test_2] : 6700 Validate for data matched on topic [test_1] : PASSED Validate for data matched on topic [test_2] : FAILED Validate leader election successful : PASSED ======================================================== _test_case_name : testcase_1 _test_class_name : ReplicaBasicTest validation_status : ======================================================== _test_case_name : testcase_5001 _test_class_name : MirrorMakerTest validation_status : ========================================================