Fixed in kafka-483-v1.patch

Uploaded kafka-483-v1.patch with the following fixed:

KAFKA-487 (Sub-Task 19) Make sure all background running processes are terminated properly

KAFKA-486 (Sub-Task 18) Rethrow exceptions to top level as much as possible

KAFKA-483 (Sub-Task 15) The system test is preferred to run out of the box without the need to update cluster_config.json

KAFKA-477 (Sub-Task 12) broker-list in testcase_1_properties.json should be picked up by the framework automatically

Uploaded kafka-483-v2.patch with the following changes:

1. Changes inherited from kafka-483-v1.patch: KAFKA-477, KAFKA-486, KAFKA-487

2. The system test will run out of the box on a single Linux machine without the need to update cluster_config.json (this JIRA: KAFKA-483)

3. In the beginning of a testcase, clean up logs, csv, svg and html files from previous test

4. Keep track of all pids (including JmxTool) and all running processes will be terminated properly at the end of the test

5. Display leader re-election latency (currently, this is the difference between the timestamp of server "shut down completed" and the timestamp of "leader state transition" from the log4j messages - it is not accurate but for temporary reference only)

6. Code cleanup

Re-open for patch review

Uploaded kafka-483-v3.patch with the following changes:

1. Rebased from latest 0.8 branch which has kafka-489-v2.patch committed.

2. Changes inherited from kafka-483-v1.patch: KAFKA-477, KAFKA-486, KAFKA-487

3. The system test will run out of the box on a single Linux machine without the need to update cluster_config.json (this JIRA: KAFKA-483)

4. In the beginning of a testcase, clean up logs, csv, svg and html files from previous test

5. Keep track of all pids (including JmxTool) and all running processes will be terminated properly at the end of the test

6. Display leader re-election latency (currently, this is the difference between the timestamp of server "shut down completed" and the timestamp of "leader state transition" from the log4j messages - it is not accurate but for temporary reference only)

7. Code cleanup

Thanks for the patch. Overall, it looks great !

A couple of minor comments -

1. Let's print the leader election latency at the end of the test run. Right now, it's getting lost amongst the other log messages.
2. How about printing the full path to the java home that the script picks up by default ? Right now, it strips off bin/java, so it is a little confusing to read.

Thanks for fixing the matplotlib usage to remove the dependency on Xserver.

Thanks for reviewing, Neha.

Uploaded kafka-483-v4.patch with the following changes:

1. Display the Java binary full path name instead of the JAVA_HOME at the beginning of test setup

2. The leader election latency is added to the dictionary as testcaseEnv.validationStatusDict["Leader Election Latency"] such that it will be reported at end of the test.

+1. Thanks for the latest patch!

Committed patch v4