[KAFKA-440] Create a regression test framework for distributed environment testing

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Labels: replication-testing
• Assignee: John Fung
• Reporter: John Fung
• Created: 06/Aug/12 17:43
• Updated: 25/Aug/12 14:53
• Resolved: 22/Aug/12 17:17

Description

Initial requirements:

1. The whole test framework is preferably coded in Python (a common scripting language which has well supported features)

2. Test framework driver should be generic (distributed environment can be local host)

3. Test framework related configurations are defined in JSON format

4. Test environment, suite, case definitions may be defined in the following levels:
4-a entity_id is used as a key for looking up related config from different levels

4-b Cluster level defines: entity_id, hostname, kafka_home, java_home, ...

4-c Test suite / case level defines:
4-c-1 zookeeper: entity_id, clientPort, dataDir, log_filename, config_filename
4-c-2 broker: entity_id, port, log.file.size, log.dir, log_filename, config_filename
4-c-3 producer: entity_id, topic, threads, compression-codec, message-size, log_filename, config_filename

Issue Links

• is blocked by: KAFKA-203 Improve Kafka internal metrics

Sub-Tasks

• Simple message replication test: Closed
• Simple message replication test with varied producer acks: Closed
• Simple message replication test with multiple partitions: Closed
• Simple message replication test with multiple partitions and log segments: Closed
• Message replication with failures: Closed
• Message replication with failures and varied producer acks: Closed
• Message replication with failures and multiple partitions and log segments: Closed
• Message replication with failures and varied replication factor: Closed
• Leader election test: Closed
• Test in sync replica changes due to slow or stuck followers: Closed
• Refactor and optimize system tests: Resolved
• broker-list in testcase_1_properties.json should be picked up by the framework automatically: Resolved
• Move start_consumer & start_producer inside "start_entity_in_background": Resolved
• Make producer to run for the entire duration of the System Test: Resolved
• The system test is preferred to run out of the box without the need to update cluster_config.json: Resolved
• Move log settings in the beginning of system_test_runner.py to an external log properties file: Resolved
• Support MacOS for this test framework: Resolved
• Rethrow exceptions to top level as much as possible: Resolved
• Make sure all background running processes are terminated properly: Resolved
• Port Mirroring System Test to this python system test framework: Resolved
• Sometimes the python system test framework doesn't terminate all running processes: Resolved
• Relative paths should be used for svg URLs in dashboards html: Resolved
• Simplify setup / initialization in replication_basic_test.py: Resolved
• Support "testcase_to_run" or "testcase_to_skip": Resolved
• Shut down ZK last to avoid hanging brokers running processes: Resolved
• Add more test cases to System Test: Closed
• Add documentation for system tests: Resolved