Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-440

Create a regression test framework for distributed environment testing

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None

      Description

      Initial requirements:

      1. The whole test framework is preferably coded in Python (a common scripting language which has well supported features)

      2. Test framework driver should be generic (distributed environment can be local host)

      3. Test framework related configurations are defined in JSON format

      4. Test environment, suite, case definitions may be defined in the following levels:
      4-a entity_id is used as a key for looking up related config from different levels

      4-b Cluster level defines: entity_id, hostname, kafka_home, java_home, ...

      4-c Test suite / case level defines:
      4-c-1 zookeeper: entity_id, clientPort, dataDir, log_filename, config_filename
      4-c-2 broker: entity_id, port, log.file.size, log.dir, log_filename, config_filename
      4-c-3 producer: entity_id, topic, threads, compression-codec, message-size, log_filename, config_filename

        Attachments

        1. kafka-440-v1.patch
          64 kB
          John Fung
        2. kafka-440-v2.patch
          72 kB
          John Fung
        3. kafka-440-v3.patch
          84 kB
          John Fung
        4. kafka-440-v5.patch
          89 kB
          John Fung
        5. kafka-440-v6.patch
          93 kB
          John Fung
        6. kafka-440-v7.patch
          96 kB
          John Fung

          Issue Links

          1.
          Simple message replication test Sub-task Closed John Fung

          0%

          Original Estimate - 48h
          Remaining Estimate - 48h
          2.
          Simple message replication test with varied producer acks Sub-task Closed John Fung

          0%

          Original Estimate - 24h
          Remaining Estimate - 24h
          3.
          Simple message replication test with multiple partitions Sub-task Closed John Fung

          0%

          Original Estimate - 24h
          Remaining Estimate - 24h
          4.
          Simple message replication test with multiple partitions and log segments Sub-task Closed John Fung

          0%

          Original Estimate - 24h
          Remaining Estimate - 24h
          5.
          Message replication with failures Sub-task Closed John Fung

          0%

          Original Estimate - 48h
          Remaining Estimate - 48h
          6.
          Message replication with failures and varied producer acks Sub-task Closed John Fung

          0%

          Original Estimate - 24h
          Remaining Estimate - 24h
          7.
          Message replication with failures and multiple partitions and log segments Sub-task Closed John Fung

          0%

          Original Estimate - 24h
          Remaining Estimate - 24h
          8.
          Message replication with failures and varied replication factor Sub-task Closed John Fung

          0%

          Original Estimate - 48h
          Remaining Estimate - 48h
          9.
          Leader election test Sub-task Closed John Fung

          0%

          Original Estimate - 24h
          Remaining Estimate - 24h
          10.
          Test in sync replica changes due to slow or stuck followers Sub-task Closed John Fung

          0%

          Original Estimate - 72h
          Remaining Estimate - 72h
          11.
          Refactor and optimize system tests Sub-task Resolved Unassigned  
          12.
          broker-list in testcase_1_properties.json should be picked up by the framework automatically Sub-task Resolved John Fung  
          13.
          Move start_consumer & start_producer inside "start_entity_in_background" Sub-task Resolved John Fung  
          14.
          Make producer to run for the entire duration of the System Test Sub-task Resolved John Fung  
          15.
          The system test is preferred to run out of the box without the need to update cluster_config.json Sub-task Resolved John Fung  
          16.
          Move log settings in the beginning of system_test_runner.py to an external log properties file Sub-task Resolved John Fung  
          17.
          Support MacOS for this test framework Sub-task Resolved John Fung  
          18.
          Rethrow exceptions to top level as much as possible Sub-task Resolved John Fung  
          19.
          Make sure all background running processes are terminated properly Sub-task Resolved John Fung  
          20.
          Port Mirroring System Test to this python system test framework Sub-task Resolved John Fung  
          21.
          Sometimes the python system test framework doesn't terminate all running processes Sub-task Resolved John Fung  
          22.
          Relative paths should be used for svg URLs in dashboards html Sub-task Resolved Unassigned  
          23.
          Simplify setup / initialization in replication_basic_test.py Sub-task Resolved John Fung  
          24.
          Support "testcase_to_run" or "testcase_to_skip" Sub-task Resolved John Fung  
          25.
          Shut down ZK last to avoid hanging brokers running processes Sub-task Resolved John Fung  
          26.
          Add more test cases to System Test Sub-task Closed John Fung  
          27.
          Add documentation for system tests Sub-task Resolved Ewen Cheslack-Postava  

            Activity

              People

              • Assignee:
                jfung John Fung
                Reporter:
                jfung John Fung
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 360h
                  360h
                  Remaining:
                  Remaining Estimate - 360h
                  360h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified