Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It would be great to get a comprehensive test suite that we could run against clients to certify them.

      The first step here would be work out a design approach that makes it easy to certify the correctness of a client.

        Activity

        Jay Kreps created issue -
        Hide
        Jay Kreps added a comment -

        Thread is here: http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201111.mbox/%3CCAOeJiJjRHB2c1T-OCre-ZwpDhg+HkVBo2TMyVaYKXsTvEL9PnQ@mail.gmail.com%3E

        Some thoughts:

        • It would be good to break out specific features to get a level of compliance: producer, compression types, consumer, etc
        • It would be good to test clients against various protocol versions to test forwards and backwards compatibility
        Show
        Jay Kreps added a comment - Thread is here: http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201111.mbox/%3CCAOeJiJjRHB2c1T-OCre-ZwpDhg+HkVBo2TMyVaYKXsTvEL9PnQ@mail.gmail.com%3E Some thoughts: It would be good to break out specific features to get a level of compliance: producer, compression types, consumer, etc It would be good to test clients against various protocol versions to test forwards and backwards compatibility
        Hide
        Jeffrey Damick added a comment -

        I'd like to see packet dumps of all the message types (from all versions) so that they can be used int unit tests of the clients and with the packet dumps that definition should include information about what to expect inside the data once it has been parsed, for example the CRC value, the value of a message that is compressed. Then the implementation can assert that it is in fact parsing and generating the packets correctly. These could be encapsulated into a test harness or test mode of the broker that the client could run a full integration test against as well.

        Show
        Jeffrey Damick added a comment - I'd like to see packet dumps of all the message types (from all versions) so that they can be used int unit tests of the clients and with the packet dumps that definition should include information about what to expect inside the data once it has been parsed, for example the CRC value, the value of a message that is compressed. Then the implementation can assert that it is in fact parsing and generating the packets correctly. These could be encapsulated into a test harness or test mode of the broker that the client could run a full integration test against as well.
        Hide
        David Ormsbee added a comment -

        I like the packet dumps idea from a simplicity point of view. We could have a set of YAML files that represent types of compliance (e.g. simpleconsumer_0.7.yaml), maybe even put the message data in fields as base64 encoded strings. We don't have to worry about invoking clients on the command line – each client would just know to read the file and the file would specify what the raw data is and the intended results. It also saves us from having to bring up older versions of Kafka to live test against.

        That being said, is there any simple way to test a ZooKeeper Consumer/Producer? The only thing I can think of is to specify a standard command-line interface that clients need to implement, and then have a test script that spins up ZooKeeper and a few Kafka brokers, calls these client commands, and then inspects the output and the ZK state.

        Show
        David Ormsbee added a comment - I like the packet dumps idea from a simplicity point of view. We could have a set of YAML files that represent types of compliance (e.g. simpleconsumer_0.7.yaml), maybe even put the message data in fields as base64 encoded strings. We don't have to worry about invoking clients on the command line – each client would just know to read the file and the file would specify what the raw data is and the intended results. It also saves us from having to bring up older versions of Kafka to live test against. That being said, is there any simple way to test a ZooKeeper Consumer/Producer? The only thing I can think of is to specify a standard command-line interface that clients need to implement, and then have a test script that spins up ZooKeeper and a few Kafka brokers, calls these client commands, and then inspects the output and the ZK state.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jay Kreps
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development