It would be helpful to refactor the system tests to make them as concise and
clear as possible, and get them to complete faster. Obviously this is
non-urgent but should ideally be done prior to adding new system tests if
1 - Several functions (e.g., wait_for_zero_consumer_lag, start_cluster,
etc.) are all util methods that are relevant across system tests. Also,
these util functions can be switched to getopt (preferrably with long
options) which will make the scripts more readable. Anything that is not
super-specific to any test should be moved to the global util script.
2 - Might help to have all test output and logs sent to a directory in tmp
(use mktemp) so everything is contained there - right now some files are
local and some are in tmp. Also, clean up would simply be rm -rf the temp
directory instead of handling multiple locations.
3 - We should get rid of as many sleeps as possible. E.g., with producer
acks I don't think we need to sleep after producer-performance. Likewise
for all other sleeps, let us see if they are really needed and provide
tooling (if necessary) to eliminate/reduce them.
4 - User-variable test parameters should be command-line options (getopt) to