Functionality for starting up the test cluster processes is split between start-impala-cluster.py, tests/common/impala_cluster.py and bin/start-*.d.sh. The shell scripts serve as an additional layer of indirection between the python code and the actual binaries that manipulates command-line flags and environment variables. I suspect this is a historical accident rather than design, since it would be just as easy to set environment variables and flags from Python code. Git history shows that the shell scripts were added first and start-impala-cluster.py was added as a layer on top.
This was all ok for a long time, but is now causing problems for running within docker, since we'd like to support some of the same functionality but we don't want to just shove those test scripts into our docker container (we want those containers to be close to what we'd run in a production environment).
The best thing to do would be to move as much logic as possible into Python helpers to construct env vars and classpaths and turn the shell scripts into minimalist wrappers. Some things could remain in the shell script, like set-classpath.sh,.