Here's a thought: summary of my objections are:
1. introduces maintenance and manual discovery of updated image ids
2. not a well documented problem (ex. did this happen during a jclouds upgrade or really randomly)
3. doesn't cover the inputs that can produce problems (ex. location+image+hardware+options such as spot pricing are better)
I think we can get more data about change when it happens, and also have an additional benefit of documentation of what was last tested. I think we can accomplish the goal of understanding change without preventing it or making change very manual.
Apologies for not offering a solution before, that was lazy of me how's this?
We can create a code helper to check template values against last tested file before running a test. When the values change, warn and overwrite the file. On some flag, forcibly use old values.
In this case, we should get our documentation automatically, only implying a check-in. We also don't need to discover new ids as they will come in automatically. Finally, on error, testing is easy as you just run with the flag that uses last tested.
(note I don't care json vs yaml)
use basedir to establish lasttested directory (ex. services/cassandra)
serialize inputs to templateBuilder to a string and lookup its corresponding json file. (ex. gogrid/default.json or aws-ec2/ubuntu-10.04.json)
build the template and make a map of ids, check this vs what's in that file
warn me that we last tested something different
map = fromJson(lastTested)
template = templateBuilder.imageId(map.get("imageId")...
// serialize to disk
Note that in clouds such as vCloud, or any private cloud image and location ids are different per-user, so we'll probably have to think about this more. However, this should just "work" with gogrid, rackspace, aws-ec2, elastichosts or any other cloud with public scoped image ids. If we find "thrashing id" problem on public clouds, it is a sign we should revise our templateBuilder expression. FWIW: I'm happy to also implement this on the jclouds side so that when whirr goes to next version, you can inspect the last ids jclouds tested against.