Details
-
Improvement
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
With the introduction of version probing in 2.0 and cooperative rebalancing in 2.4, the specific upgrade path depends heavily on the to & from version. This can be a complex operation, and we should make sure to test a realistic upgrade scenario across all possible combinations. The current system tests have gaps however, which have allowed bugs in the upgrade path to slip by unnoticed for several versions.
Our current system tests include a "simple upgrade downgrade" test, a metadata upgrade test, a version probing test, and a cooperative upgrade test. This has a few drawbacks and current issues:
a) only the version probing test tests "forwards compatibility" (upgrade from latest to future version)
b) nothing tests version probing "backwards compatibility" (upgrade from older version to latest), except:
c) the cooperative rebalancing test actually happens to involve a version probing step, and so coincidentally DOES test VP (but only starting with 2.4)
d) each test itself tries to test the upgrade across different versions, meaning there may be overlap and/or unnecessary tests. For example, currently both the metadata_upgrade and cooperative_upgrade tests will test the upgrade of 1.0 -> 2.4
e) worse, a number of (to, from) pairs are not tested according to the correct upgrade path at all, which has lead to easily reproducible bugs slipping past for several versions.
f) we have a test_simple_upgrade_downgrade test which does not actually do a downgrade, and for some reason only tests upgrading within the range [0.10.1 - 1.1]
g) as new versions are released, it is unclear to those not directly involved in these tests and/or projects whether and what needs to be updated (eg should this new version be added to the cooperative test? should the old version be aded to the metadata test?)
We should definitely fill in the testing gap here, but how to do so is of course up for discussion.
I would propose to refactor the upgrade tests, and rather than maintain different lists of versions to pass as input to each different test, we should have a single matrix that we update with each new version that specifies which upgrade path that version combination actually requires. We can then loop through each version combination and test only the actual upgrade path that users would actually need to follow. This way we can be sure we are not missing anything, as each and every possible upgrade would be tested.