Currently all 3 tests in the S3IncrementalBackupTest fail sporadically (between 1-5% of the time on the jenkins instances).
From my investigations, these failures mainly happen because S3Repository will create a directory node, and then assume it exists later on. However, the S3Mock may be too slow and later on when the assumption is made that this directory node exists, S3Mock is returning that the directory node does not exist.
The first fix for this was to just always check twice to see if a node is there, kind of hacky but it gives the S3Repository one more chance to find the right information.
However in the v2 S3 API, there is a concept of waiters, which will allow us to wait until S3 verifies the state we are looking for (i.e. that the directory node exists). We can either put this waiter in after creating the node, and not return until the waiter says the node is created. Or we can put it in when checking whether the node exists. I think the former is preferable, but we can do testing to see which actually preforms better.