Thanks Yiqun Lin.
Yes, I do see the addVolume volume operation from the stack trace. But, that volume adding thread shown in the stack trace is just in the DataStorage#prepareVolume phase and it is only traversing the storageDirs in Storage#containsStorageDir. That is, there is yet another thread which is mutating the same ArrayList around the same time when the first Volume Add was happening.
Looked at the test code again and there are 2 Volume Add happening as part of the test. As you said, these Volume add operations are run in Executors via FutureTask and hence these are submitted in quick succession and are running parallely and are mutating the same storageDirs list.
I am able to recreate the ConcurrentModificationException by following 2 ways:
- Running a new Thread which continuously runs a read operation on the StorageDirs (like listStorageDirectories) and then having Volume add in parallel, OR
- Adding 10 volumes with a small delay between each other, so that each of volume add's list traversing will trip over previous volume add list modification.
My fix proposal is to create Storage#storageDirs as new CopyOnWriteArrayList<StorageDirectory>(); Will test out and submit a patch.