[MESOS-9190] Test `StorageLocalResourceProviderTest.ROOT_CreateDestroyDiskRecovery` is flaky.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.8.0
• Component/s: storage
• Labels: mesosphere storage
• Epic Link: Resource Provider and CSI Tech Debt
• Sprint: Mesosphere Sprint 2018-27
• Story Points: 2
• Assignee: Chun-Hung Hsiao
• Reporter: Chun-Hung Hsiao
• Shepherd: Jie Yu
• Created: 29/Aug/18 01:11
• Updated: 29/Aug/18 01:12
• Resolved: 29/Aug/18 01:12

Description

The test is flaky in 1.7.x:

I0824 22:20:01.018494 4208 provider.cpp:1520] Received DESTROY_DISK operation '' (uuid: 7aaadd15-1f6d-4d4e-9000-4c250495f7ba)
W0824 22:20:01.018517 4208 provider.cpp:3008] Dropping operation (uuid: 7aaadd15-1f6d-4d4e-9000-4c250495f7ba): Cannot apply operation when reconciling storage pools
...
I0824 22:20:01.086668 4209 master.cpp:9445] Sending offers [ 0ab2c552-4d85-40fd-8717-8e4d19c7a65e-O4 ] to framework 0ab2c552-4d85-40fd-8717-8e4d19c7a65e-0000 (default) at scheduler-0af22a76-f591-43ba-8470-f4b863292d61@172.16.10.36:35916
../../src/tests/storage_local_resource_provider_tests.cpp:995: Failure
Mock function called more times than expected - returning directly.
Function call: resourceOffers(0x7ffe7ba8c240, @0x7f04a09808c0 { 160-byte object <98-7C 05-AD 04-7F 00-00 00-00 00-00 00-00 00-00 5F-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 05-00 00-00 05-00 00-00 10-A7 03-84 04-7F 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 ... 90-F2 08-84 04-7F 00-00 10-71 00-84 04-7F 00-00 40-71 00-84 04-7F 00-00 00-51 02-84 04-7F 00-00 C0-6A 03-84 04-7F 00-00 00-00 00-00 00-00 00-00 10-F0 00-84 04-7F 00-00 00-00 00-00 00-00 00-00> })
Expected: to be called once
Actual: called twice - over-saturated and active

This is because of `DESTRY_DISK` races with a profile poll. If the poll finishes first, SLRP will start reconciling storage pools, and drop certain incoming operations during reconciliation.