Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9190

Test `StorageLocalResourceProviderTest.ROOT_CreateDestroyDiskRecovery` is flaky.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.8.0
    • Component/s: storage

      Description

      The test is flaky in 1.7.x:

      I0824 22:20:01.018494 4208 provider.cpp:1520] Received DESTROY_DISK operation '' (uuid: 7aaadd15-1f6d-4d4e-9000-4c250495f7ba)
      W0824 22:20:01.018517 4208 provider.cpp:3008] Dropping operation (uuid: 7aaadd15-1f6d-4d4e-9000-4c250495f7ba): Cannot apply operation when reconciling storage pools
      ...
      I0824 22:20:01.086668 4209 master.cpp:9445] Sending offers [ 0ab2c552-4d85-40fd-8717-8e4d19c7a65e-O4 ] to framework 0ab2c552-4d85-40fd-8717-8e4d19c7a65e-0000 (default) at scheduler-0af22a76-f591-43ba-8470-f4b863292d61@172.16.10.36:35916
      ../../src/tests/storage_local_resource_provider_tests.cpp:995: Failure
      Mock function called more times than expected - returning directly.
      Function call: resourceOffers(0x7ffe7ba8c240, @0x7f04a09808c0 { 160-byte object <98-7C 05-AD 04-7F 00-00 00-00 00-00 00-00 00-00 5F-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 05-00 00-00 05-00 00-00 10-A7 03-84 04-7F 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 ... 90-F2 08-84 04-7F 00-00 10-71 00-84 04-7F 00-00 40-71 00-84 04-7F 00-00 00-51 02-84 04-7F 00-00 C0-6A 03-84 04-7F 00-00 00-00 00-00 00-00 00-00 10-F0 00-84 04-7F 00-00 00-00 00-00 00-00 00-00> })
      Expected: to be called once
      Actual: called twice - over-saturated and active
      

      This is because of `DESTRY_DISK` races with a profile poll. If the poll finishes first, SLRP will start reconciling storage pools, and drop certain incoming operations during reconciliation.

        Attachments

          Activity

            People

            • Assignee:
              chhsia0 Chun-Hung Hsiao
              Reporter:
              chhsia0 Chun-Hung Hsiao
              Shepherd:
              Jie Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: