Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9765

Test `ROOT_CreateDestroyPersistentMountVolumeWithReboot` is flaky.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.9.0
    • Component/s: test
    • Labels:
    • Sprint:
      Storage: RI-13 Sprint 45, Storage: RI-14 Sprint 46
    • Story Points:
      1

      Description

      Observed a failure on test CSIVersion/StorageLocalResourceProviderTest.ROOT_CreateDestroyPersistentMountVolumeWithReboot/v0 with the following error:

      unknown file: Failure
      
      Unexpected mock function call - returning directly.
          Function call: statusUpdate(0x7ffea80beb70, @0x7f6027ff6b90 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>)
      Google Mock tried the following 6 expectations, but none matched:
      
      /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2552: tried expectation #0: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_STARTING)))...
        Expected arg #1: task status state eq TASK_STARTING
                 Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
               Expected: to be called once
                 Actual: called once - saturated and active
      /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2553: tried expectation #1: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_RUNNING)))...
        Expected arg #1: task status state eq TASK_RUNNING
                 Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
               Expected: to be called once
                 Actual: called once - saturated and active
      /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2554: tried expectation #2: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_FINISHED)))...
        Expected arg #1: task status state eq TASK_FINISHED
                 Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
               Expected: to be called once
                 Actual: called once - saturated and active
      /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2630: tried expectation #3: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_STARTING)))...
        Expected arg #1: task status state eq TASK_STARTING
                 Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
               Expected: to be called once
                 Actual: called once - saturated and active
      /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2631: tried expectation #4: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_RUNNING)))...
        Expected arg #1: task status state eq TASK_RUNNING
                 Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
               Expected: to be called once
                 Actual: never called - unsatisfied and active
      /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2632: tried expectation #5: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_FINISHED)))...
        Expected arg #1: task status state eq TASK_FINISHED
                 Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
               Expected: to be called once
                 Actual: never called - unsatisfied and active
      

      The problem is that the task was OOM-killed:

      I0503 01:04:24.410918 24105 memory.cpp:515] OOM detected for container df47d711-3ee8-4430-b90d-b29bcb90d407
      I0503 01:04:24.411118 24105 memory.cpp:555] Memory limit exceeded: Requested: 32MB Maximum Used: 32MB
      
      MEMORY STATISTICS:
      cache 471040
      rss 33083392
      rss_huge 14680064
      mapped_file 0
      swap 0
      pgpgin 4662
      pgpgout 47
      pgfault 77439
      pgmajfault 0
      inactive_anon 471040
      active_anon 33083392
      inactive_file 0
      active_file 0
      unevictable 0
      hierarchical_memory_limit 33554432
      hierarchical_memsw_limit 9223372036854771712
      total_cache 471040
      total_rss 33083392
      total_rss_huge 14680064
      total_mapped_file 0
      total_swap 0
      total_pgpgin 4662
      total_pgpgout 47
      total_pgfault 77439
      total_pgmajfault 0
      total_inactive_anon 471040
      total_active_anon 33083392
      total_inactive_file 0
      total_active_file 0
      total_unevictable 0
      I0503 01:04:24.412195 24105 containerizer.cpp:3147] Container df47d711-3ee8-4430-b90d-b29bcb90d407 has reached its limit for resource [{"name":"mem","scalar":{"value":32.0},"type":"SCALAR"}] and will be terminated
      I0503 01:04:24.412314 24105 containerizer.cpp:2586] Destroying container df47d711-3ee8-4430-b90d-b29bcb90d407 in RUNNING state
      

      This might happen to other SLRP root tests as well.

        Attachments

          Activity

            People

            • Assignee:
              chhsia0 Chun-Hung Hsiao
              Reporter:
              chhsia0 Chun-Hung Hsiao
              Shepherd:
              Benjamin Bannier
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: