Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-2744

MasterAuthorizationTest.SlaveRemoved is flaky

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • None
    • None
    • None

    Description

      See (1) and (2), just executed in that order.

      Results make for me - from a blackbox point of view - no sense at all. My two cents/theory - tests themselfs(t.i. the framework's they use) seem to affect each other.

      Will file an issue in your JIRA. Pls provide info for access/handling your JIRA e.g. is this email as description enough information for your investigation?

      (1)

      joma@kopernikus-u:~/dev/programme/mesos/build/mesos/build$ make check GTEST_FILTER="MasterAuthorizationTest.SlaveRemoved" GTEST_REPEAT=1000 GTEST_BREAK_ON_FAILURE=1
      ...
      Repeating all tests (iteration 1000) . . .

      Note: Google Test filter = MasterAuthorizationTest.SlaveRemoved-DockerContainerizerTest.ROOT_DOCKER_Launch_Executor:DockerContainerizerTest.ROOT_DOCKER_Launch_Executor_Bridged:DockerContainerizerTest.ROOT_DOCKER_Launch:DockerContainerizerTest.ROOT_DOCKER_Kill:DockerContainerizerTest.ROOT_DOCKER_Usage:DockerContainerizerTest.ROOT_DOCKER_Update:DockerContainerizerTest.DISABLED_ROOT_DOCKER_Recover:DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker:DockerContainerizerTest.ROOT_DOCKER_Logs:DockerContainerizerTest.ROOT_DOCKER_Default_CMD:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Override:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Args:DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer:DockerContainerizerTest.DISABLED_ROOT_DOCKER_SlaveRecoveryExecutorContainer:DockerContainerizerTest.ROOT_DOCKER_PortMapping:DockerContainerizerTest.ROOT_DOCKER_LaunchSandboxWithColon:DockerContainerizerTest.ROOT_DOCKER_DestroyWhileFetching:DockerContainerizerTest.ROOT_DOCKER_Destr
      o
      yWhilePulling:DockerTest.ROOT_DOCKER_interface:DockerTest.ROOT_DOCKER_CheckCommandWithShell:DockerTest.ROOT_DOCKER_CheckPortResource:DockerTest.ROOT_DOCKER_CancelPull:CpuIsolatorTest/1.UserCpuUsage:CpuIsolatorTest/1.SystemCpuUsage:LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs:LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs_Big_Quota:MemIsolatorTest/0.MemUsage:MemIsolatorTest/1.MemUsage:MemIsolatorTest/2.MemUsage:PerfEventIsolatorTest.ROOT_CGROUPS_Sample:SharedFilesystemIsolatorTest.ROOT_RelativeVolume:SharedFilesystemIsolatorTest.ROOT_AbsoluteVolume:NamespacesPidIsolatorTest.ROOT_PidNamespace:UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceBackward:SlaveTest.ROOT_RunTaskWithCommandInfoWithoutUser:SlaveTest.DI
      S
      ABLED_ROOT_RunTaskWithCommandInfoWithUser:ContainerizerTest.ROOT_CGROUPS_BalloonFramework:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Enabled:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Subsystems:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Mounted:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Get:CgroupsAnyHierarchyTest.ROOT_CGROUPS_NestedCgroups:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Tasks:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Read:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Write:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Cfs_Big_Quota:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FindCgroupSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FreezeNonFreezer:CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_
      M
      ountUnmountHierarchy:CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Freeze:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Kill:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Destroy:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_AssignThreads:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyStoppedProcess:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyTracedProcess:CgroupsAnyHierarchyWithPerfEventTest.ROOT_CGROUPS_Perf:NsTest.ROOT_setns:NsTest.ROOT_setnsMultipleThreads:NsTest.ROOT_getns:NsTest.ROOT_destroy:PerfTest.ROOT_Events:PerfTest.ROOT_SampleInit:SlaveCount/Registrar_BENCHMARK_Test.performance/0:SlaveCount/Registrar_BENCHMARK_Test.performance/1:SlaveCount/Registrar_BENCHMARK_Test.performance/2:SlaveCount/Registrar_BENCHMARK_Test.performance/3
      [==========] Running 1 test from 1 test case.
      [----------] Global test environment set-up.
      [----------] 1 test from MasterAuthorizationTest
      [ RUN ] MasterAuthorizationTest.SlaveRemoved
      [ OK ] MasterAuthorizationTest.SlaveRemoved (483 ms)
      [----------] 1 test from MasterAuthorizationTest (484 ms total)

      [----------] Global test environment tear-down
      [==========] 1 test from 1 test case ran. (499 ms total)
      [ PASSED ] 1 test.

      YOU HAVE 9 DISABLED TESTS

      make[3]: Leaving directory `/home/joma/entwicklung/programme/mesos/build/mesos/build/src'
      make[2]: Leaving directory `/home/joma/entwicklung/programme/mesos/build/mesos/build/src'
      make[1]: Leaving directory `/home/joma/entwicklung/programme/mesos/build/mesos/build/src'
      Making check in ec2
      make[1]: Entering directory `/home/joma/entwicklung/programme/mesos/build/mesos/build/ec2'
      make[1]: Nothing to be done for `check'.
      make[1]: Leaving directory `/home/joma/entwicklung/programme/mesos/build/mesos/build/ec2'

      (2)
      joma@kopernikus-u:~/dev/programme/mesos/build/mesos/build$ make check
      ...
      [----------] 13 tests from MasterAuthorizationTest
      [ RUN ] MasterAuthorizationTest.AuthorizedTask
      [ OK ] MasterAuthorizationTest.AuthorizedTask (164 ms)
      [ RUN ] MasterAuthorizationTest.UnauthorizedTask
      [ OK ] MasterAuthorizationTest.UnauthorizedTask (160 ms)
      [ RUN ] MasterAuthorizationTest.KillTask
      [ OK ] MasterAuthorizationTest.KillTask (171 ms)
      [ RUN ] MasterAuthorizationTest.SlaveRemoved
      F0516 23:00:28.584900 4444 mesos.cpp:362] CHECK_SOME(parse): syntax error at line 1 near: ,"master\/valid_framework_to_executor_messages":0,"master\/valid_status_update_acknowledgements":0,"master\/valid_status_updates":0,"registrar\/queued_operations":0,"registrar\/registry_size_bytes":90,"registrar\/state_fetch_ms":35.44704,"registrar\/state_store_ms":12.803072,"registrar\/state_store_ms\/count":3,"registrar\/state_store_ms\/max":16.69632,"registrar\/state_store_ms\/min":12.803072,"registrar\/state_store_ms\/p50":14.125824,"registrar\/state_store_ms\/p90":16.1822208,"registrar\/state_store_ms\/p95":16.4392704,"registrar\/state_store_ms\/p99":16.64491008,"registrar\/state_store_ms\/p999":16.691179008,"registrar\/state_store_ms\/p9999":16.6958059008,"scheduler\/event_queue_dispatches":0,"scheduler\/event_queue_messages":0,"system\/cpus_total":8,"system\/load_15min":0.38,"system\/load_1min":0.79,"system\/load_5min":0.53,"system\/mem_free_bytes":5243604992,"system\/mem_total_bytes":82
      6
      1496832}

          • Check failure stack trace: ***
            @ 0x2b95db57a800 google::LogMessage::Fail()
            @ 0x2b95db57a74c google::LogMessage::SendToLog()
            @ 0x2b95db57a14e google::LogMessage::Flush()
            @ 0x2b95db57d062 google::LogMessageFatal::~LogMessageFatal()
            @ 0xa26e98 _CheckFatal::~_CheckFatal()
            @ 0xe60353 mesos::internal::tests::MesosTest::Metrics()
            @ 0xd881f1 mesos::internal::tests::MasterAuthorizationTest_SlaveRemoved_Test::TestBody()
            @ 0x113db61 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
            @ 0x1138d1c testing::internal::HandleExceptionsInMethodIfSupported<>()
            @ 0x11210dd testing::Test::Run()
            @ 0x1121800 testing::TestInfo::Run()
            @ 0x1121d88 testing::TestCase::Run()
            @ 0x1126a52 testing::internal::UnitTestImpl::RunAllTests()
            @ 0x113e9d3 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
            @ 0x1139a0d testing::internal::HandleExceptionsInMethodIfSupported<>()
            @ 0x112595e testing::UnitTest::Run()
            @ 0xd2c87d main
            @ 0x2b95ddb12ec5 (unknown)

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              haosdent@gmail.com haosdent
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: