Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-4394

MasterAuthorizationTest.SlaveRemoved is flaky

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.26.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
    • Environment:

      ubuntu 14.04.3 LTS kernel 3.16.0-55-generic

      Description

      Hi, I have the following error during "make check":

      [ RUN ] MasterAuthorizationTest.SlaveRemoved
      F0115 15:21:05.297031 3542 utils.cpp:47] CHECK_SOME(parse): syntax error at line 1 near: ,"frameworks\/test-principal\/messages_processed":1,,"frameworks\/test-principal\/messages_received":1,,"master\/cpus_percent":0,,"master\/cpus_revocable_percent":0,,"master\/cpus_revocable_total":0,,"master\/cpus_revocable_used":0,,"master\/cpus_total":0,,"master\/cpus_used":0,,"master\/disk_percent":0,,"master\/disk_revocable_percent":0,,"master\/disk_revocable_total":0,,"master\/disk_revocable_used":0,,"master\/disk_total":0,,"master\/disk_used":0,,"master\/dropped_messages":0,,"master\/elected":1,,"master\/event_queue_dispatches":0,,"master\/event_queue_http_requests":0,,"master\/event_queue_messages":0,,"master\/frameworks_active":1,,"master\/frameworks_connected":1,,"master\/frameworks_disconnected":0,,"master\/frameworks_inactive":0,,"master\/invalid_executor_to_framework_messages":0,,"master\/invalid_framework_to_executor_messages":0,,"master\/invalid_status_update_acknowledgements":0,,"master\/invalid_status_updates":0,,"master\/mem_percent":0,,"master\/mem_revocable_percent":0,,"master\/mem_revocable_total":0,,"master\/mem_revocable_used":0,,"master\/mem_total":0,,"master\/mem_used":0,,"master\/messages_authenticate":2,,"master\/messages_deactivate_framework":0,,"master\/messages_decline_offers":0,,"master\/messages_executor_to_framework":0,,"master\/messages_exited_executor":0,,"master\/messages_framework_to_executor":0,,"master\/messages_kill_task":0,,"master\/messages_launch_tasks":1,,"master\/messages_reconcile_tasks":0,,"master\/messages_register_framework":0,,"master\/messages_register_slave":1,,"master\/messages_reregister_framework":0,,"master\/messages_reregister_slave":0,,"master\/messages_resource_request":0,,"master\/messages_revive_offers":0,,"master\/messages_status_update":0,,"master\/messages_status_update_acknowledgement":0,,"master\/messages_suppress_offers":0,,"master\/messages_unregister_framework":0,,"master\/messages_unregister_slave":1,,"master\/messages_update_slave":1,,"master\/outstanding_offers":0,,"master\/recovery_slave_removals":0,,"master\/slave_registrations":1,,"master\/slave_removals":1,,"master\/slave_removals\/reason_registered":0,,"master\/slave_removals\/reason_unhealthy":0,,"master\/slave_removals\/reason_unregistered":1,,"master\/slave_reregistrations":0,,"master\/slave_shutdowns_canceled":0,,"master\/slave_shutdowns_completed":0,,"master\/slave_shutdowns_scheduled":0,,"master\/slaves_active":0,,"master\/slaves_connected":0,,"master\/slaves_disconnected":0,,"master\/slaves_inactive":0,,"master\/task_lost\/source_master\/reason_slave_removed":1,,"master\/tasks_error":0,,"master\/tasks_failed":0,,"master\/tasks_finished":0,,"master\/tasks_killed":0,,"master\/tasks_lost":1,,"master\/tasks_running":0,,"master\/tasks_staging":1,,"master\/tasks_starting":0,,"master\/uptime_secs":0,072886016,"master\/valid_executor_to_framework_messages":0,,"master\/valid_framework_to_executor_messages":0,,"master\/valid_status_update_acknowledgements":0,,"master\/valid_status_updates":0,,"registrar\/queued_operations":0,,"registrar\/registry_size_bytes":182,,"registrar\/state_fetch_ms":12,898816,"registrar\/state_store_ms":5,33888,"registrar\/state_store_ms\/count":3,"registrar\/state_store_ms\/max":5,33888,"registrar\/state_store_ms\/min":5,106944,"registrar\/state_store_ms\/p50":5,12896,"registrar\/state_store_ms\/p90":5,296896,"registrar\/state_store_ms\/p95":5,317888,"registrar\/state_store_ms\/p99":5,3346816,"registrar\/state_store_ms\/p999":5,33846016,"registrar\/state_store_ms\/p9999":5,338838016,"scheduler\/event_queue_dispatches":1,,"scheduler\/event_queue_messages":0,,"system\/cpus_total":4,,"system\/load_15min":0,26,"system\/load_1min":0,84,"system\/load_5min":0,49,"system\/mem_free_bytes":4223746048,,"system\/mem_total_bytes":6040068096,}

          • Check failure stack trace: ***
            @ 0x2b95be981740 google::LogMessage::Fail()
            @ 0x2b95be98168c google::LogMessage::SendToLog()
            @ 0x2b95be98108e google::LogMessage::Flush()
            @ 0x2b95be983fa2 google::LogMessageFatal::~LogMessageFatal()
            @ 0x9433b0 _CheckFatal::~_CheckFatal()
            @ 0x13a857c mesos::internal::tests::Metrics()
            @ 0xe2c1e9 mesos::internal::tests::MasterAuthorizationTest_SlaveRemoved_Test::TestBody()
            @ 0x15a0068 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
            @ 0x159afbe testing::internal::HandleExceptionsInMethodIfSupported<>()
            @ 0x157c6d3 testing::Test::Run()
            @ 0x157ce56 testing::TestInfo::Run()
            @ 0x157d49c testing::TestCase::Run()
            @ 0x1583be4 testing::internal::UnitTestImpl::RunAllTests()
            @ 0x15a0c8d testing::internal::HandleSehExceptionsInMethodIfSupported<>()
            @ 0x159bb34 testing::internal::HandleExceptionsInMethodIfSupported<>()
            @ 0x1582980 testing::UnitTest::Run()
            @ 0xd7ab14 RUN_ALL_TESTS()
            @ 0xd7a739 main
            @ 0x2b95c082eec5 (unknown)
            @ 0x9419f9 (unknown)
            make[3]: *** [check-local] Annullato (creato dump del core)
            make[3]: uscita dalla directory "/home/paride/mesos/build/src"
            make[2]: *** [check-am] Errore 2
            make[2]: uscita dalla directory "/home/paride/mesos/build/src"
            make[1]: *** [check] Errore 2
            make[1]: uscita dalla directory "/home/paride/mesos/build/src"
            make: *** [check-recursive] Errore 1
            paride@paride-HP-EliteBook-8440p-SK656UC-ABA:~/mesos/build$

      I tried also run this test alone and finished with success:

      Source directory: /home/paride/mesos
      Build directory: /home/paride/mesos/build
      /bin/nc
      /usr/bin/curl
      Note: Google Test filter = MasterAuthorizationTest.SlaveRemoved-HealthCheckTest.ROOT_DOCKER_DockerHealthyTask:HealthCheckTest.ROOT_DOCKER_DockerHealthStatusChange:HierarchicalAllocator_BENCHMARK_Test.DeclineOffers:HookTest.ROOT_DOCKER_VerifySlavePreLaunchDockerHook:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceBackward:SlaveTest.ROOT_RunTaskWithCommandInfoWithoutUser:SlaveTest.DISABLED_ROOT_RunTaskWithCommandInfoWithUser:DockerContainerizerTest.ROOT_DOCKER_Launch_Executor:DockerContainerizerTest.DISABLED_ROOT_DOCKER_Launch_Executor_Bridged:DockerContainerizerTest.ROOT_DOCKER_Launch:DockerContainerizerTest.ROOT_DOCKER_Kill:DockerContainerizerTest.ROOT_DOCKER_Usage:DockerContainerizerTest.ROOT_DOCKER_Update:DockerContainerizerTest.ROOT_DOCKER_Recover:DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker:DockerContainerizerTest.ROOT_DOCKER_Logs:DockerContainerizerTest.ROOT_DOCKER_Default_CMD:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Override:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Args:DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer:DockerContainerizerTest.DISABLED_ROOT_DOCKER_SlaveRecoveryExecutorContainer:DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping:DockerContainerizerTest.ROOT_DOCKER_LaunchSandboxWithColon:DockerContainerizerTest.ROOT_DOCKER_DestroyWhileFetching:DockerContainerizerTest.ROOT_DOCKER_DestroyWhilePulling:DockerContainerizerTest.ROOT_DOCKER_ExecutorCleanupWhenLaunchFailed:DockerContainerizerTest.ROOT_DOCKER_FetchFailure:DockerContainerizerTest.ROOT_DOCKER_DockerPullFailure:DockerContainerizerTest.ROOT_DOCKER_DockerInspectDiscard:DockerTest.ROOT_DOCKER_interface:DockerTest.ROOT_DOCKER_parsing_version:DockerTest.ROOT_DOCKER_CheckCommandWithShell:DockerTest.ROOT_DOCKER_CheckPortResource:DockerTest.ROOT_DOCKER_CancelPull:DockerTest.ROOT_DOCKER_MountRelative:DockerTest.ROOT_DOCKER_MountAbsolute:CpuIsolatorTest/1.UserCpuUsage:CpuIsolatorTest/1.SystemCpuUsage:RevocableCpuIsolatorTest.ROOT_CGROUPS_RevocableCpu:LimitedCpuIsolatorTest.ROOT_CGROUPS_CFS_Enable_Cfs:LimitedCpuIsolatorTest.ROOT_CGROUPS_CFS_Big_Quota:LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids:MemIsolatorTest/0.MemUsage:MemIsolatorTest/1.MemUsage:MemIsolatorTest/2.MemUsage:PerfEventIsolatorTest.ROOT_CGROUPS_Sample:SharedFilesystemIsolatorTest.DISABLED_ROOT_RelativeVolume:SharedFilesystemIsolatorTest.DISABLED_ROOT_AbsoluteVolume:NamespacesPidIsolatorTest.ROOT_PidNamespace:UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup:ProvisionerAppcTest.ROOT_Provision:BindBackendTest.ROOT_BindBackend:CopyBackendTest.ROOT_CopyBackend:ContainerizerTest.ROOT_CGROUPS_BalloonFramework:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Enabled:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Subsystems:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Mounted:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Get:CgroupsAnyHierarchyTest.ROOT_CGROUPS_NestedCgroups:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Tasks:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Read:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Write:CgroupsAnyHierarchyTest.ROOT_CGROUPS_CFS_Big_Quota:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FindCgroupSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FreezeNonFreezer:CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_MountUnmountHierarchy:CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat:CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_CpuAcctsStats:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Freeze:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Kill:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Destroy:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_AssignThreads:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyStoppedProcess:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyTracedProcess:CgroupsAnyHierarchyWithPerfEventTest.ROOT_CGROUPS_Perf:CgroupsAnyHierarchyMemoryPressureTest.ROOT_IncreaseRSS:CgroupsAnyHierarchyMemoryPressureTest.ROOT_IncreasePageCache:LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystem:LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystemCommandExecutor:LinuxFilesystemIsolatorTest.ROOT_Metrics:LinuxFilesystemIsolatorTest.ROOT_VolumeFromSandbox:LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost:LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint:LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithRootFilesystem:LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithoutRootFilesystem:LinuxFilesystemIsolatorTest.ROOT_ImageInVolumeWithoutRootFilesystem:LinuxFilesystemIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem:LinuxFilesystemIsolatorTest.ROOT_MultipleContainers:LinuxFilesystemIsolatorTest.ROOT_SandboxEnvironmentVariable:LinuxFilesystemIsolatorTest.ROOT_WorkDirMount:LinuxFilesystemIsolatorTest.ROOT_WorkDirMountPreExists:FsTest.ROOT_SharedMount:FsTest.ROOT_SlaveMount:MesosContainerizerLaunchTest.ROOT_ChangeRootfs:MemoryPressureMesosTest.CGROUPS_ROOT_Statistics:MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery:NsTest.ROOT_setns:NsTest.ROOT_setnsMultipleThreads:NsTest.ROOT_getns:NsTest.ROOT_destroy:PerfTest.ROOT_Events:PerfTest.ROOT_Sample:SchedTest.ROOT_PolicySelf:SchedTest.ROOT_PolicyChild:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/0:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/1:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/2:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/3:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/4:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/5:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/6:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/7:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/8:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/9:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/10:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/11:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/12:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/13:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/14:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/15:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/16:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/17:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/18:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/19:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/20:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/21:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/22:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/23:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/24:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/25:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/26:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/27:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/28:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/29:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/30:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/31:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/32:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/33:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/34:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/35:SlaveCount/Registrar_BENCHMARK_Test.Performance/0:SlaveCount/Registrar_BENCHMARK_Test.Performance/1:SlaveCount/Registrar_BENCHMARK_Test.Performance/2:SlaveCount/Registrar_BENCHMARK_Test.Performance/3
      [==========] Running 1 test from 1 test case.
      [----------] Global test environment set-up.
      [----------] 1 test from MasterAuthorizationTest
      [ RUN ] MasterAuthorizationTest.SlaveRemoved
      [ OK ] MasterAuthorizationTest.SlaveRemoved (171 ms)
      [----------] 1 test from MasterAuthorizationTest (171 ms total)

      [----------] Global test environment tear-down
      [==========] 1 test from 1 test case ran. (187 ms total)
      [ PASSED ] 1 test.

      I saw that there was another issue very similar here in Jira but It was closed without any further description.

      Thank you

      Paride

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                paridee87 Paride Casulli
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: