[KUDU-3437] Starting Master server runs into error first time

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17.0
• Component/s: cfile
• Environment: Macos 13.0.1 Arm64, fastdebug build
• Assignee: Bakai Ádám
• Reporter: Bakai Ádám
• Created: 27/Jan/23 15:06
• Updated: 03/Apr/23 16:26
• Resolved: 31/Mar/23 22:37

Description

Reproduction script:

#Please make sure that stop_kudu.sh and start_kudu.sh and kudu command is in your path to run this script.
export PATH=$PATH:./bin/

stop_kudu.sh;rm -rf master-*; mkdir -p master-3/data master-3/wal master-3/log; start_kudu.sh -m 3

echo just to make sure it returns three masters as expected
kudu master list localhost:8764,localhost:8766,localhost:8768

echo This returns error:
kudu-master --master_addresses=127.0.0.1:8764,127.0.0.1:8766,127.0.0.1:8768,localhost:8770 --fs_data_dirs=$PWD/master-3/data --fs_wal_dir=$PWD/master-3/wal --log_dir=$PWD/master-3/log --rpc_bind_addresses=127.0.0.1:8770 --time_source=system_unsync --unlock_unsafe_flags --webserver_interface=127.0.0.1 --webserver_port=8771

echo just to make sure it returns three masters
kudu master list localhost:8764,localhost:8766,localhost:8768


echo Now it should be running fine:
kudu-master --master_addresses=127.0.0.1:8764,127.0.0.1:8766,127.0.0.1:8768,localhost:8770 --fs_data_dirs=$PWD/master-3/data --fs_wal_dir=$PWD/master-3/wal --log_dir=$PWD/master-3/log --rpc_bind_addresses=127.0.0.1:8770 --time_source=system_unsync --unlock_unsafe_flags --webserver_interface=127.0.0.1 --webserver_port=8771 &

sleep 2 
echo just to make sure it returns four masters
kudu master list localhost:8764,localhost:8766,localhost:8768
 

My error for the first run:

F20230127 15:58:28.923543 360774 cache.cc:579] Check failed: IsGTest() Metrics should only be set once per Cache
*** Check failure stack trace: ***
    @        0x1025950d4  google::LogMessageFatal::~LogMessageFatal()
    @        0x102591930  google::LogMessageFatal::~LogMessageFatal()
    @        0x10b56c4a8  kudu::(anonymous namespace)::ShardedCache<>::SetMetrics()
    @        0x103522c1c  kudu::cfile::BlockCache::StartInstrumentation()
    @        0x101249230  kudu::master::Master::Init()
    @        0x10126cbf0  kudu::master::RunMasterServer()
    @        0x1005eae64  kudu::master::MasterMain()
    @        0x1005ead10  main 

 If I uncomment the line

CHECK(IsGTest()) << "Metrics should only be set once per Cache";
 

from kudu/util/cache.cc:579  it seems to work fine.