[root@ip-10-0-100-90 centos]# sudo -u kudu kudu fs check -fs_data_dirs=/data0/kudu/tablet -fs_wal_dir=/data0/kudu/tabletwal W0705 06:50:47.607836 26099 data_dirs.cc:553] IO error: Could not lock /data0/kudu/tablet/data/block_manager_instance: Could not lock /data0/kudu/tablet/data/block_manager_instance: lock /data0/kudu/tablet/data/block_manager_instance: Resource temporarily unavailable (error 11) W0705 06:50:47.608680 26099 data_dirs.cc:554] Proceeding without lock I0705 06:50:47.629567 26099 fs_manager.cc:372] Time spent opening directory manager: real 0.023s user 0.001s sys 0.000s I0705 06:50:47.634166 26099 env_posix.cc:1596] Not raising process file limit of 32768; it is already as high as it can go I0705 06:50:47.635413 26099 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0705 06:50:57.672050 26115 log_block_manager.cc:2368] Opened 595 log block containers in /data0/kudu/tablet/data I0705 06:51:01.097707 26115 log_block_manager.cc:2438] Read-only block manager, skipping repair I0705 06:51:01.098453 26099 fs_manager.cc:384] Time spent opening block manager: real 13.460s user 0.000s sys 0.000s I0705 06:51:01.100960 26099 fs_manager.cc:404] Opened local filesystem: /data0/kudu/tablet,/data0/kudu/tabletwal uuid: "9e73c2acd39f4a91b965e0e0c177be70" format_stamp: "Formatted at 2018-02-26 15:22:01 on ip-10-0-100-90.twm.dmp.internal" Block manager report -------------------- 1 data directories: /data0/kudu/tablet/data Total live blocks: 619101 Total live bytes: 138371220530 Total live bytes (after alignment): 140620517376 Total number of LBM containers: 820 (83 full) Total missing blocks: 0 Total orphaned blocks: 329 (0 repaired) Total orphaned block bytes: 80015856 (0 repaired) Total full LBM containers with extra space: 2 (0 repaired) Total full LBM container extra space in bytes: 456695808 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) [centos@ip-10-0-100-52 ~]$ sudo -u kudu kudu fs check -fs_data_dirs=/data0/kudu/tablet -fs_wal_dir=/data0/kudu/tabletwal W0705 06:53:19.896780 15671 data_dirs.cc:553] IO error: Could not lock /data0/kudu/tablet/data/block_manager_instance: Could not lock /data0/kudu/tablet/data/ block_manager_instance: lock /data0/kudu/tablet/data/block_manager_instance: Resource temporarily unavailable (error 11) W0705 06:53:19.896881 15671 data_dirs.cc:554] Proceeding without lock I0705 06:53:19.923751 15671 fs_manager.cc:372] Time spent opening directory manager: real 0.028s user 0.001s sys 0.000s I0705 06:53:19.923810 15671 env_posix.cc:1596] Not raising process file limit of 32768; it is already as high as it can go I0705 06:53:19.923835 15671 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0705 06:53:29.930253 15685 log_block_manager.cc:2368] Opened 706 log block containers in /data0/kudu/tablet/data I0705 06:53:31.705355 15685 log_block_manager.cc:2438] Read-only block manager, skipping repair I0705 06:53:31.705648 15671 fs_manager.cc:384] Time spent opening block manager: real 11.779s user 0.000s sys 0.000s I0705 06:53:31.706430 15671 fs_manager.cc:404] Opened local filesystem: /data0/kudu/tablet,/data0/kudu/tabletwal uuid: "a1e5520262394966a176dd5a338da589" format_stamp: "Formatted at 2018-02-26 15:20:14 on ip-10-0-100-52.twm.dmp.internal" Block manager report -------------------- 1 data directories: /data0/kudu/tablet/data Total live blocks: 632998 Total live bytes: 137985195886 Total live bytes (after alignment): 140290686976 Total number of LBM containers: 845 (94 full) Total missing blocks: 0 Total orphaned blocks: 60 (0 repaired) Total orphaned block bytes: 78443302 (0 repaired) Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) [centos@ip-10-0-100-52 ~]$ [centos@ip-10-0-100-105 ~]$ sudo -u kudu kudu fs check -fs_data_dirs=/data0/kudu/tablet -fs_wal_dir=/data0/kudu/tabletwal W0705 06:54:26.783419 13623 data_dirs.cc:553] IO error: Could not lock /data0/kudu/tablet/data/block_manager_instance: Could not lock /data0/kudu/tablet/data/block_manager_instance: lock /data0/kudu/tablet/data/block_manager_instance: Resource temporarily unavailable (error 11) W0705 06:54:26.783543 13623 data_dirs.cc:554] Proceeding without lock I0705 06:54:26.810619 13623 fs_manager.cc:372] Time spent opening directory manager: real 0.029s user 0.001s sys 0.000s I0705 06:54:26.810672 13623 env_posix.cc:1596] Not raising process file limit of 32768; it is already as high as it can go I0705 06:54:26.810696 13623 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0705 06:54:36.815227 13637 log_block_manager.cc:2368] Opened 739 log block containers in /data0/kudu/tablet/data I0705 06:54:37.994369 13637 log_block_manager.cc:2438] Read-only block manager, skipping repair I0705 06:54:37.994662 13623 fs_manager.cc:384] Time spent opening block manager: real 11.182s user 0.000s sys 0.000s I0705 06:54:37.995589 13623 fs_manager.cc:404] Opened local filesystem: /data0/kudu/tablet,/data0/kudu/tabletwal uuid: "51b51a7ce22143b292960891f829df64" format_stamp: "Formatted at 2018-02-26 15:23:36 on ip-10-0-100-105.twm.dmp.internal" Block manager report -------------------- 1 data directories: /data0/kudu/tablet/data Total live blocks: 626859 Total live bytes: 138386876622 Total live bytes (after alignment): 140666134528 Total number of LBM containers: 843 (109 full) Total missing blocks: 0 Total orphaned blocks: 59 (0 repaired) Total orphaned block bytes: 84744877 (0 repaired) Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) [centos@ip-10-0-100-105 ~]$