Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
-
Normal
Description
For one node, the cpu usage jumped to 100%, and logs are full of:
2015-09-14_16:34:45.56407 WARN 16:34:45 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:46.66616 WARN 16:34:46 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:47.76830 WARN 16:34:47 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:48.87043 WARN 16:34:48 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:49.97253 WARN 16:34:49 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:51.07462 WARN 16:34:51 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:52.17669 WARN 16:34:52 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:53.27880 WARN 16:34:53 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:54.38090 WARN 16:34:54 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:55.48301 WARN 16:34:55 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:56.58509 WARN 16:34:56 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:57.68721 WARN 16:34:57 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:58.78932 WARN 16:34:58 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:34:59.89142 WARN 16:34:59 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:00.99352 WARN 16:35:00 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:02.09563 WARN 16:35:02 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:03.19775 WARN 16:35:03 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:04.29982 WARN 16:35:04 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:05.40187 WARN 16:35:05 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:06.50369 WARN 16:35:06 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:07.60577 WARN 16:35:07 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:08.70779 WARN 16:35:08 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:09.80968 WARN 16:35:09 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:10.91157 WARN 16:35:10 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:12.01365 WARN 16:35:12 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:13.11569 WARN 16:35:13 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:14.21757 WARN 16:35:14 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:15.31942 WARN 16:35:15 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:16.42132 WARN 16:35:16 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:17.52332 WARN 16:35:17 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:18.62511 WARN 16:35:18 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:19.72697 WARN 16:35:19 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:20.82872 WARN 16:35:20 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:21.93074 WARN 16:35:21 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:23.03281 WARN 16:35:23 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:24.13478 WARN 16:35:24 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:25.23672 WARN 16:35:25 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:26.33862 WARN 16:35:26 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:27.44068 WARN 16:35:27 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:28.54267 WARN 16:35:28 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:29.64461 WARN 16:35:29 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:30.74646 WARN 16:35:30 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:35:31.84823 WARN 16:35:31 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
2015-09-14_16:40:09.97371 WARN 16:40:09 /2401:db00:2020:716b:face:0:21:0 has 2311 dropped hints, because node is down past configured hint window.
This happened to several nodes in our cluster, but one node each time.