Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Not A Problem
-
None
-
None
-
None
-
Normal
Description
Since our move from 0.6.8 to 0.7, all of the nodes which speak with clients have been having periodic, abrupt load spikes going into the hundreds. We have been seeing these load spikes 1 to 2 times per hour on every node which clients are speaking with. The load graph for a typical spike: http://i.imgur.com/jY8AV.png
I have verified that client connections are not spiking at the same time via TCP statistics. I have also verified that we aren't seeing any spikes in reads/mutations/etc.
We were using the DynamicSnitch, but I turned that off as a troubleshooting step. The issue was unchanged.
When the spikes occur, the box's responsiveness slows to a crawl so I am unable to do much in terms of real-time diagnostics. I was able to get a thread dump a few seconds after a spike, which I have attached to this ticket. Not sure if it will show anything since I couldn't capture it immediately during the spike.
I should note that David King noticed a similar problem (#2058) when he tried moving us from 0.6.8 to 0.6.10. The main issue at the time was a long-lasting load spike, but he also saw occasional abrupt load spikes like we are seeing now. When we moved back to 0.6.8, we didn't see the problem again, until the move to 0.7.
I realize this information is somewhat nebulous. If there is any further info I can provide, please let me know. The spikes are causing quite a bit of instability, so we are considering retreating back to 0.6.8. I'd like to investigate every possible solution before we resort to that.
Attachments
Attachments
Issue Links
- relates to
-
-
- Resolved
-