[CASSANDRA-12685] Add retry to hints dispatcher

• Type: Improvement
• Status: Resolved
• Priority: Low
• Resolution: Not A Problem
• Fix Version/s: 4.1-alpha1, 4.1
• Component/s: Legacy/Coordination
• Assignee: Dikang Gu
• Reporter: Dikang Gu

Authors

Dikang Gu

• Created: 22/Sep/16 03:19
• Updated: 16/Dec/22 21:56
• Resolved: 20/Jul/17 12:38

Description

Problem: I often see timeout in hints replay, I find there is no retry for hints replay, I think it would be great to add some retry logic for timeout exception.

2016-09-20_07:32:01.16610 INFO  07:32:01 [HintedHandoff:3]: Started hinted handoff for host: 859af100-5d45-42bd-92f5-2bc78822158b with IP: /2401:db00:12:30d7:face:0:39:0
2016-09-20_07:58:49.29983 INFO  07:58:49 [HintedHandoff:3]: Timed out replaying hints to /2401:db00:12:30d7:face:0:39:0; aborting (55040 delivered)
2016-09-20_07:58:49.29984 INFO  07:58:49 [HintedHandoff:3]: Enqueuing flush of hints: 15962349 (0%) on-heap, 2049808 (0%) off-heap
2016-09-20_08:02:17.55072 INFO  08:02:17 [HintedHandoff:1]: Started hinted handoff for host: 859af100-5d45-42bd-92f5-2bc78822158b with IP: /2401:db00:12:30d7:face:0:39:0
2016-09-20_08:05:45.25723 INFO  08:05:45 [HintedHandoff:1]: Timed out replaying hints to /2401:db00:12:30d7:face:0:39:0; aborting (7936 delivered)
2016-09-20_08:05:45.25725 INFO  08:05:45 [HintedHandoff:1]: Enqueuing flush of hints: 2301605 (0%) on-heap, 259744 (0%) off-heap
2016-09-20_08:12:19.92910 INFO  08:12:19 [HintedHandoff:2]: Started hinted handoff for host: 859af100-5d45-42bd-92f5-2bc78822158b with IP: /2401:db00:12:30d7:face:0:39:0
2016-09-20_08:51:44.72191 INFO  08:51:44 [HintedHandoff:2]: Timed out replaying hints to /2401:db00:12:30d7:face:0:39:0; aborting (83456 delivered)