[RATIS-666] Coalesced heartbeat in multiraft

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: raft-group
• Assignee: Unassigned
• Reporter: Li Cheng
• Created: 22/Aug/19 12:05
• Updated: 29/Aug/19 21:52

Description

I'm using this issue to discuss the coalesced heartbeat plan in multi-raft. We are looking at incorporating multi-raft feature in ratis into Hadoop Ozone. So in ozone, every datanode would be in multiple raft groups or say pipelines with multi-raft, which brings:

1. Is there any plan for coalesced heartbeat on single node? 
2. Are we going to use gRPC to achieve coalesced heartbeat like what cockroach does? Shall we assume only Java APIs are required?
3. Either we have coalesced heartbeat, every node would have chances to be selected as leader in each raft group. So to the extreme extend, one node, say node A, would be the leader to all raft groups. If we implement coalesced heartbeat, there would more easily push node A to be the bottleneck for future stumbling in performance. Any idea on how to avoid this extremity? Maybe do a candidate scrub?
4. How do we plan to test the 'single node, multi raft groups' scenario? Furthermore, if we allow coalesced heartbeat configurable, how to determine when and whether to use it?

 

szetszwo Sammi xyao waterlx