[GOBBLIN-1043] Implement a Helix assigned participant check as a CommitStep

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.15.0
• Component/s: gobblin-cluster
• Assignee: Hung Tran
• Reporter: Sudarshan Vasudevan
• Created: 07/Feb/20 00:28
• Updated: 12/Feb/20 19:14
• Resolved: 12/Feb/20 19:14

Description

In Gobblin Helix cluster, we encounter scenarios where a participant continues to work on a Helix partition even after the partition has been assigned by Helix to a different participant leading to duplicate consumption. We implement a check where each Helix task checks if it is indeed the currently assigned participant for the Helix partition. This check is implemented as a CommitStep and can be used as an assertion before data publish occurs. 

Issue Links

• links to: GitHub Pull Request #2883