Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Today GobblinClusterManager leverages single Helix cluster responsible for both job distribution and cluster manager HA. This all-in-one mode cannot works with Helix super controller, because GobblinClusterManager will create its own dedicated controller for HA handling, which is internal to Gobblin framework. This architect works fine but gradually we find it's hard to monitor Helix behavior and debug Helix related issues due to the lack of Helix task framework metrics, which is enabled for free, but only available when using a dedicated controllers under Helix super controller's supervision.
To allow the migration, we separated existing cluster into two clusters:
1. Our existing cluster will remain the same, but called as "job distribution cluster" in the separation mode. In unit test or local deployment mode, we will create a dedicated controller for this cluster. In production mode, we can assume Helix will provide a dedicated controller for us.
2. A new cluster will be created, now called 'manager cluster', which is responsible for cluster manager leadership change. This will provide leadership change callback just like we did earlier in all-in-one mode.
The new 'two cluster mode' can be turned on/off by user configuration. Similarly user can configure whether a controller for job distribution should be created.