[ZEPPELIN-3626] Cluster server and client module design

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Component/s: zeppelin-server
• Flags: Important
• External issue URL: https://issues.apache.org/jira/browse/ZEPPELIN-3471
• External issue ID: ZEPPELIN-3471
• Docs Text: https://docs.google.com/document/d/1a8QLSyR3M5AhlG1GIYuDTj6bwazeuVDKCRRBm-Qa3Bw/edit#heading=h.qbcgqhd0wwh8
• Assignee: Xun Liu
• Reporter: Xun Liu
• Created: 16/Jul/18 03:27
• Updated: 16/May/19 15:48
• Resolved: 16/May/19 15:48

Description

Cluster management server

The cluster management server uses the Raft algorithm library Atomix to form a service cluster with consistent service status in the Zeppelin cluster.

1. The cluster management service runs in each Zeppelin-Server;

1. The cluster management service establishes a cluster by using the copycatServer class of the Raft algorithm library, maintains the ClusterStateMachine, and manages the service state metadata of each Zeppelin-Server through the PutCommand, GetQuery, and DeleteCommand operation commands.

1. Launch the Thrift service in the cluster management service to enable the cluster interpreter process to be created by remote calls in each Zeppelin-Server;

Cluster management client

The cluster management client connects to the cluster management server for metadata operations of services and processes through the Raft algorithm library Atomix.

1. The cluster management client runs in each Zeppelin-Server and Zeppelin Interpreter process;

1. The cluster management client manages the Zeppelin-Server and Zeppelin Interpreter process state (metadata information) in the ClusterStateMachine by using the copycatClient class of the Raft library to connect to the copycatServer. When the Zeppelin-Server and Zeppelin Interpreter processes are started, They are added to the ClusterStateMachine and are removed from the ClusterStateMachine when the Zeppelin-Server and Zeppelin Interpreter processes are closed.

1. In a distributed environment, network anomalies, network delays, or service exceptions may occur. After copycatClient submits metadata to the cluster, it checks whether the submission is successful. After the submission fails, the metadata is saved in the local message queue. Retrying by copycatClient through a separate commit thread;

Cluster monitoring module

The cluster monitoring module checks if each Zeppelin-Server and Zeppelin Interpreter process in the cluster is active

1. The cluster monitoring module runs in each Zeppelin-Server and Zeppelin Interpreter process, periodically sending heartbeat data of the service or process to the cluster;

1. When the cluster monitoring module runs in Zeppelin-Server, it collects the CPU and MEMORY usage of the server, and sends the resource usage rate to the cluster's ClusterStateMachine. When the cluster interpreter process needs to be created, the server is idle from the resource. Created in ;

1. Resource usage statistics strategy, in order to avoid the instantaneous high peak and low peak of the server, the cluster monitoring will collect the average resource usage in the most recent period for reporting, and improve the reasonable line and effectiveness of the server resources as much as possible;

1. When the cluster monitoring module runs in Zeppelin-Server, it checks the heartbeat data of each Zeppelin-Server and Zeppelin Interpreter process. If it times out, it considers that the service or process is abnormally unavailable and removes it from the cluster.

Issue Links

• links to: GitHub Pull Request #3203
• links to: GitHub Pull Request #3203