[ZEPPELIN-3471] [Umbrella] Distributed zeppelin cluster services

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: zeppelin-server
• Flags: Important
• Docs Text: https://docs.google.com/document/d/1a8QLSyR3M5AhlG1GIYuDTj6bwazeuVDKCRRBm-Qa3Bw/edit?usp=sharing
• Assignee: Xun Liu
• Reporter: Xun Liu
• Created: 18/May/18 02:08
• Updated: 12/Mar/21 13:03

Description

1. At present, Zeppelin's Interpreter is only started on the same machine of the Zeppelin server. It cannot provide load balancing capability and more reasonable server resource utilization for high availability and a large number of users.
2. Zeppelin-server needs to provide multiple copy services on multiple servers;
3. Each zeppelin-server manages the life cycle of Interpreter processes running on this machine;
4. The Interpreter process can be started on any server through any zeppelin-web;

Please refer to on-going design doc, and add your thoughts:https://docs.google.com/document/d/1a8QLSyR3M5AhlG1GIYuDTj6bwazeuVDKCRRBm-Qa3Bw/edit?usp=sharing

Issue Links

• relates to: ZEPPELIN-4104 Zeppelin Interpreter On Docker

Sub-Tasks

• Cluster Raft module design: Closed
• Cluster server and client module design: Resolved
• Create a cluster interpreter process module design: Closed
• Cluster High utilization module design: Closed
• Cluster High availability module design: Closed
• Cluster synchronize notes & authorization: Closed
• Cluster server node and interpreter process management web page: Closed
• Cluster synchronize InterpreterSetting: Closed
• Fixed the ClusterInterpreterLauncher can not listen to cluster events: Closed
• NotebookAuthorization does not need to synchronize cluster event: Closed
• Cluster mode support interpreter running in docker: Closed
• Fix netty-all-4.0.23.Final.jar version conflict: Closed
• Formatted display service time in the cluster management page: Closed
• Fix cluster metadata interpreter offline timeout calculation error: Closed
• Fixed unusable after cluster mode restarts interpreter: Closed
• Choose which server to run the interpreter process: Open
• Cluster Session module design: Open
• View the zeppelin service log in the cluster management page: Open
• Restart the zeppelin service in the cluster management page: Open
• Restart the zeppelin interpreter in the cluster management page: Open
• Optimized cluster metadata calculation interpreter offline status: Open