Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.7.0
-
None
-
None
-
Kubernetes
Description
Since Spark on Kubernetes has been released relatively long ago this can be a good idea to include Kubernetes support to Livy project as well, as it can solve much problems related to working with Spark on Kubernetes, it can fully replace Yarn in case of working atop Kubernetes cluster:
- Livy UI has cached logs/diagnostics page
- Livy UI shows links to Spark UI and Spark History Server
- With Kubernetes Ingress resource Livy can be configured to serve as an orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support option to create routes to Spark UI)
- Nginx Ingress solves `basePath` support for Spark UI and History Server as well as has lots of auth integrations available: https://github.com/kubernetes/ingress-nginx
- Livy UI can be integrated with Grafana Loki logs (PR provides solution for that)
Dockerfiles repo: https://github.com/jahstreet/spark-on-kubernetes-docker
Helm charts: https://github.com/jahstreet/spark-on-kubernetes-helm
Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588
Design concept: https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md
Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes v1.11.8:
- Image: Spark 2.4.3 with Hadoop 3.2.0 (https://github.com/jahstreet/spark-on-kubernetes-docker)
- History Server: https://github.com/helm/charts/tree/master/stable/spark-history-server
- Jupyter Notebook with Sparkmagic: https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter
Unit and/or integration tests are in progress.
Attachments
Issue Links
- links to