[KYLIN-4445] Provide a kylin on kubernetes solution

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v3.1.0
• Component/s: Integration
• Labels: Kubernetes
• Sprint: Sprint 49
• Assignee: Xiaoxiang Yu
• Reporter: Xiaoxiang Yu
• Created: 17/Apr/20 09:29
• Updated: 10/Jul/20 12:23
• Resolved: 15/May/20 05:11

Description

Backgroud

From experience of kylin user, like OLX group, DXY.cn and eBay, they choose to deploy kylin and its dependency in Kubernetes . And their approach/practice is proved works well in production env and these deployment template could be share in Kylin community.

 

This pic is an example of Kylin on Kubernetes 's architecture.

---

Following are some of the patch lists:

• https://github.com/sanjulian/kylin/pull/1
• https://github.com/apache/kylin/pull/864

 

Proposal

At high level, I proposed to provided two deployment templates, one for quick-start purpose, another for production/distributed deployment.
 

Quick-start template is for one node deployment with an ALL kylin instance.
Production template is for multi-nodes deployment with a few of ALL/job/query kylin instances; and some other service like memcached and filebeat will help to solve log collection/query cache/session sharing.

Following is the structure of provided deployment template:

1. config is the place of store configuration file of Hadoop/Kylin ;
2. docker is the place which you can build hadoop-client and kylin-client docker image yourself.
3. kubernetes is the place for storing deployment template.

apache-kylin:k8s/ (kubernetes*) $ tree -L 3                                                                                                  [16:31:09]
.
├── config
│   ├── production
│   │   ├── filebeat
│   │   ├── hadoop
│   │   ├── kerberos
│   │   ├── kylin-all
│   │   ├── kylin-job
│   │   ├── kylin-more
│   │   ├── kylin-query
│   │   ├── sso
│   │   ├── streaming-receiver
│   │   └── tomcat
│   └── quickstart
│       ├── hadoop
│       └── kylin
├── docker
│   ├── build.sh
│   ├── hadoop-client
│   │   ├── CDH-5.7
│   │   ├── README.MD
│   │   ├── apache-hadoop2.7
│   │   └── provided-binary
│   └── kylin-client
│       ├── Dockerfile
│       ├── apache-kylin-x.x.x-bin-cdh57
│       ├── bin
│       ├── build.sh
│       └── crontab.txt
└── kubernetes
    ├── production
    │   ├── check-deploy.sh
    │   ├── cleanup.sh
    │   ├── deploy-kylin.sh
    │   └── deployment
    └── quickstart
        ├── check-cluster.sh
        ├── cleanup.sh
        ├── deploy-kylin.sh
        └── deployment

Credits

• DXY.cn ztp
• eBay team julianpan

Issue Links

• contains: KYLIN-4181 Schedule Kylin using Kubernetes

Sub-Tasks

• Kylin on kubernetes in a quickstart env: Closed
• Kylin on kubernetes in a production env: Closed
• Provide documentation for Kylin on kubernetes: Closed