[KYLIN-4485] Create a self service interface for cube migration

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v3.1.0
• Component/s: Tools, Build and Test
• Sprint: Sprint 52
• Assignee: Zhong Yanghong
• Reporter: Zhong Yanghong
• Created: 09/May/20 08:57
• Updated: 10/Jul/20 12:17
• Resolved: 15/Jun/20 03:19

Description

Current cube migration process is as follows:

 

There're a few drawbacks:

• kylin admin has to manually check all related aspects to decide whether a cube is good for migration. If cube design is not good, there'll be many interaction between users & kylin admins
• kylin admin has to login to the backend server and then run a command to do the migration, which is tricky and easy to make mistakes
• there's little compatibility check between source metadata & destination cluster.

A self service interface will refine the migration process as follows:

We can see with the new migration process, there'll be few interactions between users & kylin admins and much less work for kylin admins.

 

For this new feature, add some new properties:

• kylin.cube.migration.enabled
  Specifies whether to show 'Migrate', 'Approve Migration' and 'Reject Migration' in "Action" drop-down list for cube management, the default value is false;
• kylin.metadata.model-schema-updater-checker-enabled
  Specifies whether to check for model schema update when check check model compatibility, the default value is false;
• kylin.metadata.able-change-string-to-datetime
  Specifies whether to enable to convert String to Date or Time when check table schema update, the default value is false;
• kylin.cube.rowkey-encoding-auto-convert
  Specifies whether to enable to convert date/time dictionary on date/time/varchar column to DimensionEncoding implicitly, the default value is true;
• kylin.server.self-discovery-enabled
  Specifies whether to enable to discover kylin instance from zookeeper, the default value is false;
• kylin.cube.migration.expansion-rate
  Specifies the threshold value of expansion rate when check the quality of the cube, the default value is 5;
• kylin.cube.migration.query-generator-max-dimension-number
  Specifies the max number of dimension column when generate sql for checking the quality of the cube, the default value is 3;
• kylin.cube.migration.rule-query-latency-enabled
  Specifies whether to check query latency when check the quality of the cube, the default value is true;
• kylin.cube.migration.query-latency-seconds
  Specifies the threshold value of query latency in seconds when check the quality of the cube, the default value is 2;
• kylin.cube.migration.query-latency-max-threads
  Specifies the max number of query threads to execute sql when check the quality of the cube, the default value is 5;
• kylin.cube.migration.query-latency-iteration
  Specifies the max number of query sql to be generated when check the quality of the cube, the default value is 5;
• kylin.cube.migration.source-address
  Specifies the source address of kylin instance;
• kylin.cube.migration.target-address
  Specifies the target address of kylin instance;
• kylin.cube.notification-mail-suffix
  Specifies the default suffix address of email for sending notifications.

Issue Links

• fixes: KYLIN-700 Cube Release Tool
• is blocked by: KYLIN-4419 Make it possible to change metadata in some cases without rebuilding data

Sub-Tasks

• Backend support for auto-migration to allow user to do cube migration by self service: Closed
• Extract kylin server self discovery service from CuratorScheduler: Closed
• Frontend support for auto-migration to allow user to do cube migration by self service: Closed
• Create a tool for migration cross clusters: Closed