[KYLIN-4713] Support use diff spark schedule pool for diff query

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: v4.0.0-alpha
• Component/s: Query Engine
• Assignee: Yaqian Zhang
• Reporter: Yaqian Zhang
• Created: 22/Aug/20 00:04
• Updated: 13/Sep/20 12:26
• Resolved: 03/Sep/20 07:27

Description

Kylin 4.0 use fair scheduler as spark scheduler mode in query module. The fair scheduler supports grouping jobs into pools, and setting different scheduling options (e.g. weight) for each pool.

User can config different spark pool in project level and sql level, and the configuration priority of SQL level is higher than that of project level. There are four spark pool in Kylin, which are 'query_pushdown', 'heavy_tasks', 'lightweight_tasks' and 'vip_tasks'.

If the user does not specify spark pool at both the SQL level and the project level, kylin will automatically adjust the spark pool used by SQL according to some configuration.

This feature can achieve resource isolation between different SQL to a certain extent, and avoid blocking the operation of lightweight SQL by SQL that takes more resources.