[KYLIN-5527] Kylin job engine adds the ability to check entries with data source(e.g. hive)

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 5.0-alpha
• Fix Version/s: 5.0-beta
• Component/s: Job Engine
• Assignee: Yaguang Jia
• Reporter: Yaguang Jia
• Created: 26/Apr/23 02:51
• Updated: 14/Jun/23 02:28
• Resolved: 14/Jun/23 02:28

Description

After adding some new indexes in the Kylin5 model, the operation of completing the indexes is needed. If the previous data in the data source (e.g.:hive) has been deleted, then there will be a situation that the newly added index data is empty, which will lead to wrong query results. Here we need to add a new data source check function, if the data in the data source has been deleted, then do not fill in the data of the corresponding time segment to ensure the correctness of the query results.

 

在Kylin5的模型中新增加一些索引后，需要进行补齐索引的操作，如果数据源（如:hive）里之前的数据被删掉了，那么就会出现新补出来的索引数据为空的情况，从而导致查询结果有误。这里需要新增加一个数据源核对的功能，如果数据源中的数据已经被删掉了，那么就不去补对应时间segment的数据，保证查询结果的正确性。