[KYLIN-4021] Async Broadcast of project schema may cause creating cube failed

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Metadata
• Assignee: Wang Rupeng
• Reporter: Shaohui Liu
• Created: 29/May/19 04:29
• Updated: 29/Jul/19 07:47

Description

In our prod cluster, we found some creating cube requests failed for the model not found.

The problem is that users will create the cube right after creating the model success. But the the two requests may be routed to two different servers.

When the other server receive creating cube request, the project schema may be not updated for the async Broadcast and the server can not found the model related to the cube.

the log at query server 1

kylin.log.11:2019-05-27 10:26:44,143 INFO  [http-bio-7070-exec-962] model.DataModelManager:248 : Saving Model model_k1_bb_83_uyyyyyyy3636 to Project BigBI_Hive with bigbi_kylin as owner
kylin.log.11:2019-05-27 10:26:44,144 INFO  [http-bio-7070-exec-962] model.DataModelManager:185 : Model model_k1_bb_83_uyyyyyyy3636 is missing or unloaded yet
kylin.log.11:2019-05-27 10:26:44,145 INFO  [http-bio-7070-exec-962] persistence.ResourceStore:309 : Update resource: /model_desc/model_k1_bb_83_uyyyyyyy3636.json with content:

and the log at query server 2

2019-05-27 10:26:44,296 WARN  [http-bio-7070-exec-132] cube.CubeDescManager:195 : Broken cube desc CubeDesc [name=cube_b_bb_83_uyyyyyyy3636]
java.lang.NullPointerException: DateModelDesc(model_k1_bb_83_uyyyyyyy3636) not found
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:235)
        at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:664)
        at org.apache.kylin.cube.CubeDescManager.createCubeDesc(CubeDescManager.java:193)
        at org.apache.kylin.rest.service.CubeService.createCubeAndDesc(CubeService.java:216)
        at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
        at org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$20946622.createCubeAndDesc(<generated>)
        at org.apache.kylin.rest.controller.CubeController.saveCubeDesc(CubeController.java:735)
        at sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)