[KYLIN-4136] Real time streaming segment building may save the same cube segment twice in some case

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: v3.0.0
• Fix Version/s: Future
• Assignee: wangxiaojing
• Reporter: wangxiaojing
• Created: 13/Aug/19 13:05
• Updated: 30/Sep/19 08:44

Description

Real time streaming segment building may save the same cube segment whit new state twice in some case , that 's lead to can't build the segment success.

// exception
2019-08-13 20:22:09,644 ERROR [http-bio-8081-exec-64] coordinator.Coordinator:1127 : streaming job submit fail, cubeName:rt_common_task_state_change_v3_cl2 segment:20190809195000_20190809200000
java.lang.IllegalStateException: Segments overlap: rt_common_task_state_change_v3_cl2[20190809195000_20190809200000] and rt_common_task_state_change_v3_cl2[20190809195000_20190809200000]
at org.apache.kylin.metadata.model.Segments.validate(Segments.java:426)
at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:388)
at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:301)
at org.apache.kylin.cube.CubeManager$SegmentAssist.appendSegment(CubeManager.java:742)
at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:662)
at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:650)
at org.apache.kylin.stream.coordinator.Coordinator.triggerSegmentBuild(Coordinator.java:1110)
at org.apache.kylin.stream.coordinator.Coordinator.tryFindAndBuildSegment(Coordinator.java:982)
at org.apache.kylin.stream.coordinator.Coordinator.segmentRemoteStoreComplete(Coordinator.java:294)
at org.apache.kylin.rest.service.StreamingCoordinatorService.onSegmentRemoteStoreComplete(StreamingCoordinatorService.java:114)
at org.apache.kylin.rest.controller.StreamingCoordinatorController.segmentRemoteStoreComplete(StreamingCoordinatorController.java:184)
at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)

//debug