Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-5081

Kylin Can't read datasource from hive-3.1.2

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • v4.0.0
    • None
    • Environment , Metadata
    • None

    Description

      Visit Model -> DataSource -> [Click here to load your table] ,And the error happens:

      web ui as follow:

      tkylin.log as follow:

      2021-09-03 16:57:45,087 INFO  [http-bio-7070-exec-6] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.source.SourceManager
      2021-09-03 16:57:45,133 INFO  [http-bio-7070-exec-6] conf.HiveConf:187 : Found configuration file file:/home/hadoop/hive-3.1.2/conf/hive-site.xml
      2021-09-03 16:57:45,785 ERROR [http-bio-7070-exec-6] controller.TableController:200 : org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V
      java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V
              at org.apache.kylin.source.hive.HiveMetaStoreClientFactory.getHiveMetaStoreClient(HiveMetaStoreClientFactory.java:47)
              at org.apache.kylin.source.hive.CLIHiveClient.getMetaStoreClient(CLIHiveClient.java:164)
              at org.apache.kylin.source.hive.CLIHiveClient.getHiveDbNames(CLIHiveClient.java:118)
              at org.apache.kylin.source.hive.HiveMetadataExplorer.listDatabases(HiveMetadataExplorer.java:51)
              at org.apache.kylin.rest.service.TableService.getSourceDbNames(TableService.java:274)
              at org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:198)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
              at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
              at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
              at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
              at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
              at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
              at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
              at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
              at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
              at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
              at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
              at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
              at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
              at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
              at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
              at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
              at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
              at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
              at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
              at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
              at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
              at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1195)
              at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
              at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
              at java.lang.Thread.run(Thread.java:748)
      2021-09-03 16:57:45,791 ERROR [http-bio-7070-exec-6] controller.BasicController:65 :
      org.apache.kylin.rest.exception.InternalErrorException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V
              at org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:201)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
              at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
              at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
              at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
              at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
              at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
              at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
              at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
              at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
              at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
              at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
              at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
              at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
              at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
              at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
              at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
              at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
              at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
              at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
              at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
              at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
              at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
              at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1195)
              at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
              at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V
              at org.apache.kylin.source.hive.HiveMetaStoreClientFactory.getHiveMetaStoreClient(HiveMetaStoreClientFactory.java:47)
              at org.apache.kylin.source.hive.CLIHiveClient.getMetaStoreClient(CLIHiveClient.java:164)
              at org.apache.kylin.source.hive.CLIHiveClient.getHiveDbNames(CLIHiveClient.java:118)
              at org.apache.kylin.source.hive.HiveMetadataExplorer.listDatabases(HiveMetadataExplorer.java:51)
              at org.apache.kylin.rest.service.TableService.getSourceDbNames(TableService.java:274)
              at org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:198)
              ... 76 more
      2021-09-03 16:57:51,161 DEBUG [BadQueryDetector] service.BadQueryDetector:148 : Detect bad query.
      2021-09-03 16:57:54,023 INFO  [FetcherRunner 387303529-29] threadpool.DefaultFetcherRunner:117 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 0 error, 0 discarded, 0 others
      2021-09-03 16:58:24,023 INFO  [FetcherRunner 387303529-29] threadpool.DefaultFetcherRunner:117 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 0 error, 0 discarded, 0 others
      2021-09-03 16:58:51,161 DEBUG [BadQueryDetector] service.BadQueryDetector:148 : Detect bad query.
      2021-09-03 16:58:54,023 INFO  [FetcherRunner 387303529-29] threadpool.DefaultFetcherRunner:117 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 0 error, 0 discarded, 0 others
      2021-09-03 16:59:24,023 INFO  [FetcherRunner 387303529-29] threadpool.DefaultFetcherRunner:117 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 0 error, 0 discarded, 0 others
      
      

       I can see [org.apache.hadoop.hive.metastore.HiveMetaStoreClient]  loaded by $HIVE_HOME/lib/hive-standalone-metastore-3.1.2.jar

       There is not the class of `org.apache.hadoop.hive.metastore.HiveMetaStoreClient` in the hive-standalone-metastore-3.1.2.jar .

      I have tried to use spark-3.1.2-bin-hadoop3.2/jars/hive-metastore-2.3.7.jar to run kylin,there are also another compatibility issues.

       

       

       

       

      Attachments

        1. setenv.sh
          4 kB
          曹勇
        2. setenv.sh
          4 kB
          曹勇
        3. kylin.sh
          15 kB
          曹勇
        4. kylin.properties
          13 kB
          曹勇
        5. kylin.properties
          13 kB
          曹勇
        6. kylin.out.tar.gz2
          70 kB
          曹勇
        7. kylin.log.tar.gz2
          63 kB
          曹勇
        8. image-2021-09-06-14-01-55-759.png
          15 kB
          Xiaoxiang Yu
        9. image-2021-09-06-13-58-15-676.png
          175 kB
          Xiaoxiang Yu
        10. image-2021-09-06-13-57-27-665.png
          166 kB
          Xiaoxiang Yu
        11. image-2021-09-06-11-02-19-695.png
          98 kB
          曹勇
        12. image-2021-09-03-21-11-55-718.png
          122 kB
          曹勇
        13. image-2021-09-03-21-06-21-865.png
          30 kB
          曹勇
        14. image-2021-09-03-18-40-35-716.png
          125 kB
          曹勇
        15. image-2021-09-03-18-00-06-359.png
          73 kB
          mukvin
        16. image-2021-09-03-17-18-52-134.png
          62 kB
          曹勇
        17. image-2021-09-03-17-04-46-325.png
          50 kB
          曹勇
        18. image-2021-09-03-17-01-39-891.png
          100 kB
          曹勇
        19. image-2021-09-03-16-58-15-453.png
          68 kB
          曹勇
        20. core-site.xml
          2 kB
          曹勇

        Issue Links

          Activity

            People

              mukvin mukvin
              cyxinda 曹勇
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: