Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Invalid
-
1.2.0
-
None
-
kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}kubectl version Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Description
Guacamole fails loading the main page, due to failing API calls (HTTP 403 / 405) at:
- https://example.com/api/session/data/null/users/null
HTTP_GET, HTTP 403 - https://example.com/api/tokens
HTTP_POST, HTTP 405
Once timeout occurs, an error page is displayed:
Utilized image is guacamole/guacamole:latest (1.2.0). It was working yesterday; since upgrading Istio from 1.7.4 to 1.8.1 all pods, but guacamole work fine. No errors occur within logs.
guacamole-app
kubectl logs -f pod/guacamole-app-5544ffbf4c-f5zvf -n guacamole guacamole-app 23-Dec-2020 16:20:16.994 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.41 23-Dec-2020 16:20:16.996 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: May 4 2019 09:17:16 UTC 23-Dec-2020 16:20:16.997 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.5.41.0 23-Dec-2020 16:20:16.997 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux 23-Dec-2020 16:20:16.997 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.8.0-33-generic 23-Dec-2020 16:20:16.997 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64 23-Dec-2020 16:20:16.997 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-8-openjdk-amd64/jre 23-Dec-2020 16:20:16.998 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01 23-Dec-2020 16:20:16.998 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation 23-Dec-2020 16:20:16.998 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /usr/local/tomcat 23-Dec-2020 16:20:16.998 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/local/tomcat 23-Dec-2020 16:20:16.999 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties 23-Dec-2020 16:20:16.999 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 23-Dec-2020 16:20:16.999 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048 23-Dec-2020 16:20:16.999 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources 23-Dec-2020 16:20:16.999 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 23-Dec-2020 16:20:17.000 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs= 23-Dec-2020 16:20:17.000 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat 23-Dec-2020 16:20:17.000 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat 23-Dec-2020 16:20:17.000 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp 23-Dec-2020 16:20:17.000 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.21] using APR version [1.5.2]. 23-Dec-2020 16:20:17.001 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 23-Dec-2020 16:20:17.001 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 23-Dec-2020 16:20:17.008 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.0j 20 Nov 2018] 23-Dec-2020 16:20:17.129 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"] 23-Dec-2020 16:20:17.145 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read 23-Dec-2020 16:20:17.162 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"] 23-Dec-2020 16:20:17.163 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read 23-Dec-2020 16:20:17.165 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 822 ms 23-Dec-2020 16:20:17.188 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina] 23-Dec-2020 16:20:17.188 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.41 23-Dec-2020 16:20:17.215 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/local/tomcat/webapps/guacamole.war] 23-Dec-2020 16:20:19.471 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. 16:20:19.725 [localhost-startStop-1] INFO o.a.g.environment.LocalEnvironment - GUACAMOLE_HOME is "/root/.guacamole". 16:20:19.992 [localhost-startStop-1] INFO o.a.g.rest.auth.HashTokenSessionMap - Sessions will expire after 60 minutes of inactivity. 16:20:20.260 [localhost-startStop-1] INFO o.a.g.environment.LocalEnvironment - GUACAMOLE_HOME is "/root/.guacamole". 16:20:21.652 [localhost-startStop-1] INFO o.a.g.extension.ExtensionModule - Extension "MySQL Authentication" loaded. 16:20:21.654 [localhost-startStop-1] INFO o.a.g.environment.LocalEnvironment - GUACAMOLE_HOME is "/root/.guacamole". 16:20:21.776 [localhost-startStop-1] INFO o.a.g.t.w.WebSocketTunnelModule - Loading JSR-356 WebSocket support... 23-Dec-2020 16:20:22.255 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.register Registering org.apache.guacamole.rest.RESTExceptionMapper as a provider class 23-Dec-2020 16:20:22.256 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.register Registering org.apache.guacamole.rest.extension.ExtensionRESTService as a root resource class 23-Dec-2020 16:20:22.256 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.register Registering org.apache.guacamole.rest.language.LanguageRESTService as a root resource class 23-Dec-2020 16:20:22.257 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.register Registering org.apache.guacamole.rest.patch.PatchRESTService as a root resource class 23-Dec-2020 16:20:22.257 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.register Registering org.apache.guacamole.rest.auth.TokenRESTService as a root resource class 23-Dec-2020 16:20:22.257 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.register Registering org.apache.guacamole.rest.session.SessionRESTService as a root resource class 23-Dec-2020 16:20:22.257 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.register Registering org.codehaus.jackson.jaxrs.JacksonJsonProvider as a provider class 23-Dec-2020 16:20:22.262 INFO [localhost-startStop-1] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.17.1 02/28/2013 12:47 PM' 23-Dec-2020 16:20:22.405 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.getComponentProvider Binding org.apache.guacamole.rest.RESTExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton" 23-Dec-2020 16:20:22.408 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.getComponentProvider Binding org.codehaus.jackson.jaxrs.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton" 23-Dec-2020 16:20:23.098 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.getComponentProvider Binding org.apache.guacamole.rest.extension.ExtensionRESTService to GuiceManagedComponentProvider with the scope "PerRequest" 23-Dec-2020 16:20:23.108 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.getComponentProvider Binding org.apache.guacamole.rest.language.LanguageRESTService to GuiceManagedComponentProvider with the scope "PerRequest" 23-Dec-2020 16:20:23.110 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.getComponentProvider Binding org.apache.guacamole.rest.patch.PatchRESTService to GuiceManagedComponentProvider with the scope "PerRequest" 23-Dec-2020 16:20:23.116 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.getComponentProvider Binding org.apache.guacamole.rest.auth.TokenRESTService to GuiceManagedComponentProvider with the scope "PerRequest" 23-Dec-2020 16:20:23.120 INFO [localhost-startStop-1] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory.getComponentProvider Binding org.apache.guacamole.rest.session.SessionRESTService to GuiceManagedComponentProvider with the scope "PerRequest" 23-Dec-2020 16:20:23.150 INFO [localhost-startStop-1] org.webjars.servlet.WebjarsServlet.init WebjarsServlet initialization completed 23-Dec-2020 16:20:23.181 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/usr/local/tomcat/webapps/guacamole.war] has finished in [5,965] ms 23-Dec-2020 16:20:23.182 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/host-manager] 23-Dec-2020 16:20:23.226 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/host-manager] has finished in [44] ms 23-Dec-2020 16:20:23.226 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/ROOT] 23-Dec-2020 16:20:23.248 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/ROOT] has finished in [21] ms 23-Dec-2020 16:20:23.248 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/docs] 23-Dec-2020 16:20:23.269 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/docs] has finished in [21] ms 23-Dec-2020 16:20:23.269 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/examples] 23-Dec-2020 16:20:23.538 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/examples] has finished in [269] ms 23-Dec-2020 16:20:23.539 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/manager] 23-Dec-2020 16:20:23.568 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/manager] has finished in [30] ms 23-Dec-2020 16:20:23.585 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"] 23-Dec-2020 16:20:23.624 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"] 23-Dec-2020 16:20:23.627 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 6462 ms
guacamole-daemon
kubectl logs -f pod/guacamole-daemon-865f5d4476-z479l -n guacamole guacamole-daemon guacd[6]: INFO: Guacamole proxy daemon (guacd) version 1.2.0 started guacd[6]: INFO: Listening on host 0.0.0.0, port 4822
Environment
Routing into the cluster is realized using istio. Setup has been rolled-out using automation server (jenkins). Same roll-out worked out yesterday.
kubectl get all -o wide -n guacamole Kubernetes-Master: Wed Dec 23 17:53:28 2020 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/guacamole-app-5544ffbf4c-f5zvf 2/2 Running 0 33m 192.168.0.22 kubernetes-slave02 <none> <none> pod/guacamole-daemon-865f5d4476-z479l 2/2 Running 0 33m 192.168.4.224 kubernetes-slave <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/guacamole-app ClusterIP 10.105.111.53 <none> 8080/TCP 33m run=guacamole-app service/guacamole-daemon ClusterIP 10.101.198.81 <none> 4822/TCP 33m run=guacamole-daemon NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/guacamole-app 1/1 1 1 33m guacamole-app guacamole/guacamole:latest run=guacamole-app deployment.apps/guacamole-daemon 1/1 1 1 33m guacamole-daemon guacamole/guacd run=guacamole-daemon NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/guacamole-app-5544ffbf4c 1 1 1 33m guacamole-app guacamole/guacamole:latest pod-template-hash=5544ffbf4c,run=guacamole-app replicaset.apps/guacamole-daemon-865f5d4476 1 1 1 33m guacamole-daemon guacamole/guacd pod-template-hash=865f5d4476,run=guacamole-daemon