Details
Description
A StringIndexOutOfBoundsException is thrown in some cases during event publication.
Example: a stopVirtualMachine API request is executed, and fails with:
2016-04-15 09:24:43,080 ERROR [o.a.c.f.m.MessageDispatcher] (catalina-exec-1:ctx-840cbaa7 ctx-8daf0e9c ctx-f63af073) Unexpected exception when calling com.cloud.api.ApiServer.handleAsyncJobPublishEvent java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor307.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.cloudstack.framework.messagebus.MessageDispatcher.dispatch(MessageDispatcher.java:75) at org.apache.cloudstack.framework.messagebus.MessageDispatcher.onPublishMessage(MessageDispatcher.java:45) at org.apache.cloudstack.framework.messagebus.MessageBusBase$SubscriptionNode.notifySubscribers(MessageBusBase.java:441) at org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(MessageBusBase.java:178) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.publishOnEventBus(AsyncJobManagerImpl.java:1052) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:180) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:168) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:687) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:528) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:296) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:127) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:124) at com.cloud.api.ApiServlet.doGet(ApiServlet.java:86) at javax.servlet.http.HttpServlet.service(HttpServlet.java:620) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1720) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -468 at java.lang.String.substring(String.java:1967) at com.cloud.api.ApiServer.handleAsyncJobPublishEvent(ApiServer.java:270) at sun.reflect.GeneratedMethodAccessor307.invoke(Unknown Source) ... 41 more
The VM is not stopped, and the API returns the following error:
HTTP/1.1 530 : InvocationTargetException when invoking event handler for subject: job.eventpublish
The error comes from JSON parsing in https://github.com/apache/cloudstack/blob/339355594ca537ebbc8e0a0f6e49a63ee4b884ac/server/src/com/cloud/api/ApiServer.java#L270
The substring method throws a StringIndexOutOfBoundsException when cmdEventType is the last field in the JSON object.
The JSON object being the JSON serialization of a HashMap object, the order of the fields is not guaranteed. In my tests, I observed that there is no error with Java 7, but it fails with Java 8, which has a different Hashmap implementation.
For example, with request http://127.0.0.1:8096/client/api?command=stopVirtualMachine&id=734ef752-fe99-4d46-8f31-18130496473c, cmdInfo is:
{"ctxUserId":"1","httpmethod":"GET","ctxStartEventId":"105","id":"734ef752-fe99-4d46-8f31-18130496473c","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"734ef752-fe99-4d46-8f31-18130496473c\"}","ctxAccountId":"1","uuid":"734ef752-fe99-4d46-8f31-18130496473c","cmdEventType":"VM.STOP"}
and it fails.
Please note that the error is not present for all API requests, depending on the parameters fields.