[IGNITE-8437] Control utility fails to connect to cluster if zookeeper discovery used

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Done
• Affects Version/s: 2.5
• Fix Version/s: 2.7
• Component/s: control.sh, zookeeper
• Assignee: Alexey Scherbakov
• Reporter: Dmitry Sherstobitov
• Created: 04/May/18 10:53
• Updated: 15/Jul/20 10:34
• Resolved: 10/May/18 09:21

Description

Start cluster with zookeeper discovery and try to run control.sh --tx utility

 

2018-05-03 16:56:36.225 [ERROR][mgmt-#115268%DPL_GRID%DplGridNodeName%][o.a.i.i.p.r.p.t.GridTcpRestProtocol] Failed to process client request [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=395 lim=395 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-rest-0, igniteInstanceName=DPL_GRID%DplGridNodeName, finished=false, hashCode=1766410348, interrupted=false, runner=grid-nio-worker-tcp-rest-0-#48%DPL_GRID%DplGridNodeName%]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/10.116.158.48:11211, rmtAddr=/10.78.10.31:55847, createTime=1525355795984, closeTime=1525355796217, bytesSent=521553, bytesRcvd=461, bytesSent0=521553, bytesRcvd0=461, sndSchedTime=1525355796217, lastSndTime=1525355796075, lastRcvTime=1525355796175, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser [marsh=JdkMarshaller [clsFilter=o.a.i.i.IgniteKernal$5@3e3d1d0d], routerClient=false], directMode=false]], accepted=true]], msg=GridClientTaskRequest [taskName=o.a.i.i.v.tx.VisorTxTask, arg=VisorTaskArgument [debug=false]]]
org.apache.ignite.internal.util.nio.GridNioException: class org.apache.ignite.IgniteCheckedException: Failed to serialize object: GridClientResponse [clientId=587ea745-dd1e-4631-aa85-feb5d49acc36, reqId=2, destId=null, status=0, errMsg=null, result=GridClientTaskResultBean [res={ZookeeperClusterNode [id=c4cc818d-b29f-427b-86b5-9a625287feb6, addrs=[10.116.159.100], order=30, loc=false, client=true]=VisorTxTaskResult []}, error=null, finished=true, id=~a7245b33-a37a-4084-a954-460c31834442]]
at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionWrite(GridNioCodecFilter.java:100)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:121)
at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionWrite(GridNioFilterChain.java:269)
at org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionWrite(GridNioFilterChain.java:192)
at org.apache.ignite.internal.util.nio.GridNioSessionImpl.send(GridNioSessionImpl.java:110)
at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:261)
at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:232)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:165)
at org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:162)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler$2.apply(GridTaskCommandHandler.java:294)
at org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler$2.apply(GridTaskCommandHandler.java:257)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at org.apache.ignite.internal.processors.task.GridTaskWorker.finishTask(GridTaskWorker.java:1617)
at org.apache.ignite.internal.processors.task.GridTaskWorker.finishTask(GridTaskWorker.java:1585)
at org.apache.ignite.internal.processors.task.GridTaskWorker.reduce(GridTaskWorker.java:1175)
at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:958)
at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1077)
at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1312)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: class org.apache.ignite.IgniteCheckedException: Failed to serialize object: GridClientResponse [clientId=587ea745-dd1e-4631-aa85-feb5d49acc36, reqId=2, destId=null, status=0, errMsg=null, result=GridClientTaskResultBean [res={ZookeeperClusterNode [id=c4cc818d-b29f-427b-86b5-9a625287feb6, addrs=[10.116.159.100], order=30, loc=false, client=true]=VisorTxTaskResult []}, error=null, finished=true, id=~a7245b33-a37a-4084-a954-460c31834442]]
at org.apache.ignite.internal.client.marshaller.optimized.GridClientOptimizedMarshaller.marshal(GridClientOptimizedMarshaller.java:101)
at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestParser.encode(GridTcpRestParser.java:235)
at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionWrite(GridNioCodecFilter.java:95)
... 49 common frames omitted
Caused by: org.apache.ignite.IgniteCheckedException: Failed to serialize object: GridClientResponse [clientId=587ea745-dd1e-4631-aa85-feb5d49acc36, reqId=2, destId=null, status=0, errMsg=null, result=GridClientTaskResultBean [res={ZookeeperClusterNode [id=c4cc818d-b29f-427b-86b5-9a625287feb6, addrs=[10.116.159.100], order=30, loc=false, client=true]=VisorTxTaskResult []}, error=null, finished=true, id=~a7245b33-a37a-4084-a954-460c31834442]]
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.marshal0(OptimizedMarshaller.java:206)
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10044)
at org.apache.ignite.internal.client.marshaller.optimized.GridClientOptimizedMarshaller.marshal(GridClientOptimizedMarshaller.java:88)
... 51 common frames omitted
Caused by: java.io.IOException: Failed to serialize object [typeName=org.apache.ignite.internal.processors.rest.client.message.GridClientResponse]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:244)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObjectOverride(OptimizedObjectOutputStream.java:158)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.marshal0(OptimizedMarshaller.java:201)
... 54 common frames omitted
Caused by: java.io.IOException: Failed to serialize object [typeName=org.apache.ignite.internal.processors.rest.client.message.GridClientTaskResultBean]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:244)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObjectOverride(OptimizedObjectOutputStream.java:158)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at org.apache.ignite.internal.processors.rest.client.message.GridClientResponse.writeExternal(GridClientResponse.java:103)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeExternalizable(OptimizedObjectOutputStream.java:317)
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.write(OptimizedClassDescriptor.java:814)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:240)
... 57 common frames omitted
Caused by: java.io.IOException: Failed to serialize object [typeName=java.util.TreeMap]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:244)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObjectOverride(OptimizedObjectOutputStream.java:158)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at org.apache.ignite.internal.processors.rest.client.message.GridClientTaskResultBean.writeExternal(GridClientTaskResultBean.java:106)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeExternalizable(OptimizedObjectOutputStream.java:317)
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.write(OptimizedClassDescriptor.java:814)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:240)
... 63 common frames omitted
Caused by: java.io.IOException: java.io.IOException: Failed to register class: org.apache.ignite.spi.discovery.zk.internal.ZookeeperClusterNode
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeSerializable(OptimizedObjectOutputStream.java:345)
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.write(OptimizedClassDescriptor.java:827)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:240)
... 69 common frames omitted
Caused by: java.io.IOException: Failed to register class: org.apache.ignite.spi.discovery.zk.internal.ZookeeperClusterNode
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:207)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:199)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObjectOverride(OptimizedObjectOutputStream.java:158)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.TreeMap.writeObject(TreeMap.java:2437)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeSerializable(OptimizedObjectOutputStream.java:339)
... 71 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.ignite.internal.MarshallerContextImpl.registerClassName(MarshallerContextImpl.java:280)
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:204)
... 80 common frames omitted

Issue Links

• is related to: IGNITE-13165 NPE when control utility connect to cluster (VisorTxTask)
• links to: GitHub Pull Request #3950
• links to: TC
• links to: TC Run all