Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-11235

Sending Continuous Query with absent class on client knocks out server node

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 2.7
    • None
    • None

    Description

      In 2.7, when you create a Continuous Query on client with class which is not present in server classpath, server node will be terminated immediately.

      [2019-02-05 18:20:22,508][ERROR][tcp-disco-msg-worker-#2][root] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to unmarshal discovery custom message: TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=18424a1f-a22a-4401-83f7-2040a30e28f0, id=992de3eb861-18424a1f-a22a-4401-83f7-2040a30e28f0, verifierNodeId=61ff15f0-6337-4e00-ab29-7173a0fb0e1f, topVer=2, pendingIdx=0, failedNodes=null, isClient=false]]]]
      class org.apache.ignite.IgniteException: Failed to unmarshal discovery custom message: TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=18424a1f-a22a-4401-83f7-2040a30e28f0, id=992de3eb861-18424a1f-a22a-4401-83f7-2040a30e28f0, verifierNodeId=61ff15f0-6337-4e00-ab29-7173a0fb0e1f, topVer=2, pendingIdx=0, failedNodes=null, isClient=false]]
      

      This means that one rogue client (e.g. with mismatched code base version) can knock whole cluster offline.

      I think we should disconnect client from topology instead when it sends incorrect discovery message.

      In 2.6, if you tried to create a Continuous Query on client with class which is not present in server classpath, you will get the following erorr on server:

      [15:32:57,844][SEVERE][tcp-disco-msg-worker-#2][TcpDiscoverySpi] Failed to unmarshal discovery custom message.
      class org.apache.ignite.IgniteCheckedException: Failed to find class with given class loader for unmarshalling (make sure same versions of all classes are available on all nodes or enable peer-class-loading) [clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, cls=com.pany.CQTest$1]
              at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:144)
              at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
              at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:161)
              at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
              at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9962)
              at org.apache.ignite.spi.discovery.tcp.messages.TcpDiscoveryCustomEventMessage.message(TcpDiscoveryCustomEventMessage.java:81)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:5476)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processCustomMessage(ServerImpl.java:5305)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2765)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
              at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
      Caused by: java.lang.ClassNotFoundException: com.pany.CQTest$1
              at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
              at java.lang.Class.forName0(Native Method)
              at java.lang.Class.forName(Class.java:348)
              at org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:8608)
              at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.resolveClass(JdkMarshallerObjectInputStream.java:59)
              at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
              at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
              at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandlerV2.readExternal(CacheContinuousQueryHandlerV2.java:169)
              at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2076)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2025)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
              at org.apache.ignite.internal.processors.continuous.StartRequestData.readExternal(StartRequestData.java:260)
              at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2076)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2025)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
              at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:139)
              ... 12 more
      [15:32:57,846][SEVERE][tcp-disco-msg-worker-#2][TcpDiscoverySpi] Failed to unmarshal discovery custom message.
      class org.apache.ignite.IgniteCheckedException: Failed to find class with given class loader for unmarshalling (make sure same versions of all classes are available on all nodes or enable peer-class-loading) [clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, cls=com.pany.CQTest$1]
              at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:144)
              at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
              at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:161)
              at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
              at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9962)
              at org.apache.ignite.spi.discovery.tcp.messages.TcpDiscoveryCustomEventMessage.message(TcpDiscoveryCustomEventMessage.java:81)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processCustomMessage(ServerImpl.java:5323)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2765)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
              at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
              at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
      Caused by: java.lang.ClassNotFoundException: com.pany.CQTest$1
              at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
              at java.lang.Class.forName0(Native Method)
              at java.lang.Class.forName(Class.java:348)
              at org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:8608)
              at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.resolveClass(JdkMarshallerObjectInputStream.java:59)
              at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
              at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
              at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandlerV2.readExternal(CacheContinuousQueryHandlerV2.java:169)
              at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2076)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2025)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
              at org.apache.ignite.internal.processors.continuous.StartRequestData.readExternal(StartRequestData.java:260)
              at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2076)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2025)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
              at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:139)
              ... 11 more
      

      and client thread will wait forever:

      "main" #1 prio=5 os_prio=0 tid=0x00007f403800d000 nid=0x1fa2 waiting on condition [0x00007f403e74a000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
      	at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
      	at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
      	at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.executeQuery0(CacheContinuousQueryManager.java:694)
      	at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.executeQuery(CacheContinuousQueryManager.java:517)
      	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.queryContinuous(IgniteCacheProxyImpl.java:569)
      	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:658)
      	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:344)
      

      But otherwise both client and server are intact and no hanging PME occurs.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ilyak Ilya Kasnacheev
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: