Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-3440

Ignite Services: ServiceTopologyCallable is executed before system cache is started

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.6
    • 1.7
    • None
    • None

    Description

      In rare cases it's possible to get the following exception

      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-0] 2016-07-01 16:03:54,120 22971 INFO  [com.workday.fabric.ignite.management.QuorumEventDispatcher] Storage count changed - oldvalue: 3 - newvalue: 4 
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 2016-07-01 16:03:54,242 1492  ERROR [org.apache.ignite.internal.processors.job.GridJobWorker] Failed to execute job [jobId=ac1b337a551-b26cd969-888f-4580-a2cb-ad10019da322, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=o.a.i.i.processors.service.GridServiceProcessor$ServiceTopologyCallable, dep=LocalDeployment [super=GridDeployment [ts=1467389033352, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@14dad5dc, clsLdrId=989b337a551-b26cd969-888f-4580-a2cb-ad10019da322, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=o.a.i.i.processors.service.GridServiceProcessor$ServiceTopologyCallable, sesId=9c1b337a551-33dedfed-ef93-405c-af02-cd37d7426df9, startTime=1467389034118, endTime=9223372036854775807, taskNodeId=33dedfed-ef93-405c-af02-cd37d7426df9, clsLdr=sun.misc.Launcher$AppClassLoader@14dad5dc, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, subjId=33dedfed-ef93-405c-af02-cd37d7426df9, mapFut=IgniteFuture [orig=GridFutureAdapter [resFlag=0, res=null, startTime=1467389034228, endTime=0, ignoreInterrupts=false, lsnr=null, state=INIT]]], jobId=ac1b337a551-b26cd969-888f-4580-a2cb-ad10019da322]] 
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 2016-07-01 16:03:54,242 1492  ERROR [org.apache.ignite.internal.processors.job.GridJobWorker] Failed to execute job [jobId=ac1b337a551-b26cd969-888f-4580-a2cb-ad10019da322, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=o.a.i.i.processors.service.GridServiceProcessor$ServiceTopologyCallable, dep=LocalDeployment [super=GridDeployment [ts=1467389033352, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@14dad5dc, clsLdrId=989b337a551-b26cd969-888f-4580-a2cb-ad10019da322, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=o.a.i.i.processors.service.GridServiceProcessor$ServiceTopologyCallable, sesId=9c1b337a551-33dedfed-ef93-405c-af02-cd37d7426df9, startTime=1467389034118, endTime=9223372036854775807, taskNodeId=33dedfed-ef93-405c-af02-cd37d7426df9, clsLdr=sun.misc.Launcher$AppClassLoader@14dad5dc, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, subjId=33dedfed-ef93-405c-af02-cd37d7426df9, mapFut=IgniteFuture [orig=GridFutureAdapter [resFlag=0, res=null, startTime=1467389034228, endTime=0, ignoreInterrupts=false, lsnr=null, state=INIT]]], jobId=ac1b337a551-b26cd969-888f-4580-a2cb-ad10019da322]] 
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] class org.apache.ignite.IgniteException: null
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] class org.apache.ignite.IgniteException: null
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2V2.execute(GridClosureProcessor.java:2007)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2V2.execute(GridClosureProcessor.java:2007)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6484)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6484)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1101)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1101)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1772)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1772)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1058)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1058)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:836)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:836)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:104)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:104)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:799)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:799)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at java.lang.Thread.run(Thread.java:745)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at java.lang.Thread.run(Thread.java:745)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] Caused by: java.lang.NullPointerException
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] Caused by: java.lang.NullPointerException
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.service.GridServiceProcessor.serviceTopology(GridServiceProcessor.java:662)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.service.GridServiceProcessor.serviceTopology(GridServiceProcessor.java:662)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.service.GridServiceProcessor.access$3800(GridServiceProcessor.java:106)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.service.GridServiceProcessor.access$3800(GridServiceProcessor.java:106)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.service.GridServiceProcessor$ServiceTopologyCallable.call(GridServiceProcessor.java:1798)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.service.GridServiceProcessor$ServiceTopologyCallable.call(GridServiceProcessor.java:1798)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.service.GridServiceProcessor$ServiceTopologyCallable.call(GridServiceProcessor.java:1774)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.service.GridServiceProcessor$ServiceTopologyCallable.call(GridServiceProcessor.java:1774)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2V2.execute(GridClosureProcessor.java:2004)
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2V2.execute(GridClosureProcessor.java:2004)
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      [FabricRunner-7] 	... 14 more
      
      [16:03:54] :		 [:fabric-tests:test]     [FabricRunner-7] 	... 14 more
      [16:03:54] :				 [testQuorumEvents] [Test Output]
      

      The exception happens because ServiceTopologyCallable gets access to the system cache that hasn't been started yet. The callable has to postpone its execution until the cache is up and ready.

      Attachments

        1. ignite-3440.diff
          6 kB
          Denis A. Magda

        Issue Links

          Activity

            People

              dmagda Denis A. Magda
              dmagda Denis A. Magda
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: