Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-10013

Node restart may lead to NPE in GridDhtPartitionsExchangeFuture

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.7
    • Fix Version/s: 2.7
    • Component/s: general
    • Labels:
      None

      Description

      1. Start 4 nodes with ~100 caches, create a few dynamic caches
      2. Load data (~500 entries per cache)
      3. Restart every node starting from first
      The first node may throw NPE:

      [20:00:18,355][INFO][exchange-worker-#63][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=c7a332ee-ee85-4604-aa93-919f15e34e36, crdVer=1540497597005, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=cc8ed618-2f14-46cf-be03-645291c397aa, customEvt=null, allowMerge=true]
      [20:00:18,358][INFO][exchange-worker-#63][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], err=null]
      [20:00:18,364][SEVERE][sys-#77][GridCacheIoManager] Failed processing message [senderId=c50165fb-1afd-4b3e-8fba-ac825aa532fa, msg=GridDhtPartitionsSingleMessage [parts=HashMap {-553318226=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553318133=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], -553318412=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553315529=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553315497=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553315498=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=100], -553318319=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], 374280891=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=32], 374280890=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=64], 374280889=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=128], 374280888=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], 374280887=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=23], 374280886=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=31], 374280885=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=61], 374280884=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553315527=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], -553315495=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], -553315528=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=4, size=490], -553315496=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=4, size=512], -553315526=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=4, size=512]}, partCntrs=null, partsSizes=HashMap {-553318226=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553318133=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 8=1, 9=1, 10=1, 11=1, 12=1, 13=1, 14=1, 15=1, 16=1, 17=1, 18=1, 19=1, 20=1, 21=1, 22=1, 23=1, 24=1, 25=1, 26=1, 27=1, 28=1, 29=1, 30=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 37=1, 38=1, 39=1, 40=1, 41=1, 42=1, 43=1, 44=1, 45=1, 46=1, 47=1, 48=1, 49=1, 50=1, 51=1, 52=1, 53=1, 54=1, 55=1, 56=1, 57=1, 58=1, 59=1, 60=1, 61=1, 62=1, 63=1, 64=1, 65=1, 66=1, 67=1, 68=1, 69=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}, -553318412=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553315529=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553315497=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553315498=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553318319=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 8=1, 9=1, 10=1, 11=1, 12=1, 13=1, 14=1, 15=1, 16=1, 17=1, 18=1, 19=1, 20=1, 21=1, 22=1, 23=1, 24=1, 25=1, 26=1, 27=1, 28=1, 29=1, 30=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 37=1, 38=1, 39=1, 40=1, 41=1, 42=1, 43=1, 44=1, 45=1, 46=1, 47=1, 48=1, 49=1, 50=1, 51=1, 52=1, 53=1, 54=1, 55=1, 56=1, 57=1, 58=1, 59=1, 60=1, 61=1, 62=1, 63=1, 64=1, 65=1, 66=1, 67=1, 68=1, 69=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}, 374280891=HashMap {0=18, 1=24, 2=24, 3=24, 4=24, 5=18, 6=18, 7=18, 8=18, 9=18, 10=18, 11=18, 12=18, 13=18, 14=18, 15=18, 16=18, 17=18, 18=18, 19=18, 20=18, 21=18, 22=18, 23=18, 24=18, 25=18, 26=18, 27=18, 28=18, 29=18, 30=18, 31=18}, 374280890=HashMap {0=26, 1=52, 2=52, 3=52, 4=52, 5=52, 6=52, 7=52, 8=52, 9=52, 10=52, 11=52, 12=52, 13=52, 14=52, 15=52, 16=52, 17=52, 18=52, 19=52, 20=52, 21=52, 22=52, 23=52, 24=52, 25=52, 26=52, 27=52, 28=52, 29=52, 30=52, 31=52, 32=52, 33=52, 34=52, 35=52, 36=52, 37=26, 38=26, 39=26, 40=26, 41=26, 42=26, 43=26, 44=26, 45=26, 46=26, 47=26, 48=26, 49=26, 50=26, 51=26, 52=26, 53=26, 54=26, 55=26, 56=26, 57=26, 58=26, 59=26, 60=26, 61=26, 62=26, 63=26}, 374280889=HashMap {1=14, 2=14, 3=14, 4=14, 5=14, 6=14, 7=14, 8=14, 9=14, 10=14, 11=14, 12=14, 13=14, 14=14, 15=14, 16=14, 17=14, 18=14, 19=14, 20=14, 21=14, 22=14, 23=14, 24=14, 25=14, 26=14, 27=14, 28=14, 29=14, 30=14, 31=14, 32=14, 33=14, 34=14, 35=14, 36=14, 37=14, 38=14, 39=14, 40=14, 41=14, 42=14, 43=14, 44=14, 45=14, 46=14, 47=14, 48=14, 49=14, 50=14, 51=14, 52=14, 53=14, 54=14, 55=14, 56=14, 57=14, 58=14, 59=14, 60=14, 61=14, 62=14, 63=14, 64=14, 65=14, 66=14, 67=14, 68=14, 69=14, 70=14, 71=14, 72=14, 73=14, 74=14, 75=14, 76=14, 77=14, 78=14, 79=14, 80=14, 81=14, 82=14, 83=14, 84=14, 85=14, 86=14, 87=14, 88=14, 89=14, 90=14, 91=14, 92=14, 93=14, 94=14, 95=14, 96=14, 97=14, 98=14, 99=14, 100=14}, 374280888=HashMap {1=12, 2=12, 3=12, 4=12, 5=12, 6=12, 7=12, 8=12, 9=12, 10=12, 11=12, 12=12, 13=12, 14=12, 15=12, 16=12, 17=12, 18=12, 19=12, 20=12, 21=12, 22=12, 23=12, 24=12, 25=12, 26=12, 27=12, 28=12, 29=12, 30=12, 31=12, 32=12, 33=12, 34=12, 35=12, 36=12, 37=12, 38=12, 39=12, 40=12, 41=12, 42=12, 43=12, 44=12, 45=12, 46=12, 47=12, 48=12, 49=12, 50=12, 51=12, 52=12, 53=12, 54=12, 55=12, 56=12, 57=12, 58=12, 59=12, 60=12, 61=12, 62=12, 63=12, 64=12, 65=12, 66=12, 67=12, 68=12, 69=12, 70=12, 71=12, 72=12, 73=12, 74=12, 75=12, 76=12, 77=12, 78=12, 79=12, 80=12, 81=12, 82=12, 83=12, 84=12, 85=12, 86=12, 87=12, 88=12, 89=12, 90=12, 91=12, 92=12, 93=12, 94=12, 95=12, 96=12, 97=12, 98=12, 99=12, 100=12}, 374280887=HashMap {0=18, 1=24, 2=24, 3=24, 4=24, 5=18, 6=18, 7=18, 9=18, 11=18, 13=18, 15=18, 16=18, 19=18, 20=18, 21=18, 22=18, 24=18, 25=18, 27=18, 28=18, 30=18, 31=18}, 374280886=HashMap {0=26, 1=52, 2=52, 3=52, 4=52, 5=52, 6=52, 7=52, 9=52, 11=52, 13=52, 20=52, 21=52, 22=52, 24=52, 27=52, 28=52, 31=52, 32=52, 33=52, 34=52, 35=52, 36=52, 39=26, 42=26, 43=26, 47=26, 55=26, 57=26, 58=26, 63=26}, 374280885=HashMap {1=14, 2=14, 3=14, 4=14, 5=14, 6=14, 7=14, 9=14, 11=14, 13=14, ... and 3711 skipped ...75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}, -553315528=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553315496=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 8=1, 9=1, 10=1, 11=1, 12=1, 13=1, 14=1, 15=1, 16=1, 17=1, 18=1, 19=1, 20=1, 21=1, 22=1, 23=1, 24=1, 25=1, 26=1, 27=1, 28=1, 29=1, 30=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 37=1, 38=1, 39=1, 40=1, 41=1, 42=1, 43=1, 44=1, 45=1, 46=1, 47=1, 48=1, 49=1, 50=1, 51=1, 52=1, 53=1, 54=1, 55=1, 56=1, 57=1, 58=1, 59=1, 60=1, 61=1, 62=1, 63=1, 64=1, 65=1, 66=1, 67=1, 68=1, 69=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}, -553315526=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 8=1, 9=1, 10=1, 11=1, 12=1, 13=1, 14=1, 15=1, 16=1, 17=1, 18=1, 19=1, 20=1, 21=1, 22=1, 23=1, 24=1, 25=1, 26=1, 27=1, 28=1, 29=1, 30=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 37=1, 38=1, 39=1, 40=1, 41=1, 42=1, 43=1, 44=1, 45=1, 46=1, 47=1, 48=1, 49=1, 50=1, 51=1, 52=1, 53=1, 54=1, 55=1, 56=1, 57=1, 58=1, 59=1, 60=1, 61=1, 62=1, 63=1, 64=1, 65=1, 66=1, 67=1, 68=1, 69=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}}, partHistCntrs=null, err=null, client=false, compress=false, finishMsg=null, activeQryTrackers=null, super=GridDhtPartitionsAbstractMessage [exchId=null, lastVer=GridCacheVersion [topVer=0, order=1540497610955, nodeOrder=0], super=GridCacheMessage [msgId=19, depInfo=null, err=null, skipPrepare=false]]]]
      java.lang.NullPointerException
      	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.partitionChangesInProgress(GridDhtPartitionsExchangeFuture.java:4577)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.exchangeInProgress(GridCachePartitionExchangeManager.java:2307)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1000(GridCachePartitionExchangeManager.java:142)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:350)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:333)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3102)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3081)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
      	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      [20:00:18,365][INFO][exchange-worker-#63][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=c7a332ee-ee85-4604-aa93-919f15e34e36, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=cc8ed618-2f14-46cf-be03-645291c397aa, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=5, intOrder=5, lastExchangeTime=1540497616253, loc=false, ver=2.7.1#20181023-sha1:b29a1bd9, isClient=true], done=true], topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], durationFromInit=23]
      [20:00:18,372][SEVERE][sys-#77][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]]
      java.lang.NullPointerException
      	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.partitionChangesInProgress(GridDhtPartitionsExchangeFuture.java:4577)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.exchangeInProgress(GridCachePartitionExchangeManager.java:2307)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1000(GridCachePartitionExchangeManager.java:142)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:350)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:333)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3102)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3081)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
      	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      [20:00:18,373][WARNING][sys-#77][FailureProcessor] No deadlocked threads detected.
      [20:00:18,527][INFO][exchange-worker-#63][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=true]
      [20:00:18,530][WARNING][sys-#77][FailureProcessor] Thread dump at 2018/10/25 20:00:18 UTC
      Thread [name="srvc-deploy-#88", id=129, state=WAITING, blockCnt=0, waitCnt=1]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7769e8eb, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#87", id=128, state=TIMED_WAITING, blockCnt=1, waitCnt=17]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#86", id=127, state=TIMED_WAITING, blockCnt=5, waitCnt=13]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#85", id=126, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#84", id=125, state=TIMED_WAITING, blockCnt=0, waitCnt=22]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#83", id=124, state=TIMED_WAITING, blockCnt=0, waitCnt=15]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#81", id=123, state=TIMED_WAITING, blockCnt=0, waitCnt=18]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#82", id=122, state=TIMED_WAITING, blockCnt=0, waitCnt=19]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#80", id=121, state=TIMED_WAITING, blockCnt=0, waitCnt=6]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#79", id=120, state=TIMED_WAITING, blockCnt=0, waitCnt=21]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#78", id=119, state=TIMED_WAITING, blockCnt=0, waitCnt=18]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#77", id=118, state=RUNNABLE, blockCnt=2, waitCnt=11]
              at sun.management.ThreadImpl.dumpThreads0(Native Method)
              at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
              at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1364)
              at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:132)
              - locked o.a.i.i.processors.failure.FailureProcessor@26c61451
              at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:108)
              at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1071)
              at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
              at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
              at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
              at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
              at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
              at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
              at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
              at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
              at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
          Locked synchronizers:
              java.util.concurrent.ThreadPoolExecutor$Worker@7ba7092f
      Thread [name="Thread-7", id=117, state=TIMED_WAITING, blockCnt=0, waitCnt=306]
          Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="Thread-5", id=116, state=TIMED_WAITING, blockCnt=0, waitCnt=635]
          Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="Thread-6", id=115, state=TIMED_WAITING, blockCnt=0, waitCnt=643]
          Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#76", id=114, state=TIMED_WAITING, blockCnt=0, waitCnt=12]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="checkpoint-runner-#75", id=113, state=WAITING, blockCnt=2, waitCnt=625]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14a5a8d6, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="checkpoint-runner-#74", id=112, state=WAITING, blockCnt=5, waitCnt=606]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14a5a8d6, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="checkpoint-runner-#73", id=111, state=WAITING, blockCnt=1, waitCnt=617]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14a5a8d6, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="checkpoint-runner-#72", id=110, state=WAITING, blockCnt=5, waitCnt=595]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14a5a8d6, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="db-checkpoint-thread-#71", id=109, state=RUNNABLE, blockCnt=0, waitCnt=5]
              at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
              at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
              at sun.nio.ch.SimpleAsynchronousFileChannelImpl.force(SimpleAsynchronousFileChannelImpl.java:162)
              at o.a.i.i.processors.cache.persistence.file.AsyncFileIO.force(AsyncFileIO.java:183)
              at o.a.i.i.processors.cache.persistence.file.AsyncFileIO.force(AsyncFileIO.java:178)
              at o.a.i.i.processors.cache.persistence.file.FilePageStore.sync(FilePageStore.java:638)
              at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3306)
              at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3096)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
          Locked synchronizers:
              java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@40f5ed36
      Thread [name="sys-#70", id=108, state=TIMED_WAITING, blockCnt=0, waitCnt=21]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#69", id=107, state=TIMED_WAITING, blockCnt=0, waitCnt=21]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="ttl-cleanup-worker-#68", id=106, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
              at java.lang.Thread.sleep(Native Method)
              at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7774)
              at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:149)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-#67", id=105, state=TIMED_WAITING, blockCnt=1, waitCnt=21]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="Thread-4", id=104, state=TIMED_WAITING, blockCnt=0, waitCnt=18477]
          Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="wal-segment-syncer-#66", id=103, state=TIMED_WAITING, blockCnt=0, waitCnt=12]
              at java.lang.Thread.sleep(Native Method)
              at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7774)
              at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$WalSegmentSyncer.body(FileWriteAheadLogManager.java:3514)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="wal-file-archiver%null-#65", id=102, state=WAITING, blockCnt=1, waitCnt=2]
          Lock [object=o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage@56871de5, ownerName=null, ownerId=-1]
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.awaitSegment(SegmentCurrentStateStorage.java:73)
              at o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.waitNextSegmentForArchivation(SegmentCurrentStateStorage.java:90)
              at o.a.i.i.processors.cache.persistence.wal.aware.SegmentAware.waitNextSegmentForArchivation(SegmentAware.java:79)
              at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager.java:1719)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="tcp-disco-sock-reader-#8", id=101, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
              at java.net.SocketInputStream.read(SocketInputStream.java:171)
              at java.net.SocketInputStream.read(SocketInputStream.java:141)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
              at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
              - locked java.io.BufferedInputStream@6319d7c5
              at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
              at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2653)
              at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2669)
              at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
              at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
              at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
              at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:43)
              at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:137)
              at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
              at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10085)
              at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6246)
              at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
      
      Thread [name="sys-#64", id=100, state=TIMED_WAITING, blockCnt=2, waitCnt=20]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="Keep-Alive-Timer", id=97, state=TIMED_WAITING, blockCnt=0, waitCnt=4]
              at java.lang.Thread.sleep(Native Method)
              at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:172)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="session-timeout-worker-#54", id=79, state=TIMED_WAITING, blockCnt=0, waitCnt=22]
              at java.lang.Thread.sleep(Native Method)
              at o.a.i.i.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:495)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="exchange-worker-#63", id=92, state=RUNNABLE, blockCnt=2, waitCnt=23214]
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionMap.<init>(GridDhtPartitionMap.java:96)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap.<init>(GridDhtPartitionFullMap.java:86)
              at o.a.i.i.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.partitionMap(GridDhtPartitionTopologyImpl.java:1326)
              at o.a.i.i.processors.cache.GridCachePartitionExchangeManager.createPartitionsFullMessage(GridCachePartitionExchangeManager.java:1175)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createPartitionsMessage(GridDhtPartitionsExchangeFuture.java:1900)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$700(GridDhtPartitionsExchangeFuture.java:139)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$3.apply(GridDhtPartitionsExchangeFuture.java:2688)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$3.apply(GridDhtPartitionsExchangeFuture.java:2653)
              at o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
              at o.a.i.i.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitAndReplyToNode(GridDhtPartitionsExchangeFuture.java:2653)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processSingleMessage(GridDhtPartitionsExchangeFuture.java:2708)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:139)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2612)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2600)
              at o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
              at o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
              at o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
              at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
              at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
              at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:2195)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:138)
              at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
              at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:823)
              at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)
              at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="ignite-update-notifier-timer", id=91, state=TIMED_WAITING, blockCnt=1, waitCnt=3]
          Lock [object=java.util.TaskQueue@5ac33068, ownerName=null, ownerId=-1]
              at java.lang.Object.wait(Native Method)
              at java.util.TimerThread.mainLoop(Timer.java:552)
              at java.util.TimerThread.run(Timer.java:505)
      
      Thread [name="upd-ver-checker", id=90, state=TIMED_WAITING, blockCnt=0, waitCnt=5]
              at java.lang.Thread.sleep(Native Method)
              at o.a.i.i.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="disco-event-worker-#62", id=89, state=WAITING, blockCnt=0, waitCnt=18]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35b0ea5b, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2856)
              at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2825)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="tcp-disco-srvr-#3", id=88, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at java.net.PlainSocketImpl.socketAccept(Native Method)
              at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
              at java.net.ServerSocket.implAccept(ServerSocket.java:545)
              at java.net.ServerSocket.accept(ServerSocket.java:513)
              at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5845)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5763)
              at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
      
      Thread [name="tcp-disco-msg-worker-#2", id=87, state=TIMED_WAITING, blockCnt=0, waitCnt=2074]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bb6b13d, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
              at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
              at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7183)
              at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
              at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
      
      Thread [name="disco-notifier-worker-#61", id=86, state=WAITING, blockCnt=0, waitCnt=20]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@763ce271, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2660)
              at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-data-loader-flusher-#60", id=85, state=WAITING, blockCnt=0, waitCnt=1]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35bfdcae, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
              at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-rest-3-#58", id=83, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7ef1a70
              - locked java.util.Collections$UnmodifiableSet@693e7ce9
              - locked sun.nio.ch.EPollSelectorImpl@788024d5
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-rest-2-#57", id=82, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3391ddc7
              - locked java.util.Collections$UnmodifiableSet@4cd27d5f
              - locked sun.nio.ch.EPollSelectorImpl@56119918
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-rest-1-#56", id=81, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@2fceb983
              - locked java.util.Collections$UnmodifiableSet@120c2658
              - locked sun.nio.ch.EPollSelectorImpl@4deb89c3
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-rest-0-#55", id=80, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@17e8003f
              - locked java.util.Collections$UnmodifiableSet@4db0b1e0
              - locked sun.nio.ch.EPollSelectorImpl@3e7d70e8
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="nio-acceptor-tcp-rest-#59", id=84, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked sun.nio.ch.Util$3@2fc94375
              - locked java.util.Collections$UnmodifiableSet@78f7574b
              - locked sun.nio.ch.EPollSelectorImpl@3080c604
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
              at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-client-listener-3-#52", id=77, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@65af1acc
              - locked java.util.Collections$UnmodifiableSet@182fe7
              - locked sun.nio.ch.EPollSelectorImpl@42176da0
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-client-listener-2-#51", id=76, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@74113300
              - locked java.util.Collections$UnmodifiableSet@69752e34
              - locked sun.nio.ch.EPollSelectorImpl@28d343ea
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-client-listener-1-#50", id=75, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5150513d
              - locked java.util.Collections$UnmodifiableSet@2ca4f124
              - locked sun.nio.ch.EPollSelectorImpl@6ce06e0a
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-client-listener-0-#49", id=74, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5e9eac6f
              - locked java.util.Collections$UnmodifiableSet@d14c738
              - locked sun.nio.ch.EPollSelectorImpl@6b23a6f
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="nio-acceptor-client-listener-#53", id=78, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked sun.nio.ch.Util$3@69cd848d
              - locked java.util.Collections$UnmodifiableSet@429ba00f
              - locked sun.nio.ch.EPollSelectorImpl@691d9263
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
              at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="Thread-1", id=73, state=TIMED_WAITING, blockCnt=0, waitCnt=7722]
          Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="tcp-comm-worker-#1", id=72, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e6bbc6d, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4287)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
              at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
      
      Thread [name="grid-nio-worker-tcp-comm-7-#47", id=70, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7b87b6c1
              - locked java.util.Collections$UnmodifiableSet@6403fa66
              - locked sun.nio.ch.EPollSelectorImpl@6108f20f
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-comm-6-#46", id=69, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@42a3b297
              - locked java.util.Collections$UnmodifiableSet@4d425bc
              - locked sun.nio.ch.EPollSelectorImpl@7f7f2f8a
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-comm-5-#45", id=68, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@594122e0
              - locked java.util.Collections$UnmodifiableSet@6a91455
              - locked sun.nio.ch.EPollSelectorImpl@237538f9
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-comm-4-#44", id=67, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@25705062
              - locked java.util.Collections$UnmodifiableSet@7576300e
              - locked sun.nio.ch.EPollSelectorImpl@3bb39949
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-comm-3-#43", id=66, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@310bdc36
              - locked java.util.Collections$UnmodifiableSet@16df6935
              - locked sun.nio.ch.EPollSelectorImpl@526ddda0
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-comm-2-#42", id=65, state=RUNNABLE, blockCnt=9, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@75b37f75
              - locked java.util.Collections$UnmodifiableSet@1a36e799
              - locked sun.nio.ch.EPollSelectorImpl@774da32b
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-comm-1-#41", id=64, state=RUNNABLE, blockCnt=7, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@4bf1f0bf
              - locked java.util.Collections$UnmodifiableSet@25d76663
              - locked sun.nio.ch.EPollSelectorImpl@8f0476f
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-nio-worker-tcp-comm-0-#40", id=63, state=RUNNABLE, blockCnt=4, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked o.a.i.i.util.nio.SelectedSelectionKeySet@12bee9a1
              - locked java.util.Collections$UnmodifiableSet@560e9741
              - locked sun.nio.ch.EPollSelectorImpl@4874179d
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
              at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="nio-acceptor-tcp-comm-#48", id=71, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked sun.nio.ch.Util$3@4235eac3
              - locked java.util.Collections$UnmodifiableSet@2723e0c5
              - locked sun.nio.ch.EPollSelectorImpl@535cfff
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
              at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="grid-timeout-worker-#39", id=62, state=TIMED_WAITING, blockCnt=2, waitCnt=59]
          Lock [object=java.lang.Object@6d15e83b, ownerName=null, ownerId=-1]
              at java.lang.Object.wait(Native Method)
              at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="ignite-clock", id=61, state=TIMED_WAITING, blockCnt=0, waitCnt=2247]
              at java.lang.Thread.sleep(Native Method)
              at o.a.i.i.util.IgniteUtils$22.run(IgniteUtils.java:3379)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="pub-#38", id=60, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="pub-#37", id=59, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="pub-#36", id=58, state=TIMED_WAITING, blockCnt=1, waitCnt=2]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="pub-#35", id=57, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="pub-#34", id=56, state=TIMED_WAITING, blockCnt=1, waitCnt=1]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="pub-#33", id=55, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
          Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="jvm-pause-detector-worker", id=54, state=TIMED_WAITING, blockCnt=0, waitCnt=459]
              at java.lang.Thread.sleep(Native Method)
              at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:100)
      
      Thread [name="data-streamer-stripe-15-#32", id=53, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-14-#31", id=52, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-13-#30", id=51, state=WAITING, blockCnt=2, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-12-#29", id=50, state=WAITING, blockCnt=3, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-11-#28", id=49, state=WAITING, blockCnt=2, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-10-#27", id=48, state=WAITING, blockCnt=2, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-9-#26", id=47, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-8-#25", id=46, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-7-#24", id=45, state=WAITING, blockCnt=2, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-6-#23", id=44, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-5-#22", id=43, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-4-#21", id=42, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-3-#20", id=41, state=WAITING, blockCnt=3, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-2-#19", id=40, state=WAITING, blockCnt=1, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-1-#18", id=39, state=WAITING, blockCnt=2, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="data-streamer-stripe-0-#17", id=38, state=WAITING, blockCnt=2, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-15-#16", id=37, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-14-#15", id=36, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-13-#14", id=35, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-12-#13", id=34, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-11-#12", id=33, state=WAITING, blockCnt=1, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-10-#11", id=32, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-9-#10", id=31, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-8-#9", id=30, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-7-#8", id=29, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-6-#7", id=28, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-5-#6", id=27, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-4-#5", id=26, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-3-#4", id=25, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-2-#3", id=24, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-1-#2", id=23, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="sys-stripe-0-#1", id=22, state=WAITING, blockCnt=0, waitCnt=1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
              at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
              at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
              at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="RMI TCP Accept-0", id=21, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at java.net.PlainSocketImpl.socketAccept(Native Method)
              at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
              at java.net.ServerSocket.implAccept(ServerSocket.java:545)
              at java.net.ServerSocket.accept(ServerSocket.java:513)
              at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
              at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
              at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="RMI TCP Accept-62446", id=20, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at java.net.PlainSocketImpl.socketAccept(Native Method)
              at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
              at java.net.ServerSocket.implAccept(ServerSocket.java:545)
              at java.net.ServerSocket.accept(ServerSocket.java:513)
              at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
              at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="RMI TCP Accept-0", id=19, state=RUNNABLE, blockCnt=0, waitCnt=0]
              at java.net.PlainSocketImpl.socketAccept(Native Method)
              at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
              at java.net.ServerSocket.implAccept(ServerSocket.java:545)
              at java.net.ServerSocket.accept(ServerSocket.java:513)
              at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
              at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
              at java.lang.Thread.run(Thread.java:748)
      
      Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0]
      
      Thread [name="Finalizer", id=3, state=WAITING, blockCnt=87, waitCnt=5]
          Lock [object=java.lang.ref.ReferenceQueue$Lock@380699a6, ownerName=null, ownerId=-1]
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
      
      Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=4, waitCnt=4]
          Lock [object=java.lang.ref.Reference$Lock@1367e36e, ownerName=null, ownerId=-1]
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
              at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
      
      Thread [name="main", id=1, state=WAITING, blockCnt=0, waitCnt=34]
          Lock [object=java.util.concurrent.CountDownLatch$Sync@51f18509, ownerName=null, ownerId=-1]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
              at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
              at o.a.i.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:334)
      
      
      
      [20:00:18,531][SEVERE][sys-#77][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jokser Pavel Kovalenko
                Reporter:
                skozlov Sergey Kozlov
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: