[IGNITE-11803] Re-balancing is cancelled if client node joins, reinvestigate.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7
• Labels: rebalance
• Assignee: Maxim Muzafarov
• Reporter: Evgeny Stanilovsky
• Created: 24/Apr/19 09:52
• Updated: 17/Dec/19 05:12

Description

Improvement [1] looks needs to be solved this problem, but in logs i see different behavior.

2019-04-23 15:22:56.351[INFO ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=4c9427d7-5d06-4460-98ad-6586cf177192, addrs=ArrayList [], sockAddrs=
HashSet [, discPort=0, order=159, intOrder=159, lastExchangeTime=1556022094000, loc=false, ver=2.5.1#20190327-sha1:6edfea1b, isClient=true]
2019-04-23 15:22:56.353[INFO ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager] Topology snapshot [ver=159, servers=, clients=1, CPUs=8848, offheap=190000.0GB, heap=4900.0GB]
2019-04-23 15:22:56.353[INFO ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]   ^-- Node [id=80FC83EB-F125-48F0-ACBE-8ADB80A065A3, clusterState=ACTIVE]
2019-04-23 15:22:56.353[INFO ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]   ^-- Baseline [id=0, size=158, online=158, offline=0]
2019-04-23 15:22:56.353[INFO ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager] Data Regions Configured:
2019-04-23 15:22:56.353[INFO ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]   ^-- dpl_mem_plc [initSize=256.0 MiB, maxSize=592.0 GiB, persistenceEnabled=true]
2019-04-23 15:22:56.353[INFO ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]   ^-- not-persisted [initSize=256.0 MiB, maxSize=576.0 GiB, persistenceEnabled=false]
2019-04-23 15:22:56.356[INFO ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.ignite.internal.exchange.time] Started exchange init [topVer=AffinityTopologyVersion [topVer=159, minorTopVer=0], crd=false, evt=NODE_JOINED, evtNode=4c9427d7-5d06
-4460-98ad-6586cf177192, customEvt=null, allowMerge=true]
2019-04-23 15:22:56.393[INFO ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=159, minorTopVer=0], resVer=AffinityTopologyVersion
 [topVer=159, minorTopVer=0], err=null]
2019-04-23 15:22:56.417[INFO ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.ignite.internal.exchange.time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=159, minorTopVer=0], crd=false]
2019-04-23 15:22:56.637[INFO ][wal-file-archiver%DPL_GRID%DplGridNodeName-#188%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.p.w.FileWriteAheadLogManager] Copied file [src=/gridgain/ssd/data/wal/10_124_133_10_47500/0000000000000008.wal, dst=/gridgain/sa
s/wal_archive/10_124_133_10_47500/0000000000000008.wal]
2019-04-23 15:22:57.442[INFO ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander] Cancelled rebalancing from all nodes [grp=SYSTEM_CACHEGROUP_DTR_union-module, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=6]]
2019-04-23 15:22:57.443[INFO ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=SYSTEM_CACHEGROUP_DTR_union-module], topVer=AffinityTopologyVersion [topVer=158, minorTopVer=6], rebalanceId=1003, routines=3]
2019-04-23 15:22:57.443[INFO ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander] Cancelled rebalancing from all nodes [grp=CACHEGROUP_SYSTEM_COMPONENT, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=6]]
2019-04-23 15:22:57.444[INFO ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=CACHEGROUP_SYSTEM_COMPONENT], topVer=AffinityTopologyVersion [

[1] https://issues.apache.org/jira/browse/IGNITE-7165

Mmuzaf v.pyatkov do u have ideas? thanks !

Issue Links

• is related to: IGNITE-7165 Re-balancing is cancelled if client node joins
• is related to: IGNITE-10374 Node doesn't own rebalanced partitions on rebalancing finished