Andrey Aleksandrov created IGNITE-8098:
------------------------------------------
Summary: Getting affinity for topology version earlier than affinity is calculated because of data race
Key: IGNITE-8098
URL:
https://issues.apache.org/jira/browse/IGNITE-8098 Project: Ignite
Issue Type: Bug
Affects Versions: 2.3
Reporter: Andrey Aleksandrov
From time to time the Ignite cluster with services throws next exception during restarting of some nodes:
java.lang.IllegalStateException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=c770dbcf-2908-442d-8aa0-bf26a2aecfef, addrs=[10.44.162.169, 127.0.0.1], sockAddrs=[clrv0000041279.ic.ing.net/10.44.162.169:56500, /127.0.0.1:56500], discPort=56500, order=11, intOrder=8, lastExchangeTime=1520931375337, loc=true, ver=2.3.3#20180213-sha1:f446df34, isClient=false], grp=ignite-sys-cache, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], head=AffinityTopologyVersion [topVer=15, minorTopVer=0], history=[AffinityTopologyVersion [topVer=11, minorTopVer=0], AffinityTopologyVersion [topVer=11, minorTopVer=1], AffinityTopologyVersion [topVer=12, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0]]]
Looks like the reason of this issue is the data race in GridServiceProcessor class.
How to reproduce:
1)To simulate data race you should update next place in source code:
Class: GridServiceProcessor
Method: @Override public void onEvent(final DiscoveryEvent evt, final DiscoCache discoCache) {
Place:
....
try {
svcName.set(dep.configuration().getName());
ctx.cache().internalCache(UTILITY_CACHE_NAME).context().affinity().
affinityReadyFuture(topVer).get();
//HERE (between GET and REASSIGN) you should add Thread.sleep(100) for example.
//try {
//Thread.sleep(100);
//}
//catch (InterruptedException e1) {
//e1.printStackTrace();
//}
reassign(dep, topVer);
}
catch (IgniteCheckedException ex) {
if (!(e instanceof ClusterTopologyCheckedException))
LT.error(log, ex, "Failed to do service reassignment (will retry): " +
dep.configuration().getName());
retries.add(dep);
}
...
2)After that you should imitate start/shutdown iterations. For reproducing I used GridServiceProcessorBatchDeploySelfTest (but timeout on future.get should be increased to avoid timeout error)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
