Alexandr Fedotov created IGNITE-6256:
----------------------------------------
Summary: When a node becomes segmented an AssertionError is thrown during GridDhtPartitionTopologyImpl.removeNode
Key: IGNITE-6256
URL:
https://issues.apache.org/jira/browse/IGNITE-6256 Project: Ignite
Issue Type: Bug
Components: general
Affects Versions: 1.8
Reporter: Alexandr Fedotov
Fix For: 2.3
Below is the sequence of steps that leads to the assertion:
1) A node becomes SEGMENTED when it's determined by SegmentCheckWorker, after an EVT_NODE_FAILED has been received.
2) It gets visibleRemoteNodes from it's TcpDiscoveryNodesRing
3) Clears the TcpDiscoveryNodesRing leaving only self on the list. The node ring is used to determine if a node is alive
during DiscoCache creation
4) After that, the node initiates removal of all the nodes read in step 2
5) For each node, it sends an EVT_NODE_FAILED to the corresponding DiscoverySpiListener
providing a topology containing all the nodes except already processed
6) This event gets into GridDiscoveryManager
7) The node gets removed from alive nodes for every DiscoCache in discoCacheHist
8) Topology change is detected
9) Creation of a new DiscoCache is attempted. At this moment every remote node is not available due to the
TcpDiscoveryNodesRing has been cleared, thus resulting in a DiscoCache with empty alives
10) The event with the created DiscoCache and the new topology version is passed to DiscoveryWorker
11) The event is eventually handled by DiscoveryWorker and is recorded by DiscoveryWorker#recordEvent
12) The recording is handled by GridEventStorageManager which notifies every listener for this event type (EVT_NODE_FAILED)
13) One of the listeners is GridCachePartitionExchangeManager#discoLsnr
It creates a new GridDhtPartitionsExchangeFuture with the empty DiscoCache received with the event and enqueues it
14) The future gets eventually handled by GridDhtPartitionsExchangeFuture and initialized
15) updateTopologies is called, which for each GridCacheContext gets its topology (GridDhtPartitionTopology)
and calls GridDhtPartitionTopology#updateTopologyVersion
16) DiscoCache for GridDhtPartitionTopology is assigned from the one of the GridDhtPartitionsExchangeFuture.
The assigned DiscoCache has empty alives at the moment
15) A distributed exchange is handled (GridDhtPartitionsExchangeFuture#distributedExchange)
16) For each cache context GridCacheContext, for its topology (GridDhtPartitionTopologyImpl) GridDhtPartitionTopologyImpl#beforeExchange is called
17) The fact that the node has left is determined and GridDhtPartitionTopologyImpl#removeNode is called to handle it
18) An attempt is made to get the alive coordinator node by calling DiscoCache#oldestAliveServerNode
19) null is returned which results in an AssertionError
The fix should probably prevent initiating exchange futures if a node has segmented.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Locked
