Sergey Chugunov created IGNITE-5115:
---------------------------------------
Summary: Investigation of failing tests of coordinator node failure
Key: IGNITE-5115
URL:
https://issues.apache.org/jira/browse/IGNITE-5115 Project: Ignite
Issue Type: Task
Components: messaging
Reporter: Sergey Chugunov
Fix For: 2.1
Tests *customEventCoordinatorFailure1/2* from *TcpDiscoverySelfTest* are flaky on TC and sometimes hang with the following assertion in logs:
{code}
Exception in thread "tcp-disco-msg-worker-#5245%tcp.TcpDiscoverySelfTest0%" java.lang.AssertionError
at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNodesRing.removeNode(TcpDiscoveryNodesRing.java:353)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeFailedMessage(ServerImpl.java:4670)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2567)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2366)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6485)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2456)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
{code}
It seems that this happens because tests' implementation drops connections of *TcpCommunicatonSpi* on coordinator node with *simulateNodeFailure* method.
At the same time tests leave *TcpDiscoverySpi* operational; it receives subsequent NodeFailed message and throws the assertion error shown above.
The whole situation looks legitimate as it is possible to imagine a situation when CommSPI connections on coordinator fail for some reason while DiscoSPI connections are healthy.
It is needed to investigate the situation deeper, figure out whether the root cause is using of *simulateNodeFailure* or not and propose a solution if the error may happen in the real life.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Locked
