Client node throws exception when server node killed

[Artem Shutak]

Igniters,

In process of working on [1] I've found that Ignite client node throws
CacheException which caused by IgniteClientDisconnectedException (stack
trace below) sometimes when I kill one server node at cluster.

It looks like not expected behavior for me. Is it know issue?

My configuration:
- 3 server nodes
- 2 client nodes
- discovery configuration:
        <property name="discoverySpi">
            <bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                            <list>
                                <value>127.0.0.1:47500</value>
                                <value>127.0.0.1:47501</value>
                                <value>127.0.0.1:47502</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>

[1] https://issues.apache.org/jira/browse/IGNITE-1397

-- Artem --

javax.cache.CacheException: class
org.apache.ignite.IgniteClientDisconnectedException: Operation has been
cancelled (client node disconnected).
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1621)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1749)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:815)
at
org.apache.ignite.yardstick.cache.failover.IgniteAtomicRetriesFailoverBenchmark.test(IgniteAtomicRetriesFailoverBenchmark.java:39)
at
org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:165)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteClientDisconnectedException:
Operation has been cancelled (client node disconnected).
at
org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:801)
at
org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:799)
... 6 more
Caused by: class
org.apache.ignite.internal.IgniteClientDisconnectedCheckedException:
Operation has been cancelled (client node disconnected).
at
org.apache.ignite.internal.processors.cache.GridCacheMvccManager.disconnectedError(GridCacheMvccManager.java:360)
at
org.apache.ignite.internal.processors.cache.GridCacheMvccManager.onDisconnected(GridCacheMvccManager.java:334)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedContext.onDisconnected(GridCacheSharedContext.java:141)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:997)
at
org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:2983)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onDiscovery(GridDiscoveryManager.java:592)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2024)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2005)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1422)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

[Semyon Boikov]

Artem,

This looks like a bug, client should not disconnect from cluster if there
are alive server nodes, I'll try to reproduce this error.

On Mon, Sep 21, 2015 at 8:31 PM, Artem Shutak <[hidden email]> wrote:

> Igniters,
>
> In process of working on [1] I've found that Ignite client node throws
> CacheException which caused by IgniteClientDisconnectedException (stack
> trace below) sometimes when I kill one server node at cluster.
>
> It looks like not expected behavior for me. Is it know issue?
>
> My configuration:
> - 3 server nodes
> - 2 client nodes
> - discovery configuration:
>         <property name="discoverySpi">
>             <bean
> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>                 <property name="ipFinder">
>                     <bean
>
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>                         <property name="addresses">
>                             <list>
>                                 <value>127.0.0.1:47500</value>
>                                 <value>127.0.0.1:47501</value>
>                                 <value>127.0.0.1:47502</value>
>                             </list>
>                         </property>
>                     </bean>
>                 </property>
>             </bean>
>         </property>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-1397
>
> -- Artem --
>
> javax.cache.CacheException: class
> org.apache.ignite.IgniteClientDisconnectedException: Operation has been
> cancelled (client node disconnected).
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1621)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1749)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:815)
> at
>
> org.apache.ignite.yardstick.cache.failover.IgniteAtomicRetriesFailoverBenchmark.test(IgniteAtomicRetriesFailoverBenchmark.java:39)
> at
> org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:165)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.IgniteClientDisconnectedException:
> Operation has been cancelled (client node disconnected).
> at
> org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:801)
> at
> org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:799)
> ... 6 more
> Caused by: class
> org.apache.ignite.internal.IgniteClientDisconnectedCheckedException:
> Operation has been cancelled (client node disconnected).
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMvccManager.disconnectedError(GridCacheMvccManager.java:360)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMvccManager.onDisconnected(GridCacheMvccManager.java:334)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheSharedContext.onDisconnected(GridCacheSharedContext.java:141)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:997)
> at
>
> org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:2983)
> at
>
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onDiscovery(GridDiscoveryManager.java:592)
> at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2024)
> at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2005)
> at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1422)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>

[Artem Shutak]

Filed an issue: https://issues.apache.org/jira/browse/IGNITE-1524. Attached
logs.

Let me know if I can help.

-- Artem --

On Tue, Sep 22, 2015 at 10:23 AM, Semyon Boikov <[hidden email]>
wrote:

> Artem,
>
> This looks like a bug, client should not disconnect from cluster if there
> are alive server nodes, I'll try to reproduce this error.
>
> On Mon, Sep 21, 2015 at 8:31 PM, Artem Shutak <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > In process of working on [1] I've found that Ignite client node throws
> > CacheException which caused by IgniteClientDisconnectedException (stack
> > trace below) sometimes when I kill one server node at cluster.
> >
> > It looks like not expected behavior for me. Is it know issue?
> >
> > My configuration:
> > - 3 server nodes
> > - 2 client nodes
> > - discovery configuration:
> >         <property name="discoverySpi">
> >             <bean
> > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> >                 <property name="ipFinder">
> >                     <bean
> >
> >
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> >                         <property name="addresses">
> >                             <list>
> >                                 <value>127.0.0.1:47500</value>
> >                                 <value>127.0.0.1:47501</value>
> >                                 <value>127.0.0.1:47502</value>
> >                             </list>
> >                         </property>
> >                     </bean>
> >                 </property>
> >             </bean>
> >         </property>
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-1397
> >
> > -- Artem --
> >
> > javax.cache.CacheException: class
> > org.apache.ignite.IgniteClientDisconnectedException: Operation has been
> > cancelled (client node disconnected).
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1621)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1749)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:815)
> > at
> >
> >
> org.apache.ignite.yardstick.cache.failover.IgniteAtomicRetriesFailoverBenchmark.test(IgniteAtomicRetriesFailoverBenchmark.java:39)
> > at
> >
> org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:165)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: class org.apache.ignite.IgniteClientDisconnectedException:
> > Operation has been cancelled (client node disconnected).
> > at
> >
> org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:801)
> > at
> >
> org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:799)
> > ... 6 more
> > Caused by: class
> > org.apache.ignite.internal.IgniteClientDisconnectedCheckedException:
> > Operation has been cancelled (client node disconnected).
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheMvccManager.disconnectedError(GridCacheMvccManager.java:360)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheMvccManager.onDisconnected(GridCacheMvccManager.java:334)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheSharedContext.onDisconnected(GridCacheSharedContext.java:141)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:997)
> > at
> >
> >
> org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:2983)
> > at
> >
> >
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onDiscovery(GridDiscoveryManager.java:592)
> > at
> >
> >
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2024)
> > at
> >
> >
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2005)
> > at
> >
> >
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1422)
> > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> >
>