Threads got stuck

[Alper Tekinalp]

Hi all.

We have 3 servers and cache configuration like:

    <bean class="org.apache.ignite.configuration.CacheConfiguration"
name="DEFAULT">
        <property name="atomicityMode" value="ATOMIC" />
        <property name="cacheMode" value="PARTITIONED" />
        <property name="memoryMode" value="OFFHEAP_TIERED" />
        <property name="backups" value="1" />
        <property name="affinity">
            <bean
class="org.apache.ignite.cache.affinity.fair.FairAffinityFunction">
                <constructor-arg index="0" type="int"
value="#{evamProperties['topology.cache.partition.size']}"/>
            </bean>
        </property>
        <!-- max memory is set to 0 means unlimited -->
        <property name="offHeapMaxMemory" value="0" />
        <property name="writeThrough" value="false" />
        <property name="writeBehindEnabled" value="false" />
        <property name="eagerTtl" value="false"/>
    </bean>

For our worker threads we check heartbeat and if a thread did not sent
heart beat for 10 minutes we consider it as stucked and interrrupt and
recreate it.

As I can see all our worker threads are stucked in cache.put() state and
interrupted and recreated regularly.

What can be the reason we are stucked at put? Following is stacktrace for
interruption error.

javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Failed to wait for
asynchronous operation permit (thread got interrupted).
        at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1502)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2021)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1221)
        at
com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(ConfigManagerHelperEE.java:90)
        at
com.intellica.project.helper.ee.StateMachineConfigManagerEEImpl.store(StateMachineConfigManagerEEImpl.java:53)
        at
com.evelopers.unimod.runtime.AbstractEventProcessor.storeConfig(AbstractEventProcessor.java:175)
        at
com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:130)
        at
com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:80)
        at
com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine.java:199)
        at
com.evelopers.unimod.runtime.StrictHandler.handle(StrictHandler.java:46)
        at
com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenarioLogic(AbstractScenarioWorker.java:172)
        at
com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenario(AbstractScenarioWorker.java:130)
        at
com.intellica.evam.engine.server.worker.AsyncWorker.processEvent(AsyncWorker.java:156)
        at
com.intellica.evam.engine.server.worker.AsyncWorker.run(AsyncWorker.java:88)
Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
wait for asynchronous operation permit (thread got interrupted).
        at
org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:747)
        at
org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:745)
        ... 14 more
Caused by: java.lang.InterruptedException
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOpAcquire(GridCacheAdapter.java:4597)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2215)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1214)
        ... 11 more


--
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  [phone] Fax: [phone]
www.evam.com.tr
<http://www.evam.com>

[Alper Tekinalp]

Hi.

There is also a few logs as :

 Failed to register marshalled class for more than 10 times in a row (may
affect performance).

Can it be releated?

On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[hidden email]> wrote:

> Hi all.
>
> We have 3 servers and cache configuration like:
>
>     <bean class="org.apache.ignite.configuration.CacheConfiguration"
> name="DEFAULT">
>         <property name="atomicityMode" value="ATOMIC" />
>         <property name="cacheMode" value="PARTITIONED" />
>         <property name="memoryMode" value="OFFHEAP_TIERED" />
>         <property name="backups" value="1" />
>         <property name="affinity">
>             <bean class="org.apache.ignite.cache.affinity.fair.
> FairAffinityFunction">
>                 <constructor-arg index="0" type="int"
> value="#{evamProperties['topology.cache.partition.size']}"/>
>             </bean>
>         </property>
>         <!-- max memory is set to 0 means unlimited -->
>         <property name="offHeapMaxMemory" value="0" />
>         <property name="writeThrough" value="false" />
>         <property name="writeBehindEnabled" value="false" />
>         <property name="eagerTtl" value="false"/>
>     </bean>
>
> For our worker threads we check heartbeat and if a thread did not sent
> heart beat for 10 minutes we consider it as stucked and interrrupt and
> recreate it.
>
> As I can see all our worker threads are stucked in cache.put() state and
> interrupted and recreated regularly.
>
> What can be the reason we are stucked at put? Following is stacktrace for
> interruption error.
>
> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
> Failed to wait for asynchronous operation permit (thread got interrupted).
>         at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> convertToCacheException(GridCacheUtils.java:1502)
>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.
> cacheException(IgniteCacheProxy.java:2021)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1221)
>         at com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(
> ConfigManagerHelperEE.java:90)
>         at com.intellica.project.helper.ee.StateMachineConfigManagerEEImp
> l.store(StateMachineConfigManagerEEImpl.java:53)
>         at com.evelopers.unimod.runtime.AbstractEventProcessor.
> storeConfig(AbstractEventProcessor.java:175)
>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
> AbstractEventProcessor.java:130)
>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
> AbstractEventProcessor.java:80)
>         at com.evelopers.unimod.runtime.ModelEngine.process(
> ModelEngine.java:199)
>         at com.evelopers.unimod.runtime.StrictHandler.handle(
> StrictHandler.java:46)
>         at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.
> runScenarioLogic(AbstractScenarioWorker.java:172)
>         at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.
> runScenario(AbstractScenarioWorker.java:130)
>         at com.intellica.evam.engine.server.worker.AsyncWorker.
> processEvent(AsyncWorker.java:156)
>         at com.intellica.evam.engine.server.worker.AsyncWorker.run(
> AsyncWorker.java:88)
> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
> wait for asynchronous operation permit (thread got interrupted).
>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUtils.java:747)
>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUtils.java:745)
>         ... 14 more
> Caused by: java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter.
> asyncOpAcquire(GridCacheAdapter.java:4597)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter.put(GridCacheAdapter.java:2215)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1214)
>         ... 11 more
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel:  [phone] Fax: [phone]
> www.evam.com.tr
> <http://www.evam.com>
>

--
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  [phone] Fax: [phone]
www.evam.com.tr
<http://www.evam.com>

[yzhdanov]

Alper,

There can be multiple reasons.

Can you please reproduce the issue one more time, collect and share the
following with us:

1. collect all the logs from all the nodes - clients and servers
2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>

--Yakov

2016-10-25 10:49 GMT+03:00 Alper Tekinalp <[hidden email]>:

> Hi.
>
> There is also a few logs as :
>
>  Failed to register marshalled class for more than 10 times in a row (may
> affect performance).
>
> Can it be releated?
>
> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[hidden email]> wrote:
>
>> Hi all.
>>
>> We have 3 servers and cache configuration like:
>>
>>     <bean class="org.apache.ignite.configuration.CacheConfiguration"
>> name="DEFAULT">
>>         <property name="atomicityMode" value="ATOMIC" />
>>         <property name="cacheMode" value="PARTITIONED" />
>>         <property name="memoryMode" value="OFFHEAP_TIERED" />
>>         <property name="backups" value="1" />
>>         <property name="affinity">
>>             <bean class="org.apache.ignite.cache
>> .affinity.fair.FairAffinityFunction">
>>                 <constructor-arg index="0" type="int"
>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>             </bean>
>>         </property>
>>         <!-- max memory is set to 0 means unlimited -->
>>         <property name="offHeapMaxMemory" value="0" />
>>         <property name="writeThrough" value="false" />
>>         <property name="writeBehindEnabled" value="false" />
>>         <property name="eagerTtl" value="false"/>
>>     </bean>
>>
>> For our worker threads we check heartbeat and if a thread did not sent
>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>> recreate it.
>>
>> As I can see all our worker threads are stucked in cache.put() state and
>> interrupted and recreated regularly.
>>
>> What can be the reason we are stucked at put? Following is stacktrace for
>> interruption error.
>>
>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>         at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>> onvertToCacheException(GridCacheUtils.java:1502)
>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .cacheException(IgniteCacheProxy.java:2021)
>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .put(IgniteCacheProxy.java:1221)
>>         at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>> te(ConfigManagerHelperEE.java:90)
>>         at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>> fig(AbstractEventProcessor.java:175)
>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>> AbstractEventProcessor.java:130)
>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>> AbstractEventProcessor.java:80)
>>         at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>> .java:199)
>>         at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>> ler.java:46)
>>         at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>         at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>> er.runScenario(AbstractScenarioWorker.java:130)
>>         at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>> vent(AsyncWorker.java:156)
>>         at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>> cWorker.java:88)
>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
>> wait for asynchronous operation permit (thread got interrupted).
>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>> ils.java:747)
>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>> ils.java:745)
>>         ... 14 more
>> Caused by: java.lang.InterruptedException
>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .putAsync(GridCacheAdapter.java:2541)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .put(GridCacheAdapter.java:2215)
>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .put(IgniteCacheProxy.java:1214)
>>         ... 11 more
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel:  [phone] Fax: [phone]
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel:  [phone] Fax: [phone]
> www.evam.com.tr
> <http://www.evam.com>
>

[Alper Tekinalp]

Hi Yakov.

I should also mention that we load cache data from one server and wait the
data to be replicated to others. Can that cause such a situation, too?

On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <[hidden email]> wrote:

> Alper,
>
> There can be multiple reasons.
>
> Can you please reproduce the issue one more time, collect and share the
> following with us:
>
> 1. collect all the logs from all the nodes - clients and servers
> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
>
> --Yakov
>
> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <[hidden email]>:
>
>> Hi.
>>
>> There is also a few logs as :
>>
>>  Failed to register marshalled class for more than 10 times in a row (may
>> affect performance).
>>
>> Can it be releated?
>>
>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[hidden email]> wrote:
>>
>>> Hi all.
>>>
>>> We have 3 servers and cache configuration like:
>>>
>>>     <bean class="org.apache.ignite.configuration.CacheConfiguration"
>>> name="DEFAULT">
>>>         <property name="atomicityMode" value="ATOMIC" />
>>>         <property name="cacheMode" value="PARTITIONED" />
>>>         <property name="memoryMode" value="OFFHEAP_TIERED" />
>>>         <property name="backups" value="1" />
>>>         <property name="affinity">
>>>             <bean class="org.apache.ignite.cache
>>> .affinity.fair.FairAffinityFunction">
>>>                 <constructor-arg index="0" type="int"
>>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>>             </bean>
>>>         </property>
>>>         <!-- max memory is set to 0 means unlimited -->
>>>         <property name="offHeapMaxMemory" value="0" />
>>>         <property name="writeThrough" value="false" />
>>>         <property name="writeBehindEnabled" value="false" />
>>>         <property name="eagerTtl" value="false"/>
>>>     </bean>
>>>
>>> For our worker threads we check heartbeat and if a thread did not sent
>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>>> recreate it.
>>>
>>> As I can see all our worker threads are stucked in cache.put() state and
>>> interrupted and recreated regularly.
>>>
>>> What can be the reason we are stucked at put? Following is stacktrace
>>> for interruption error.
>>>
>>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>>         at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>>> onvertToCacheException(GridCacheUtils.java:1502)
>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .cacheException(IgniteCacheProxy.java:2021)
>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .put(IgniteCacheProxy.java:1221)
>>>         at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>>> te(ConfigManagerHelperEE.java:90)
>>>         at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>>> fig(AbstractEventProcessor.java:175)
>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>> AbstractEventProcessor.java:130)
>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>> AbstractEventProcessor.java:80)
>>>         at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>>> .java:199)
>>>         at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>>> ler.java:46)
>>>         at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>>         at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>> er.runScenario(AbstractScenarioWorker.java:130)
>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>>> vent(AsyncWorker.java:156)
>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>>> cWorker.java:88)
>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
>>> to wait for asynchronous operation permit (thread got interrupted).
>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>> ils.java:747)
>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>> ils.java:745)
>>>         ... 14 more
>>> Caused by: java.lang.InterruptedException
>>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .putAsync(GridCacheAdapter.java:2541)
>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .put(GridCacheAdapter.java:2215)
>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .put(IgniteCacheProxy.java:1214)
>>>         ... 11 more
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel:  [phone] Fax: [phone]
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel:  [phone] Fax: [phone]
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>

--
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  [phone] Fax: [phone]
www.evam.com.tr
<http://www.evam.com>

[yzhdanov]

Alper, thanks for clarification, this will definitely help after we get the
info I requested. This is the only way to go with the investigation.

--Yakov

2016-10-25 11:20 GMT+03:00 Alper Tekinalp <[hidden email]>:

> Hi Yakov.
>
> I should also mention that we load cache data from one server and wait the
> data to be replicated to others. Can that cause such a situation, too?
>
> On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <[hidden email]>
> wrote:
>
>> Alper,
>>
>> There can be multiple reasons.
>>
>> Can you please reproduce the issue one more time, collect and share the
>> following with us:
>>
>> 1. collect all the logs from all the nodes - clients and servers
>> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
>>
>> --Yakov
>>
>> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <[hidden email]>:
>>
>>> Hi.
>>>
>>> There is also a few logs as :
>>>
>>>  Failed to register marshalled class for more than 10 times in a row
>>> (may affect performance).
>>>
>>> Can it be releated?
>>>
>>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[hidden email]> wrote:
>>>
>>>> Hi all.
>>>>
>>>> We have 3 servers and cache configuration like:
>>>>
>>>>     <bean class="org.apache.ignite.configuration.CacheConfiguration"
>>>> name="DEFAULT">
>>>>         <property name="atomicityMode" value="ATOMIC" />
>>>>         <property name="cacheMode" value="PARTITIONED" />
>>>>         <property name="memoryMode" value="OFFHEAP_TIERED" />
>>>>         <property name="backups" value="1" />
>>>>         <property name="affinity">
>>>>             <bean class="org.apache.ignite.cache
>>>> .affinity.fair.FairAffinityFunction">
>>>>                 <constructor-arg index="0" type="int"
>>>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>>>             </bean>
>>>>         </property>
>>>>         <!-- max memory is set to 0 means unlimited -->
>>>>         <property name="offHeapMaxMemory" value="0" />
>>>>         <property name="writeThrough" value="false" />
>>>>         <property name="writeBehindEnabled" value="false" />
>>>>         <property name="eagerTtl" value="false"/>
>>>>     </bean>
>>>>
>>>> For our worker threads we check heartbeat and if a thread did not sent
>>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>>>> recreate it.
>>>>
>>>> As I can see all our worker threads are stucked in cache.put() state
>>>> and interrupted and recreated regularly.
>>>>
>>>> What can be the reason we are stucked at put? Following is stacktrace
>>>> for interruption error.
>>>>
>>>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>>>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>>>         at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>>>> onvertToCacheException(GridCacheUtils.java:1502)
>>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .cacheException(IgniteCacheProxy.java:2021)
>>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .put(IgniteCacheProxy.java:1221)
>>>>         at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>>>> te(ConfigManagerHelperEE.java:90)
>>>>         at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>>>> fig(AbstractEventProcessor.java:175)
>>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>>> AbstractEventProcessor.java:130)
>>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>>> AbstractEventProcessor.java:80)
>>>>         at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>>>> .java:199)
>>>>         at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>>>> ler.java:46)
>>>>         at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>>>         at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>>> er.runScenario(AbstractScenarioWorker.java:130)
>>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>>>> vent(AsyncWorker.java:156)
>>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>>>> cWorker.java:88)
>>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
>>>> to wait for asynchronous operation permit (thread got interrupted).
>>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>>> ils.java:747)
>>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>>> ils.java:745)
>>>>         ... 14 more
>>>> Caused by: java.lang.InterruptedException
>>>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>>>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .putAsync(GridCacheAdapter.java:2541)
>>>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>>>         at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .put(GridCacheAdapter.java:2215)
>>>>         at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .put(IgniteCacheProxy.java:1214)
>>>>         ... 11 more
>>>>
>>>>
>>>> --
>>>> Alper Tekinalp
>>>>
>>>> Software Developer
>>>> Evam Streaming Analytics
>>>>
>>>> Atatürk Mah. Turgut Özal Bulv.
>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>> 34758 İSTANBUL
>>>>
>>>> Tel:  [phone] Fax: [phone]
>>>> www.evam.com.tr
>>>> <http://www.evam.com>
>>>>
>>>
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel:  [phone] Fax: [phone]
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel:  [phone] Fax: [phone]
> www.evam.com.tr
> <http://www.evam.com>
>

[Alper Tekinalp]

Hi.

After restarting servers, that issue did not occur again. I guess for some
reason the cluster did not start normally.

Thanks responding so quick.

Regards.

On Tue, Oct 25, 2016 at 11:31 AM, Yakov Zhdanov <[hidden email]> wrote:

> Alper, thanks for clarification, this will definitely help after we get the
> info I requested. This is the only way to go with the investigation.
>
> --Yakov
>
> 2016-10-25 11:20 GMT+03:00 Alper Tekinalp <[hidden email]>:
>
> > Hi Yakov.
> >
> > I should also mention that we load cache data from one server and wait
> the
> > data to be replicated to others. Can that cause such a situation, too?
> >
> > On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <[hidden email]>
> > wrote:
> >
> >> Alper,
> >>
> >> There can be multiple reasons.
> >>
> >> Can you please reproduce the issue one more time, collect and share the
> >> following with us:
> >>
> >> 1. collect all the logs from all the nodes - clients and servers
> >> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
> >>
> >> --Yakov
> >>
> >> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <[hidden email]>:
> >>
> >>> Hi.
> >>>
> >>> There is also a few logs as :
> >>>
> >>>  Failed to register marshalled class for more than 10 times in a row
> >>> (may affect performance).
> >>>
> >>> Can it be releated?
> >>>
> >>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[hidden email]>
> wrote:
> >>>
> >>>> Hi all.
> >>>>
> >>>> We have 3 servers and cache configuration like:
> >>>>
> >>>>     <bean class="org.apache.ignite.configuration.CacheConfiguration"
> >>>> name="DEFAULT">
> >>>>         <property name="atomicityMode" value="ATOMIC" />
> >>>>         <property name="cacheMode" value="PARTITIONED" />
> >>>>         <property name="memoryMode" value="OFFHEAP_TIERED" />
> >>>>         <property name="backups" value="1" />
> >>>>         <property name="affinity">
> >>>>             <bean class="org.apache.ignite.cache
> >>>> .affinity.fair.FairAffinityFunction">
> >>>>                 <constructor-arg index="0" type="int"
> >>>> value="#{evamProperties['topology.cache.partition.size']}"/>
> >>>>             </bean>
> >>>>         </property>
> >>>>         <!-- max memory is set to 0 means unlimited -->
> >>>>         <property name="offHeapMaxMemory" value="0" />
> >>>>         <property name="writeThrough" value="false" />
> >>>>         <property name="writeBehindEnabled" value="false" />
> >>>>         <property name="eagerTtl" value="false"/>
> >>>>     </bean>
> >>>>
> >>>> For our worker threads we check heartbeat and if a thread did not sent
> >>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
> >>>> recreate it.
> >>>>
> >>>> As I can see all our worker threads are stucked in cache.put() state
> >>>> and interrupted and recreated regularly.
> >>>>
> >>>> What can be the reason we are stucked at put? Following is stacktrace
> >>>> for interruption error.
> >>>>
> >>>> javax.cache.CacheException: class org.apache.ignite.
> IgniteInterruptedException:
> >>>> Failed to wait for asynchronous operation permit (thread got
> interrupted).
> >>>>         at org.apache.ignite.internal.processors.cache.
> GridCacheUtils.c
> >>>> onvertToCacheException(GridCacheUtils.java:1502)
> >>>>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .cacheException(IgniteCacheProxy.java:2021)
> >>>>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .put(IgniteCacheProxy.java:1221)
> >>>>         at com.intellica.project.helper.ee.ConfigManagerHelperEE.
> setSta
> >>>> te(ConfigManagerHelperEE.java:90)
> >>>>         at com.intellica.project.helper.ee.
> StateMachineConfigManagerEEI
> >>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
> >>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.
> storeCon
> >>>> fig(AbstractEventProcessor.java:175)
> >>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.
> process(
> >>>> AbstractEventProcessor.java:130)
> >>>>         at com.evelopers.unimod.runtime.AbstractEventProcessor.
> process(
> >>>> AbstractEventProcessor.java:80)
> >>>>         at com.evelopers.unimod.runtime.ModelEngine.process(
> ModelEngine
> >>>> .java:199)
> >>>>         at com.evelopers.unimod.runtime.StrictHandler.handle(
> StrictHand
> >>>> ler.java:46)
> >>>>         at com.intellica.evam.engine.server.worker.
> AbstractScenarioWork
> >>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
> >>>>         at com.intellica.evam.engine.server.worker.
> AbstractScenarioWork
> >>>> er.runScenario(AbstractScenarioWorker.java:130)
> >>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.
> processE
> >>>> vent(AsyncWorker.java:156)
> >>>>         at com.intellica.evam.engine.server.worker.AsyncWorker.run(
> Asyn
> >>>> cWorker.java:88)
> >>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
> >>>> to wait for asynchronous operation permit (thread got interrupted).
> >>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUt
> >>>> ils.java:747)
> >>>>         at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUt
> >>>> ils.java:745)
> >>>>         ... 14 more
> >>>> Caused by: java.lang.InterruptedException
> >>>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquir
> >>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
> >>>>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> >>>>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
> >>>>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
> >>>>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
> >>>>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
> >>>>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .putAsync(GridCacheAdapter.java:2541)
> >>>>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
> >>>>         at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .put(GridCacheAdapter.java:2215)
> >>>>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .put(IgniteCacheProxy.java:1214)
> >>>>         ... 11 more
> >>>>
> >>>>
> >>>> --
> >>>> Alper Tekinalp
> >>>>
> >>>> Software Developer
> >>>> Evam Streaming Analytics
> >>>>
> >>>> Atatürk Mah. Turgut Özal Bulv.
> >>>> Gardenya 5 Plaza K:6 Ataşehir
> >>>> 34758 İSTANBUL
> >>>>
> >>>> Tel:  [phone] Fax: [phone]
> >>>> www.evam.com.tr
> >>>> <http://www.evam.com>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Alper Tekinalp
> >>>
> >>> Software Developer
> >>> Evam Streaming Analytics
> >>>
> >>> Atatürk Mah. Turgut Özal Bulv.
> >>> Gardenya 5 Plaza K:6 Ataşehir
> >>> 34758 İSTANBUL
> >>>
> >>> Tel:  [phone] Fax: [phone]
> >>> www.evam.com.tr
> >>> <http://www.evam.com>
> >>>
> >>
> >>
> >
> >
> > --
> > Alper Tekinalp
> >
> > Software Developer
> > Evam Streaming Analytics
> >
> > Atatürk Mah. Turgut Özal Bulv.
> > Gardenya 5 Plaza K:6 Ataşehir
> > 34758 İSTANBUL
> >
> > Tel:  [phone] Fax: [phone]
> > www.evam.com.tr
> > <http://www.evam.com>
> >
>

--
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  [phone] Fax: [phone]
www.evam.com.tr
<http://www.evam.com>