[jira] [Created] (IGNITE-1275) 'Get' from marshaller cache hangs on topology change

[Anton Vinogradov (Jira)]

Semen Boikov created IGNITE-1275:
------------------------------------

             Summary: 'Get' from marshaller cache hangs on topology change
                 Key: IGNITE-1275
                 URL: https://issues.apache.org/jira/browse/IGNITE-1275
             Project: Ignite
          Issue Type: Bug
          Components: cache
            Reporter: Semen Boikov
            Assignee: Alexey Goncharuk
            Priority: Critical
             Fix For: ignite-1.4


Observer this hang on TC in test IgniteCachePutRetryTransactionalSelfTest.testExplicitTransactionRetries:
{noformat}
[11:27:17] : [org.apache.ignite:ignite-core] Thread [name="ignite-#91721%sys-dht.IgniteCachePutRetryTransactionalSelfTest2%", id=115833, state=WAITING, blockCnt=0, waitCnt=3]
[11:27:17] : [org.apache.ignite:ignite-core]     Lock [object=o.a.i.i.processors.cache.distributed.dht.GridPartitionedGetFuture@4a555b67, ownerName=null, ownerId=-1]
[11:27:17] : [org.apache.ignite:ignite-core]         at sun.misc.Unsafe.park(Native Method)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4404)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1448)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.MarshallerContextImpl.className(MarshallerContextImpl.java:139)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.MarshallerContextAdapter.getClass(MarshallerContextAdapter.java:166)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:252)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:246)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.lang.IgniteBiTuple.readExternal(IgniteBiTuple.java:268)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:451)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:746)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:252)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.marshaller.optimized.OptimizedObjectInputStream.readLinkedList(OptimizedObjectInputStream.java:610)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:227)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:236)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.transactions.IgniteTxEntry.unmarshal(IgniteTxEntry.java:749)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.GridCacheMessage.unmarshalTx(GridCacheMessage.java:367)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.distributed.GridDistributedTxPrepareRequest.finishUnmarshal(GridDistributedTxPrepareRequest.java:300)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareRequest.finishUnmarshal(GridDhtTxPrepareRequest.java:293)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:973)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:229)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:158)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:48)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:127)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:764)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.managers.communication.GridIoManager.access$1500(GridIoManager.java:59)
[11:27:17] : [org.apache.ignite:ignite-core]         at o.a.i.i.managers.communication.GridIoManager$5.run(GridIoManager.java:727)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[11:27:17] : [org.apache.ignite:ignite-core]         at java.lang.Thread.run(Thread.java:722)
{noformat}

GridPartitionedGetFuture hangs when GridDistributedTxPrepareRequest is unmarshalled. Looks like for marshaller cache we need implement special 'get' mode which should not wait for topology futures (similar to 'tryPutIfAbsent' used for put in marshaller cache).

Found one more issue debugging this failure: test uses only server nodes and since marshaller cache uses FULL_SYNC mode, value in marshaller cache should be available locally and there should not be necessary to create GridPartitionedGetFuture. But it seems FULL_SYNC mode is broken for 'putIfAbsent' in atomic cache.
If two threads do 'putIfAbsent' concurrenly:
- thread1 on primary node executes 'innerUpdate', creates entry, sets value and going to send messages to backups
- thread2 on primary executes 'innerUpdate', filter fails since value already set by thread1, when filter fails backups are not updated, thread2 exits from 'putIfAbsent' but thread1 did not update backups yet

Also need to check case when GridDhtAtomicUpdateFuture is completed with error during node stop, looks like error is ignored in GridDhtAtomicUpdateFuture.onDone.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)