[jira] [Created] (IGNITE-2615) SQL query fails for cache with an eviction policy to offheap

[Anton Vinogradov (Jira)]

Sergey Kozlov created IGNITE-2615:
-------------------------------------

             Summary: SQL query fails for cache with an eviction policy to offheap
                 Key: IGNITE-2615
                 URL: https://issues.apache.org/jira/browse/IGNITE-2615
             Project: Ignite
          Issue Type: Bug
    Affects Versions: 1.5.0.final
            Reporter: Sergey Kozlov
            Priority: Critical


1 Start server node
2. Start client that creates an onheap cache and performs  random put/get/sqlquery
3. Client fails at query execution after starting eviciton in offheap:

Client log
{noformat}
>>> -----------------------------------------------------------------------
>>> Cache configuation used: 1
>>> CacheConfiguration [name=cache_d37d89a5_239_09203172203377125, storeConcurrentLoadAllThreshold=5, rebalancePoolSize=
2, rebalanceTimeout=10000, evictPlc=FifoEvictionPolicy [max=30000, batchSize=1, maxMemSize=0, memSize=0], evictSync=fals
e, evictKeyBufSize=1024, evictSyncConcurrencyLvl=4, evictSyncTimeout=10000, evictFilter=null, evictMaxOverflowRatio=10.0
, eagerTtl=true, dfltLockTimeout=0, startSize=1500000, nearCfg=null, writeSync=PRIMARY_SYNC, storeFactory=null, storeKee
pBinary=false, loadPrevVal=false, aff=null, cacheMode=REPLICATED, atomicityMode=ATOMIC, atomicWriteOrderMode=null, backu
ps=0, invalidate=false, tmLookupClsName=null, rebalanceMode=SYNC, rebalanceOrder=0, rebalanceBatchSize=524288, rebalance
BatchesPrefetchCount=2, offHeapMaxMem=10000000, swapEnabled=false, maxConcurrentAsyncOps=500, writeBehindEnabled=false,
writeBehindFlushSize=10240, writeBehindFlushFreq=5000, writeBehindFlushThreadCnt=1, writeBehindBatchSize=512, memMode=ON
HEAP_TIERED, affMapper=null, rebalanceDelay=0, rebalanceThrottle=0, interceptor=null, longQryWarnTimeout=3000, readFromB
ackup=true, nodeFilter=null, sqlSchema=null, sqlEscapeAll=false, sqlOnheapRowCacheSize=10240, snapshotableIdx=false, cpO
nRead=true, topValidator=null]
>>> -----------------------------------------------------------------------
[13:07:57,506][INFO][exchange-worker-#38%null%][GridCacheProcessor] Started cache [name=cache_d37d89a5_239_0920317220337
7125, mode=REPLICATED]
>>> Node for ordered messages: 86d4d306
>>> Node for unordered messages: 86d4d306
>>>Empty cache, loading data by streamer ...
Loaded 30000 key(s)
>>> 1309 operation(s), op/sec: 261, cache size: 33840
>>> Latest value is Data [modified=0, longCol=14658, doubleCol=0.3403571235222742, stringCol='14658*****', boolean=true,
 int=14658, bytesCol='[49, 52, 54, 53, 56, 42, 42, 42, 42, 42]', innerDataCol=inner [longCol=7329]', mapCol='{map_key_14
658=map_value_14658}']
>>> Operation counters: replace=81,scanquery=86,put=444,remove=77,get=379,putall=36,removeall=44,sqlquery=81,msg=0,contq
uery=0,getall=81
>>> 2596 operation(s), op/sec: 259, cache size: 36128
>>> Latest value is Data [modified=0, longCol=33912, doubleCol=0.9348676568832568, stringCol='33912*****', boolean=true,
 int=33912, bytesCol='[51, 51, 57, 49, 50, 42, 42, 42, 42, 42]', innerDataCol=inner [longCol=16956]', mapCol='{map_key_3
3912=map_value_33912}']
>>> Operation counters: replace=157,scanquery=166,put=856,remove=154,get=782,putall=69,removeall=90,sqlquery=170,msg=0,c
ontquery=0,getall=152
>>> 4022 operation(s), op/sec: 268, cache size: 40428
>>> Latest value is Data [modified=0, longCol=31128, doubleCol=0.33278377289395045, stringCol='31128*****', boolean=true
, int=31128, bytesCol='[51, 49, 49, 50, 56, 42, 42, 42, 42, 42]', innerDataCol=inner [longCol=15564]', mapCol='{map_key_
31128=map_value_31128}']
>>> Operation counters: replace=238,scanquery=241,put=1314,remove=244,get=1255,putall=109,removeall=133,sqlquery=250,msg
=0,contquery=0,getall=238
[13:08:25,463][INFO][grid-timeout-worker-#23%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=d37d89a5, name=null]
    ^-- H/N/C [hosts=1, nodes=2, CPUs=8]
    ^-- CPU [cur=0,07%, avg=3,76%, GC=0%]
    ^-- Heap [used=68MB, free=73,05%, comm=253MB]
    ^-- Public thread pool [active=0, idle=16, qSize=0]
    ^-- System thread pool [active=0, idle=16, qSize=0]
    ^-- Outbound messages queue [size=0]
[13:09:17,128][INFO][sys-#15%null%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.IgniteChec
kedException
[13:09:17,133][INFO][sys-#15%null%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.IgniteExce
ption
[13:09:17,145][INFO][main][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary
[13:09:17,167][INFO][main][GridCacheProcessor] Stopped cache: ignite-marshaller-sys-cache
[13:09:17,170][INFO][main][GridCacheProcessor] Stopped cache: ignite-sys-cache
[13:09:17,182][INFO][main][GridCacheProcessor] Stopped cache: ignite-atomics-sys-cache
[13:09:17,193][INFO][main][GridCacheProcessor] Stopped cache: cache_d37d89a5_239_09203172203377125
[13:09:17,207][INFO][main][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1455098845210, depMode
=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1c8825a5, clsLdrId=4fa46aac251-d37d89a5-d829-4d99-8da4-6c257d577d95, us
erVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionF
ullMap, pendingUndeploy=false, undeployed=true, usage=0]
[13:09:17,252][INFO][main][IgniteKernal]

>>> +---------------------------------------------------------------------------------+
>>> Ignite ver. 1.5.6#20160130-sha1:b146c0f5ac227e270d79c6bf18017676db6a1164 stopped OK
>>> +---------------------------------------------------------------------------------+
>>> Grid uptime: 00:01:51:799


Exception in thread "main" org.apache.ignite.cache.CachePartialUpdateException: Failed to update keys (retry update if p
ossible).: [54103, 33040, 68549, 3279, 68546, 97582, 13984, 347, 50618, 17292, 34415, 74063, 33355, 34410, 75988, 84280,
 85474, 30217, 34209, 8439, 75570, 96001, 6888, 14892, 78388, 75562, 7316, 82726, 84875, 40768, 91385, 97347, 70927, 496
2, 49508, 85821, 14246, 61801, 46378, 41204, 8907, 20978, 95389, 28009, 13963, 66695, 1407, 80195, 1203, 90020, 50836, 7
362, 8512, 34437, 32427, 88272, 21571, 53578, 58815, 2750, 13348, 13549, 36041, 48852, 48083, 73862, 35538, 73675, 1647,
 35735, 18409, 9173, 14425, 47061, 60527, 43444, 27745, 21338, 14439, 86006, 96695, 23801, 63640, 70736, 68300, 56163, 4
9598, 11187, 32932, 94587, 66639, 93437, 24681, 31771, 41497, 52068, 42616, 81643, 39418, 31764, 90498, 88890, 19740, 82
246, 5739, 1575, 44544, 60614, 89745, 89741, 47726, 76518, 6602, 55724, 47513, 74433, 71287, 30603, 35099, 23117, 82409,
 32194, 5774, 93766, 12572, 99699, 57076, 24534, 18053, 18269, 43563, 55519, 96495, 26315, 76338, 92923, 49898, 59708, 1
1402, 31459, 43745, 92379, 25040, 59715, 77229, 70251, 96058, 90845, 34372, 30812, 45462, 97505, 18291, 51625, 26766, 70
469, 71746, 12170, 50567, 17148, 48202, 4435, 23184, 17459, 16041, 36916, 71468, 5992, 49183, 18973, 8578, 22737, 61982,
 8788, 61591, 97955, 5786, 45168, 10106, 20623, 3903, 45804, 84652, 45805, 3908, 90186, 63235, 35248, 84665, 95180, 6936
7, 26508, 18960, 46417, 64386, 90174, 31250, 7494, 46067, 21430, 23519, 64489, 57203, 17386, 43349, 17387, 28122, 169, 5
7213, 97166, 10623, 6367, 43166, 78079, 217, 22408, 55117, 12975, 43365, 47330, 42911, 48503, 71562, 19230, 43377, 91277
, 79740, 41909, 15254, 15901, 8843, 41716, 82805, 78890, 33832, 27830, 48745, 58043, 14122, 67095, 2234, 82832, 93133, 7
0116, 94267, 55348, 72298, 9459, 44964, 20106, 22463, 39566, 39565, 3532, 23702, 54030, 54720, 70170, 35848, 63741, 8964
2, 13235, 21693, 80664, 89643, 3314, 36194, 60456, 93367, 41340, 41724, 86156, 35612, 46290, 9299, 46960, 46013, 37875,
89435, 37879, 21465, 93191, 12558, 52710, 53845, 54967, 81795, 29630, 46936, 82628, 3175, 43804, 42960, 83659, 58706, 29
635, 79177, 49274, 6710, 43807, 60423, 33630, 80011, 42382, 43630, 99147, 4515, 68963, 49764, 92891, 54791, 49760, 56743
, 37580, 76606, 30056, 73059, 45095, 23903, 51793, 21230, 28333, 45083, 65534, 11355, 72659, 43871, 59822, 4556, 68126,
44637, 56104, 23226, 37900, 84584, 10436, 99786, 98730, 34033, 49755, 73294, 22196, 41301, 17095, 83439, 18139, 67805, 2
3240, 1003, 4304, 81014, 44472, 70357, 289, 94710, 45335, 54315, 64218, 49097, 90095, 50433, 84711, 51713, 45353, 44690,
 84516, 88547, 37123, 56561, 26272, 8658, 91588, 88760, 5895, 88766, 51994, 20099, 15463, 78108, 50482, 43076, 96156, 71
30, 70322, 46530, 97409, 68193, 235, 98931, 95295, 5868, 67335, 63130, 86321, 28102, 84762, 1966, 95280, 11783, 76010]
        at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1604)
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1876)
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.removeAll(IgniteCacheProxy.java:1323)
        at org.apache.ignite.testtools.Iron.main(Iron.java:297)
Caused by: class org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException: Failed to update keys (
retry update if possible).: [54103, 33040, 68549, 3279, 68546, 97582, 13984, 347, 50618, 17292, 34415, 74063, 33355, 344
10, 75988, 84280, 85474, 30217, 34209, 8439, 75570, 96001, 6888, 14892, 78388, 75562, 7316, 82726, 84875, 40768, 91385,
97347, 70927, 4962, 49508, 85821, 14246, 61801, 46378, 41204, 8907, 20978, 95389, 28009, 13963, 66695, 1407, 80195, 1203
, 90020, 50836, 7362, 8512, 34437, 32427, 88272, 21571, 53578, 58815, 2750, 13348, 13549, 36041, 48852, 48083, 73862, 35
538, 73675, 1647, 35735, 18409, 9173, 14425, 47061, 60527, 43444, 27745, 21338, 14439, 86006, 96695, 23801, 63640, 70736
, 68300, 56163, 49598, 11187, 32932, 94587, 66639, 93437, 24681, 31771, 41497, 52068, 42616, 81643, 39418, 31764, 90498,
 88890, 19740, 82246, 5739, 1575, 44544, 60614, 89745, 89741, 47726, 76518, 6602, 55724, 47513, 74433, 71287, 30603, 350
99, 23117, 82409, 32194, 5774, 93766, 12572, 99699, 57076, 24534, 18053, 18269, 43563, 55519, 96495, 26315, 76338, 92923
, 49898, 59708, 11402, 31459, 43745, 92379, 25040, 59715, 77229, 70251, 96058, 90845, 34372, 30812, 45462, 97505, 18291,
 51625, 26766, 70469, 71746, 12170, 50567, 17148, 48202, 4435, 23184, 17459, 16041, 36916, 71468, 5992, 49183, 18973, 85
78, 22737, 61982, 8788, 61591, 97955, 5786, 45168, 10106, 20623, 3903, 45804, 84652, 45805, 3908, 90186, 63235, 35248, 8
4665, 95180, 69367, 26508, 18960, 46417, 64386, 90174, 31250, 7494, 46067, 21430, 23519, 64489, 57203, 17386, 43349, 173
87, 28122, 169, 57213, 97166, 10623, 6367, 43166, 78079, 217, 22408, 55117, 12975, 43365, 47330, 42911, 48503, 71562, 19
230, 43377, 91277, 79740, 41909, 15254, 15901, 8843, 41716, 82805, 78890, 33832, 27830, 48745, 58043, 14122, 67095, 2234
, 82832, 93133, 70116, 94267, 55348, 72298, 9459, 44964, 20106, 22463, 39566, 39565, 3532, 23702, 54030, 54720, 70170, 3
5848, 63741, 89642, 13235, 21693, 80664, 89643, 3314, 36194, 60456, 93367, 41340, 41724, 86156, 35612, 46290, 9299, 4696
0, 46013, 37875, 89435, 37879, 21465, 93191, 12558, 52710, 53845, 54967, 81795, 29630, 46936, 82628, 3175, 43804, 42960,
 83659, 58706, 29635, 79177, 49274, 6710, 43807, 60423, 33630, 80011, 42382, 43630, 99147, 4515, 68963, 49764, 92891, 54
791, 49760, 56743, 37580, 76606, 30056, 73059, 45095, 23903, 51793, 21230, 28333, 45083, 65534, 11355, 72659, 43871, 598
22, 4556, 68126, 44637, 56104, 23226, 37900, 84584, 10436, 99786, 98730, 34033, 49755, 73294, 22196, 41301, 17095, 83439
, 18139, 67805, 23240, 1003, 4304, 81014, 44472, 70357, 289, 94710, 45335, 54315, 64218, 49097, 90095, 50433, 84711, 517
13, 45353, 44690, 84516, 88547, 37123, 56561, 26272, 8658, 91588, 88760, 5895, 88766, 51994, 20099, 15463, 78108, 50482,
 43076, 96156, 7130, 70322, 46530, 97409, 68193, 235, 98931, 95295, 5868, 67335, 63130, 86321, 28102, 84762, 1966, 95280
, 11783, 76010]
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$UpdateState.add
FailedKeys(GridNearAtomicUpdateFuture.java:1241)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$UpdateState.onR
esult(GridNearAtomicUpdateFuture.java:661)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.onResult(GridNe
arAtomicUpdateFuture.java:344)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdate
Response(GridDhtAtomicCache.java:2709)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomi
cCache.java:128)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCa
che.java:263)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCa
che.java:261)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:582)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:280)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:204)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:80)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:163)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821
)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
        at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node.
                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFa
iledKeys(GridNearAtomicUpdateResponse.java:368)
                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncI
nternal0(GridDhtAtomicCache.java:1473)
                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncI
nternal(GridDhtAtomicCache.java:1282)
                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtom
icUpdateRequest(GridDhtAtomicCache.java:2692)
                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(Grid
DhtAtomicCache.java:128)
                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDht
AtomicCache.java:257)
                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDht
AtomicCache.java:255)
                ... 11 more
                Suppressed: class org.apache.ignite.IgniteException: Failed to get value for key: 352. This can happen d
ue to a long GC pause.
                        at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2A
bstractKeyValueRow.java:253)
                        at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2A
bstractKeyValueRow.java:284)
                        at org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.compareRows(GridH2IndexBas
e.java:119)
                        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.ja
va:248)
                        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.ja
va:49)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeap
SnapTreeMap.java:1350)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeap
SnapTreeMap.java:1346)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2102)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe
apSnapTreeMap.java:2217)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.updateUnderRoot(GridOff
HeapSnapTreeMap.java:2034)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.update(GridOffHeapSnapT
reeMap.java:1915)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapT
reeMap.java:1884)
                        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapT
reeMap.java:108)
                        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.remove(GridH2TreeIndex.jav
a:408)
                        at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:434)

                        at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:340)
                        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:
586)
                        at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java
:1014)
                        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQuery
Manager.java:431)
                        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clearIndex(GridCacheMapEntry.ja
va:3760)
                        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.j
ava:2334)
                        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateS
ingle(GridDhtAtomicCache.java:1997)
                        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateA
llAsyncInternal0(GridDhtAtomicCache.java:1407)
                        ... 16 more
{noformat}

Server log:
{noformat}
[13:09:16,846][SEVERE][sys-#17%null%][GridDhtAtomicCache] <cache_d37d89a5_239_09203172203377125> Unexpected exception du
ring cache update
class org.apache.ignite.IgniteException: Failed to get value for key: 352. This can happen due to a long GC pause.
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR
ow.java:253)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR
ow.java:284)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.compareRows(GridH2IndexBase.java:119)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:248)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:49)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java
:1350)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java
:1346)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2102)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.updateUnderRoot(GridOffHeapSnapTreeMap.
java:2034)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.update(GridOffHeapSnapTreeMap.java:1915
)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:1884
)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:108)

        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.remove(GridH2TreeIndex.java:408)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:434)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:340)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:586)
        at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:1014)
        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:431
)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clearIndex(GridCacheMapEntry.java:3760)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2334)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAto
micCache.java:1997)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0
(GridDhtAtomicCache.java:1407)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(
GridDhtAtomicCache.java:1282)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdate
Request(GridDhtAtomicCache.java:2692)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomi
cCache.java:128)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCa
che.java:257)
        at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCa
che.java:255)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:582)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:280)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:204)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:80)
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:163)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821
)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
        at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[13:09:17,158][INFO][disco-event-worker-#46%null%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=d37d89
a5-d829-4d99-8da4-6c257d577d95, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.1.112, 2001:0:9d38:6ab8:2c8e:661e:a1b7:c399],
 sockAddrs=[work-pc/192.168.1.112:0, /0:0:0:0:0:0:0:1:0, work-pc/192.168.1.112:0, /127.0.0.1:0, /192.168.1.112:0, /2001:
0:9d38:6ab8:2c8e:661e:a1b7:c399:0], discPort=0, order=2, intOrder=2, lastExchangeTime=1455098845119, loc=false, ver=1.5.
6#20160130-sha1:b146c0f5, isClient=true]
[13:09:17,182][INFO][disco-event-worker-#46%null%][GridDiscoveryManager] Topology snapshot [ver=3, servers=1, clients=0,
 CPUs=8, heap=2.0GB]
[13:09:17,193][INFO][disco-event-worker-#46%null%][GridDeploymentPerVersionStore] Class was undeployed in SHARED or CONT
INUOUS mode [cls=class org.apache.ignite.testtools.Iron$Data, alias=org.apache.ignite.testtools.Iron$Data]
[13:09:17,207][INFO][disco-event-worker-#46%null%][GridDeploymentPerVersionStore] Class was undeployed in SHARED or CONT
INUOUS mode [cls=class org.apache.ignite.testtools.Iron$3, alias=org.apache.ignite.testtools.Iron$3]
[13:09:17,223][INFO][disco-event-worker-#46%null%][GridDeploymentPerVersionStore] Class was undeployed in SHARED or CONT
INUOUS mode [cls=class org.apache.ignite.testtools.Iron$2, alias=org.apache.ignite.testtools.Iron$2]
[13:09:17,244][INFO][disco-event-worker-#46%null%][GridDeploymentPerVersionStore] Class was undeployed in SHARED or CONT
INUOUS mode [cls=class org.apache.ignite.testtools.Iron$StringBinaryObjectIgniteBiPredicate, alias=org.apache.ignite.tes
ttools.Iron$StringBinaryObjectIgniteBiPredicate]
[13:09:57,574][INFO][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=86d4d306, name=null]
    ^-- H/N/C [hosts=1, nodes=1, CPUs=8]
    ^-- CPU [cur=0%, avg=4,06%, GC=0%]
    ^-- Heap [used=350MB, free=82,48%, comm=2001MB]
    ^-- Public thread pool [active=0, idle=16, qSize=0]
    ^-- System thread pool [active=0, idle=16, qSize=0]
    ^-- Outbound messages queue [size=0]
[13:10:18,266][SEVERE][exchange-worker-#49%null%][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitio
ns (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], n
odeId=d37d89a5, evt=NODE_LEFT]
class org.apache.ignite.IgniteException: Failed to get value for key: 352. This can happen due to a long GC pause.
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR
ow.java:253)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR
ow.java:284)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.compareRows(GridH2IndexBase.java:119)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:248)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:49)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java
:1350)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java
:1346)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2102)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.updateUnderRoot(GridOffHeapSnapTreeMap.
java:2034)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.update(GridOffHeapSnapTreeMap.java:1915
)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:1884
)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:108)

        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.remove(GridH2TreeIndex.java:408)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:434)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:340)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:586)
        at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:1014)
        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:431
)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clearIndex(GridCacheMapEntry.java:3760)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clear(GridCacheMapEntry.java:2615)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter.clearLocally(GridCacheAdapter.java:1247)
        at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.onUndeploy0(GridCacheDeploymentManager
.java:265)
        at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.access$300(GridCacheDeploymentManager.
java:66)
        at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$3.apply(GridCacheDeploymentManager.jav
a:237)
        at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.unwind(GridCacheDeploymentManager.java
:208)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.unwindUndeploys(GridDh
tPreloader.java:748)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(Gr
idDhtPartitionsExchangeFuture.java:574)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePa
rtitionExchangeManager.java:1297)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
[13:10:18,465][SEVERE][exchange-worker-#49%null%][GridCachePartitionExchangeManager] Failed to wait for completion of pa
rtition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reas
sign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=d37d89a5-d829-4d99-8da4-6c257d577d95, addrs=[0:0:0:0:0
:0:0:1, 127.0.0.1, 192.168.1.112, 2001:0:9d38:6ab8:2c8e:661e:a1b7:c399], sockAddrs=[work-pc/192.168.1.112:0, /0:0:0:0:0:
0:0:1:0, work-pc/192.168.1.112:0, /127.0.0.1:0, /192.168.1.112:0, /2001:0:9d38:6ab8:2c8e:661e:a1b7:c399:0], discPort=0,
order=2, intOrder=2, lastExchangeTime=1455098845119, loc=false, ver=1.5.6#20160130-sha1:b146c0f5, isClient=true], topVer
=3, nodeId8=86d4d306, msg=Node left: TcpDiscoveryNode [id=d37d89a5-d829-4d99-8da4-6c257d577d95, addrs=[0:0:0:0:0:0:0:1,
127.0.0.1, 192.168.1.112, 2001:0:9d38:6ab8:2c8e:661e:a1b7:c399], sockAddrs=[work-pc/192.168.1.112:0, /0:0:0:0:0:0:0:1:0,
 work-pc/192.168.1.112:0, /127.0.0.1:0, /192.168.1.112:0, /2001:0:9d38:6ab8:2c8e:661e:a1b7:c399:0], discPort=0, order=2,
 intOrder=2, lastExchangeTime=1455098845119, loc=false, ver=1.5.6#20160130-sha1:b146c0f5, isClient=true], type=NODE_LEFT
, tstamp=1455098957188], rcvdIds=GridConcurrentHashSet [elements=[]], rmtIds=null, exchId=GridDhtPartitionExchangeId [to
pVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=d37d89a5, evt=NODE_LEFT], init=true, ready=false, replied
=false, added=true, initFut=GridFutureAdapter [resFlag=2, res=false, startTime=1455098957260, endTime=1455099018457, ign
oreInterrupts=false, lsnr=null, state=DONE], topSnapshot=null, lastVer=null, partReleaseFut=null, skipPreload=false, cli
entOnlyExchange=false, initTs=1455098957260, oldest=86d4d306-0be1-4cc6-9134-c39556225a56, oldestOrder=1, evtLatch=0, rem
aining=[], super=GridFutureAdapter [resFlag=1, res=class o.a.i.IgniteException: Failed to get value for key: 352. This c
an happen due to a long GC pause., startTime=1455098957260, endTime=1455099018457, ignoreInterrupts=false, lsnr=null, st
ate=DONE]]
class org.apache.ignite.IgniteCheckedException: Failed to get value for key: 352. This can happen due to a long GC pause
.
        at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7005)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:166)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePa
rtitionExchangeManager.java:1299)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: Failed to get value for key: 352. This can happen due to a long GC p
ause.
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR
ow.java:253)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR
ow.java:284)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.compareRows(GridH2IndexBase.java:119)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:248)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:49)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java
:1350)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java
:1346)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2102)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja
va:2217)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.updateUnderRoot(GridOffHeapSnapTreeMap.
java:2034)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.update(GridOffHeapSnapTreeMap.java:1915
)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:1884
)
        at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:108)

        at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.remove(GridH2TreeIndex.java:408)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:434)
        at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:340)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:586)
        at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:1014)
        at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:431
)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clearIndex(GridCacheMapEntry.java:3760)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clear(GridCacheMapEntry.java:2615)
        at org.apache.ignite.internal.processors.cache.GridCacheAdapter.clearLocally(GridCacheAdapter.java:1247)
        at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.onUndeploy0(GridCacheDeploymentManager
.java:265)
        at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.access$300(GridCacheDeploymentManager.
java:66)
        at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$3.apply(GridCacheDeploymentManager.jav
a:237)
        at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.unwind(GridCacheDeploymentManager.java
:208)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.unwindUndeploys(GridDh
tPreloader.java:748)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(Gr
idDhtPartitionsExchangeFuture.java:574)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePa
rtitionExchangeManager.java:1297)
        ... 2 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)