[jira] [Created] (IGNITE-6506) Unable to activate cluster node if it was stopped during persistent storage checkpoint

[Anton Vinogradov (Jira)]

Joel Lang created IGNITE-6506:
---------------------------------

             Summary: Unable to activate cluster node if it was stopped during persistent storage checkpoint
                 Key: IGNITE-6506
                 URL: https://issues.apache.org/jira/browse/IGNITE-6506
             Project: Ignite
          Issue Type: Bug
          Components: persistence
    Affects Versions: 2.2
            Reporter: Joel Lang
            Priority: Critical


I have a cluster with two nodes: A and B.

On startup, node A and B wait for each other to be connected and then node A will attempt to activate the cluster.

While testing high availability we find that if a node is stopped during the persistent store checkpoint, we cannot activate the cluster on startup without deleting the persistent storage directory. Specifically in the case where node A is stopped during checkpointing, upon the next startup it will encounter several exceptions during activation and then hang without completing activation.

Here is the log.


{noformat}
2017-09-26 12:11:24 [tcp-disco-msg-worker-#2%mbe%] INFO  o.a.i.i.p.c.GridClusterStateProcessor - Start state transition: true
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.ignite.internal.exchange.time - Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], crd=true, evt=18, node=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], evtNode=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], customEvt=ChangeGlobalStateMessage [id=1d0cb2fbe51-7967bd11-40aa-40fe-b0a6-c43302cd4ee7, reqId=f7155dea-fede-4340-b244-7a3b65f167a8, initiatingNodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, activate=true]]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Start activation process [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.f.FilePageStoreManager - Resolved page store work directory: /opt/mbe1/ignite/db/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log work directory: /opt/mbe1/ignite/db/wal/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log archive directory: /opt/mbe1/ignite/db/wal/archive/mbe_MBE1
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN  o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - No user-defined default MemoryPolicy found; system default of 1GB size will be used.
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory [memoryAllocated=100.0 MiB, pages=48592, tableSize=2.9 MiB, checkpointBuffer=819.4 MiB]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory [memoryAllocated=3.1 GiB, pages=1544064, tableSize=91.0 MiB, checkpointBuffer=819.4 MiB]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin, end marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Checking memory state [lastValidPos=FileWALPointer [idx=139, fileOffset=31406805, len=20731, forceFlush=false], lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN  o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Ignite node stopped in the middle of checkpoint. Will restore memory state and finish checkpoint on node start.
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to activate node components [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
java.lang.ArrayIndexOutOfBoundsException: -1
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:82)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:92)
        at org.apache.ignite.internal.pagemem.wal.record.delta.DataPageInsertRecord.applyDelta(DataPageInsertRecord.java:57)
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1415)
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:539)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:770)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:574)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], waitTime=0ms]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin, end marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT]
org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553)
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484)
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938)
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO  o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Snapshot initialization completed [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], time=0ms]
2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.GridCachePartitionExchangeManager - Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], topVer=2, nodeId8=62cf0ccb, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1506445884063]], crd=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=1495762349], init=false, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], exchActions=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1506445884073, centralizedAff=false, changeGlobalStateE=java.lang.ArrayIndexOutOfBoundsException: -1, forcedRebFut=null, done=true, evtLatch=0, remaining=[bd81ae2b-91c0-45f1-b589-e73b7ffb56ee], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException: WAL history is too short [descs=[o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]], hash=927462353]]
org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]]
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290)
        at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553)
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484)
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938)
        at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:745)
{noformat}





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)