[jira] [Created] (IGNITE-10557) Control.sh validate index work long and broke down

[Anton Vinogradov (Jira)]

Alexand Polyakov created IGNITE-10557:
-----------------------------------------

             Summary: Control.sh validate index work long and broke down
                 Key: IGNITE-10557
                 URL: https://issues.apache.org/jira/browse/IGNITE-10557
             Project: Ignite
          Issue Type: Bug
          Components: visor
    Affects Versions: 2.6
            Reporter: Alexand Polyakov


cluster in the amount of 27Gb
performing validate_indexes took more than 1 hour
and execution failed

{code}
control.sh --cache validate_indexes
Control utility [ver. 2.6]
2018 Copyright(C) Apache Software Foundation
User: pprbusr
--------------------------------------------------------------------------------
Connection to cluster failed.
Error: Failed to perform request (connection failed): /10.117.102.207:11211
You have mail in /var/spool/mail/busr
{code}

analysis of the thread for 40 minutes showed that out of 32 nodes only on 3 nodes were flows ValidateIndexesClosure
at the same time, some threads are blocked

{code}
"pool-55-thread-53" #9255 prio=5 os_prio=0 tid=0x00007eb5a0073800 nid=0xb408 waiting for monitor entry [0x00007eb6554f3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.ignite.internal.pagemem.PageUtils.getBytes(PageUtils.java:63)
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:296)
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:159)
        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
        at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
        at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
        at org.apache.ignite.internal.processors.query.h2.database.io.H2InnerIO.getLookupRow(H2InnerIO.java:60)
        at org.apache.ignite.internal.processors.query.h2.database.io.H2InnerIO.getLookupRow(H2InnerIO.java:33)
        at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
        at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
        at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
        at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
        at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
        at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1157)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1124)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1091)
        at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.find(H2TreeIndex.java:201)
        at org.apache.ignite.internal.visor.verify.ValidateIndexesClosure.processPartition(ValidateIndexesClosure.java:524)
        at org.apache.ignite.internal.visor.verify.ValidateIndexesClosure.access$100(ValidateIndexesClosure.java:86)
        at org.apache.ignite.internal.visor.verify.ValidateIndexesClosure$2.call(ValidateIndexesClosure.java:394)
        at org.apache.ignite.internal.visor.verify.ValidateIndexesClosure$2.call(ValidateIndexesClosure.java:392)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)