Hello All!
Just found that if I have many entries in cache and do the scan query with page size 100 then my query seems to iterate over the whole cache and only after that I could fetch first page of data. I think that it should be reworked to return first page as fast as possible - without of waiting while scan query will be completed. Thoughts? -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
I have filed a ticket for this issue:
https://issues.apache.org/jira/browse/IGNITE-872 Sergey, since it has to do with queries, can you please take a look? Thanks, D. On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov <[hidden email]> wrote: > Hello All! > > Just found that if I have many entries in cache and do the scan query with > page size 100 > then my query seems to iterate over the whole cache and only after that I > could fetch first page of data. > > I think that it should be reworked to return first page as fast as possible > - without of waiting while scan query will be completed. > > Thoughts? > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com > |
Alexey,
How did you come to this conclusion? I don't see anything like that in our code, we just take keySet or primaryKeySet iterator and fetch it page by page. Sergi 2015-05-07 9:58 GMT+03:00 Dmitriy Setrakyan <[hidden email]>: > I have filed a ticket for this issue: > https://issues.apache.org/jira/browse/IGNITE-872 > > Sergey, since it has to do with queries, can you please take a look? > > Thanks, > D. > > On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov <[hidden email]> > wrote: > > > Hello All! > > > > Just found that if I have many entries in cache and do the scan query > with > > page size 100 > > then my query seems to iterate over the whole cache and only after that I > > could fetch first page of data. > > > > I think that it should be reworked to return first page as fast as > possible > > - without of waiting while scan query will be completed. > > > > Thoughts? > > > > -- > > Alexey Kuznetsov > > GridGain Systems > > www.gridgain.com > > > |
Sergi.
I put 1M entries in cache. Execute scan query from Visor and see 1M reads in cache metrics (and I do not even hit "Next page" button). Also more entries in cache results in more longer execution time of scan query to wait for first page to show in Visor. Please take a look at VisorQueryJob code (line 75...). By default Visor use 100 rows as page size. I expect that take first 100 rows for scan query should be very fast. On Thu, May 7, 2015 at 10:19 PM, Sergi Vladykin <[hidden email]> wrote: > Alexey, > > How did you come to this conclusion? I don't see anything like that in our > code, we just take keySet or primaryKeySet iterator and fetch it page by > page. > > Sergi > > 2015-05-07 9:58 GMT+03:00 Dmitriy Setrakyan <[hidden email]>: > > > I have filed a ticket for this issue: > > https://issues.apache.org/jira/browse/IGNITE-872 > > > > Sergey, since it has to do with queries, can you please take a look? > > > > Thanks, > > D. > > > > On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov < > [hidden email]> > > wrote: > > > > > Hello All! > > > > > > Just found that if I have many entries in cache and do the scan query > > with > > > page size 100 > > > then my query seems to iterate over the whole cache and only after > that I > > > could fetch first page of data. > > > > > > I think that it should be reworked to return first page as fast as > > possible > > > - without of waiting while scan query will be completed. > > > > > > Thoughts? > > > > > > -- > > > Alexey Kuznetsov > > > GridGain Systems > > > www.gridgain.com > > > > > > -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
Alexey,
Do you have only one server node? In this case query is considered to be local and pagination is disabled. Really looks like redundant iteration. I also noticed that IgniteCacheProxy.query(Query filter, @Nullable ClusterGroup grp) method on line 347 ignores page size parameter (added a comment in ticket). -- Val On Thu, May 7, 2015 at 8:32 AM, Alexey Kuznetsov <[hidden email]> wrote: > Sergi. > > I put 1M entries in cache. Execute scan query from Visor and see 1M reads > in cache metrics (and I do not even hit "Next page" button). > Also more entries in cache results in more longer execution time of scan > query to wait for first page to show in Visor. > > Please take a look at VisorQueryJob code (line 75...). > > By default Visor use 100 rows as page size. > > I expect that take first 100 rows for scan query should be very fast. > > On Thu, May 7, 2015 at 10:19 PM, Sergi Vladykin <[hidden email]> > wrote: > > > Alexey, > > > > How did you come to this conclusion? I don't see anything like that in > our > > code, we just take keySet or primaryKeySet iterator and fetch it page by > > page. > > > > Sergi > > > > 2015-05-07 9:58 GMT+03:00 Dmitriy Setrakyan <[hidden email]>: > > > > > I have filed a ticket for this issue: > > > https://issues.apache.org/jira/browse/IGNITE-872 > > > > > > Sergey, since it has to do with queries, can you please take a look? > > > > > > Thanks, > > > D. > > > > > > On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov < > > [hidden email]> > > > wrote: > > > > > > > Hello All! > > > > > > > > Just found that if I have many entries in cache and do the scan query > > > with > > > > page size 100 > > > > then my query seems to iterate over the whole cache and only after > > that I > > > > could fetch first page of data. > > > > > > > > I think that it should be reworked to return first page as fast as > > > possible > > > > - without of waiting while scan query will be completed. > > > > > > > > Thoughts? > > > > > > > > -- > > > > Alexey Kuznetsov > > > > GridGain Systems > > > > www.gridgain.com > > > > > > > > > > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com > |
Val, yes I remember that I tested with one node.
But I will recheck with several nodes today and give more info. On Fri, May 8, 2015 at 2:38 AM, Valentin Kulichenko < [hidden email]> wrote: > Alexey, > > Do you have only one server node? In this case query is considered to be > local and pagination is disabled. Really looks like redundant iteration. > > I also noticed that IgniteCacheProxy.query(Query filter, @Nullable > ClusterGroup grp) method on line 347 ignores page size parameter (added a > comment in ticket). > > -- > Val > > On Thu, May 7, 2015 at 8:32 AM, Alexey Kuznetsov <[hidden email]> > wrote: > > > Sergi. > > > > I put 1M entries in cache. Execute scan query from Visor and see 1M reads > > in cache metrics (and I do not even hit "Next page" button). > > Also more entries in cache results in more longer execution time of scan > > query to wait for first page to show in Visor. > > > > Please take a look at VisorQueryJob code (line 75...). > > > > By default Visor use 100 rows as page size. > > > > I expect that take first 100 rows for scan query should be very fast. > > > > On Thu, May 7, 2015 at 10:19 PM, Sergi Vladykin < > [hidden email]> > > wrote: > > > > > Alexey, > > > > > > How did you come to this conclusion? I don't see anything like that in > > our > > > code, we just take keySet or primaryKeySet iterator and fetch it page > by > > > page. > > > > > > Sergi > > > > > > 2015-05-07 9:58 GMT+03:00 Dmitriy Setrakyan <[hidden email]>: > > > > > > > I have filed a ticket for this issue: > > > > https://issues.apache.org/jira/browse/IGNITE-872 > > > > > > > > Sergey, since it has to do with queries, can you please take a look? > > > > > > > > Thanks, > > > > D. > > > > > > > > On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov < > > > [hidden email]> > > > > wrote: > > > > > > > > > Hello All! > > > > > > > > > > Just found that if I have many entries in cache and do the scan > query > > > > with > > > > > page size 100 > > > > > then my query seems to iterate over the whole cache and only after > > > that I > > > > > could fetch first page of data. > > > > > > > > > > I think that it should be reworked to return first page as fast as > > > > possible > > > > > - without of waiting while scan query will be completed. > > > > > > > > > > Thoughts? > > > > > > > > > > -- > > > > > Alexey Kuznetsov > > > > > GridGain Systems > > > > > www.gridgain.com > > > > > > > > > > > > > > > > > > > > -- > > Alexey Kuznetsov > > GridGain Systems > > www.gridgain.com > > > -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
Checked with 2 nodes - scan query works fast.
So, problem in case of single node. On Fri, May 8, 2015 at 8:17 AM, Alexey Kuznetsov <[hidden email]> wrote: > Val, yes I remember that I tested with one node. > But I will recheck with several nodes today and give more info. > > On Fri, May 8, 2015 at 2:38 AM, Valentin Kulichenko < > [hidden email]> wrote: > >> Alexey, >> >> Do you have only one server node? In this case query is considered to be >> local and pagination is disabled. Really looks like redundant iteration. >> >> I also noticed that IgniteCacheProxy.query(Query filter, @Nullable >> ClusterGroup grp) method on line 347 ignores page size parameter (added a >> comment in ticket). >> >> -- >> Val >> >> On Thu, May 7, 2015 at 8:32 AM, Alexey Kuznetsov <[hidden email] >> > >> wrote: >> >> > Sergi. >> > >> > I put 1M entries in cache. Execute scan query from Visor and see 1M >> reads >> > in cache metrics (and I do not even hit "Next page" button). >> > Also more entries in cache results in more longer execution time of scan >> > query to wait for first page to show in Visor. >> > >> > Please take a look at VisorQueryJob code (line 75...). >> > >> > By default Visor use 100 rows as page size. >> > >> > I expect that take first 100 rows for scan query should be very fast. >> > >> > On Thu, May 7, 2015 at 10:19 PM, Sergi Vladykin < >> [hidden email]> >> > wrote: >> > >> > > Alexey, >> > > >> > > How did you come to this conclusion? I don't see anything like that in >> > our >> > > code, we just take keySet or primaryKeySet iterator and fetch it page >> by >> > > page. >> > > >> > > Sergi >> > > >> > > 2015-05-07 9:58 GMT+03:00 Dmitriy Setrakyan <[hidden email]>: >> > > >> > > > I have filed a ticket for this issue: >> > > > https://issues.apache.org/jira/browse/IGNITE-872 >> > > > >> > > > Sergey, since it has to do with queries, can you please take a look? >> > > > >> > > > Thanks, >> > > > D. >> > > > >> > > > On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov < >> > > [hidden email]> >> > > > wrote: >> > > > >> > > > > Hello All! >> > > > > >> > > > > Just found that if I have many entries in cache and do the scan >> query >> > > > with >> > > > > page size 100 >> > > > > then my query seems to iterate over the whole cache and only after >> > > that I >> > > > > could fetch first page of data. >> > > > > >> > > > > I think that it should be reworked to return first page as fast as >> > > > possible >> > > > > - without of waiting while scan query will be completed. >> > > > > >> > > > > Thoughts? >> > > > > >> > > > > -- >> > > > > Alexey Kuznetsov >> > > > > GridGain Systems >> > > > > www.gridgain.com >> > > > > >> > > > >> > > >> > >> > >> > >> > -- >> > Alexey Kuznetsov >> > GridGain Systems >> > www.gridgain.com >> > >> > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com > -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
Alex, can you please create a junit and ticket?
--Yakov 2015-05-08 9:43 GMT+03:00 Alexey Kuznetsov <[hidden email]>: > Checked with 2 nodes - scan query works fast. > So, problem in case of single node. > > On Fri, May 8, 2015 at 8:17 AM, Alexey Kuznetsov <[hidden email]> > wrote: > > > Val, yes I remember that I tested with one node. > > But I will recheck with several nodes today and give more info. > > > > On Fri, May 8, 2015 at 2:38 AM, Valentin Kulichenko < > > [hidden email]> wrote: > > > >> Alexey, > >> > >> Do you have only one server node? In this case query is considered to be > >> local and pagination is disabled. Really looks like redundant iteration. > >> > >> I also noticed that IgniteCacheProxy.query(Query filter, @Nullable > >> ClusterGroup grp) method on line 347 ignores page size parameter (added > a > >> comment in ticket). > >> > >> -- > >> Val > >> > >> On Thu, May 7, 2015 at 8:32 AM, Alexey Kuznetsov < > [hidden email] > >> > > >> wrote: > >> > >> > Sergi. > >> > > >> > I put 1M entries in cache. Execute scan query from Visor and see 1M > >> reads > >> > in cache metrics (and I do not even hit "Next page" button). > >> > Also more entries in cache results in more longer execution time of > scan > >> > query to wait for first page to show in Visor. > >> > > >> > Please take a look at VisorQueryJob code (line 75...). > >> > > >> > By default Visor use 100 rows as page size. > >> > > >> > I expect that take first 100 rows for scan query should be very fast. > >> > > >> > On Thu, May 7, 2015 at 10:19 PM, Sergi Vladykin < > >> [hidden email]> > >> > wrote: > >> > > >> > > Alexey, > >> > > > >> > > How did you come to this conclusion? I don't see anything like that > in > >> > our > >> > > code, we just take keySet or primaryKeySet iterator and fetch it > page > >> by > >> > > page. > >> > > > >> > > Sergi > >> > > > >> > > 2015-05-07 9:58 GMT+03:00 Dmitriy Setrakyan <[hidden email] > >: > >> > > > >> > > > I have filed a ticket for this issue: > >> > > > https://issues.apache.org/jira/browse/IGNITE-872 > >> > > > > >> > > > Sergey, since it has to do with queries, can you please take a > look? > >> > > > > >> > > > Thanks, > >> > > > D. > >> > > > > >> > > > On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov < > >> > > [hidden email]> > >> > > > wrote: > >> > > > > >> > > > > Hello All! > >> > > > > > >> > > > > Just found that if I have many entries in cache and do the scan > >> query > >> > > > with > >> > > > > page size 100 > >> > > > > then my query seems to iterate over the whole cache and only > after > >> > > that I > >> > > > > could fetch first page of data. > >> > > > > > >> > > > > I think that it should be reworked to return first page as fast > as > >> > > > possible > >> > > > > - without of waiting while scan query will be completed. > >> > > > > > >> > > > > Thoughts? > >> > > > > > >> > > > > -- > >> > > > > Alexey Kuznetsov > >> > > > > GridGain Systems > >> > > > > www.gridgain.com > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > -- > >> > Alexey Kuznetsov > >> > GridGain Systems > >> > www.gridgain.com > >> > > >> > > > > > > > > -- > > Alexey Kuznetsov > > GridGain Systems > > www.gridgain.com > > > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com > |
Dmitriy already created issue (
https://issues.apache.org/jira/browse/IGNITE-872). Valentin already found problem root (and added comment to issue). Sergi should look and fix. On Fri, May 8, 2015 at 3:45 PM, Yakov Zhdanov <[hidden email]> wrote: > Alex, can you please create a junit and ticket? > > --Yakov > > 2015-05-08 9:43 GMT+03:00 Alexey Kuznetsov <[hidden email]>: > > > Checked with 2 nodes - scan query works fast. > > So, problem in case of single node. > > > > On Fri, May 8, 2015 at 8:17 AM, Alexey Kuznetsov < > [hidden email]> > > wrote: > > > > > Val, yes I remember that I tested with one node. > > > But I will recheck with several nodes today and give more info. > > > > > > On Fri, May 8, 2015 at 2:38 AM, Valentin Kulichenko < > > > [hidden email]> wrote: > > > > > >> Alexey, > > >> > > >> Do you have only one server node? In this case query is considered to > be > > >> local and pagination is disabled. Really looks like redundant > iteration. > > >> > > >> I also noticed that IgniteCacheProxy.query(Query filter, @Nullable > > >> ClusterGroup grp) method on line 347 ignores page size parameter > (added > > a > > >> comment in ticket). > > >> > > >> -- > > >> Val > > >> > > >> On Thu, May 7, 2015 at 8:32 AM, Alexey Kuznetsov < > > [hidden email] > > >> > > > >> wrote: > > >> > > >> > Sergi. > > >> > > > >> > I put 1M entries in cache. Execute scan query from Visor and see 1M > > >> reads > > >> > in cache metrics (and I do not even hit "Next page" button). > > >> > Also more entries in cache results in more longer execution time of > > scan > > >> > query to wait for first page to show in Visor. > > >> > > > >> > Please take a look at VisorQueryJob code (line 75...). > > >> > > > >> > By default Visor use 100 rows as page size. > > >> > > > >> > I expect that take first 100 rows for scan query should be very > fast. > > >> > > > >> > On Thu, May 7, 2015 at 10:19 PM, Sergi Vladykin < > > >> [hidden email]> > > >> > wrote: > > >> > > > >> > > Alexey, > > >> > > > > >> > > How did you come to this conclusion? I don't see anything like > that > > in > > >> > our > > >> > > code, we just take keySet or primaryKeySet iterator and fetch it > > page > > >> by > > >> > > page. > > >> > > > > >> > > Sergi > > >> > > > > >> > > 2015-05-07 9:58 GMT+03:00 Dmitriy Setrakyan < > [hidden email] > > >: > > >> > > > > >> > > > I have filed a ticket for this issue: > > >> > > > https://issues.apache.org/jira/browse/IGNITE-872 > > >> > > > > > >> > > > Sergey, since it has to do with queries, can you please take a > > look? > > >> > > > > > >> > > > Thanks, > > >> > > > D. > > >> > > > > > >> > > > On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov < > > >> > > [hidden email]> > > >> > > > wrote: > > >> > > > > > >> > > > > Hello All! > > >> > > > > > > >> > > > > Just found that if I have many entries in cache and do the > scan > > >> query > > >> > > > with > > >> > > > > page size 100 > > >> > > > > then my query seems to iterate over the whole cache and only > > after > > >> > > that I > > >> > > > > could fetch first page of data. > > >> > > > > > > >> > > > > I think that it should be reworked to return first page as > fast > > as > > >> > > > possible > > >> > > > > - without of waiting while scan query will be completed. > > >> > > > > > > >> > > > > Thoughts? > > >> > > > > > > >> > > > > -- > > >> > > > > Alexey Kuznetsov > > >> > > > > GridGain Systems > > >> > > > > www.gridgain.com > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Alexey Kuznetsov > > >> > GridGain Systems > > >> > www.gridgain.com > > >> > > > >> > > > > > > > > > > > > -- > > > Alexey Kuznetsov > > > GridGain Systems > > > www.gridgain.com > > > > > > > > > > > -- > > Alexey Kuznetsov > > GridGain Systems > > www.gridgain.com > > > -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
Ok, got it.
Sergi 2015-05-08 12:27 GMT+03:00 Alexey Kuznetsov <[hidden email]>: > Dmitriy already created issue ( > https://issues.apache.org/jira/browse/IGNITE-872). > Valentin already found problem root (and added comment to issue). > Sergi should look and fix. > > On Fri, May 8, 2015 at 3:45 PM, Yakov Zhdanov <[hidden email]> wrote: > > > Alex, can you please create a junit and ticket? > > > > --Yakov > > > > 2015-05-08 9:43 GMT+03:00 Alexey Kuznetsov <[hidden email]>: > > > > > Checked with 2 nodes - scan query works fast. > > > So, problem in case of single node. > > > > > > On Fri, May 8, 2015 at 8:17 AM, Alexey Kuznetsov < > > [hidden email]> > > > wrote: > > > > > > > Val, yes I remember that I tested with one node. > > > > But I will recheck with several nodes today and give more info. > > > > > > > > On Fri, May 8, 2015 at 2:38 AM, Valentin Kulichenko < > > > > [hidden email]> wrote: > > > > > > > >> Alexey, > > > >> > > > >> Do you have only one server node? In this case query is considered > to > > be > > > >> local and pagination is disabled. Really looks like redundant > > iteration. > > > >> > > > >> I also noticed that IgniteCacheProxy.query(Query filter, @Nullable > > > >> ClusterGroup grp) method on line 347 ignores page size parameter > > (added > > > a > > > >> comment in ticket). > > > >> > > > >> -- > > > >> Val > > > >> > > > >> On Thu, May 7, 2015 at 8:32 AM, Alexey Kuznetsov < > > > [hidden email] > > > >> > > > > >> wrote: > > > >> > > > >> > Sergi. > > > >> > > > > >> > I put 1M entries in cache. Execute scan query from Visor and see > 1M > > > >> reads > > > >> > in cache metrics (and I do not even hit "Next page" button). > > > >> > Also more entries in cache results in more longer execution time > of > > > scan > > > >> > query to wait for first page to show in Visor. > > > >> > > > > >> > Please take a look at VisorQueryJob code (line 75...). > > > >> > > > > >> > By default Visor use 100 rows as page size. > > > >> > > > > >> > I expect that take first 100 rows for scan query should be very > > fast. > > > >> > > > > >> > On Thu, May 7, 2015 at 10:19 PM, Sergi Vladykin < > > > >> [hidden email]> > > > >> > wrote: > > > >> > > > > >> > > Alexey, > > > >> > > > > > >> > > How did you come to this conclusion? I don't see anything like > > that > > > in > > > >> > our > > > >> > > code, we just take keySet or primaryKeySet iterator and fetch it > > > page > > > >> by > > > >> > > page. > > > >> > > > > > >> > > Sergi > > > >> > > > > > >> > > 2015-05-07 9:58 GMT+03:00 Dmitriy Setrakyan < > > [hidden email] > > > >: > > > >> > > > > > >> > > > I have filed a ticket for this issue: > > > >> > > > https://issues.apache.org/jira/browse/IGNITE-872 > > > >> > > > > > > >> > > > Sergey, since it has to do with queries, can you please take a > > > look? > > > >> > > > > > > >> > > > Thanks, > > > >> > > > D. > > > >> > > > > > > >> > > > On Thu, May 7, 2015 at 1:47 AM, Alexey Kuznetsov < > > > >> > > [hidden email]> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > Hello All! > > > >> > > > > > > > >> > > > > Just found that if I have many entries in cache and do the > > scan > > > >> query > > > >> > > > with > > > >> > > > > page size 100 > > > >> > > > > then my query seems to iterate over the whole cache and only > > > after > > > >> > > that I > > > >> > > > > could fetch first page of data. > > > >> > > > > > > > >> > > > > I think that it should be reworked to return first page as > > fast > > > as > > > >> > > > possible > > > >> > > > > - without of waiting while scan query will be completed. > > > >> > > > > > > > >> > > > > Thoughts? > > > >> > > > > > > > >> > > > > -- > > > >> > > > > Alexey Kuznetsov > > > >> > > > > GridGain Systems > > > >> > > > > www.gridgain.com > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > Alexey Kuznetsov > > > >> > GridGain Systems > > > >> > www.gridgain.com > > > >> > > > > >> > > > > > > > > > > > > > > > > -- > > > > Alexey Kuznetsov > > > > GridGain Systems > > > > www.gridgain.com > > > > > > > > > > > > > > > > -- > > > Alexey Kuznetsov > > > GridGain Systems > > > www.gridgain.com > > > > > > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com > |
Free forum by Nabble | Edit this page |