Initial data load and indexes

[Alexey Kuznetsov-2]

Hi,

When I do initial load of data (with a lot of indexed fields) into a cache
I see GC activity on indexing internal objects.

It is possible to turn off reindexing while I'm loading data and do reindex
after initial load is done?

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

[yzhdanov]

Sergi, please take a look at this.

--Yakov

2015-05-08 6:51 GMT+03:00 Alexey Kuznetsov <[hidden email]>:

> Hi,
>
> When I do initial load of data (with a lot of indexed fields) into a cache
> I see GC activity on indexing internal objects.
>
> It is possible to turn off reindexing while I'm loading data and do reindex
> after initial load is done?
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>

[Alexey Kuznetsov-2]

Issue created: https://issues.apache.org/jira/browse/IGNITE-878
Vasiliy will create simple test to reproduce issue.
And after that will assign issue to Sergi.

On Fri, May 8, 2015 at 3:33 PM, Yakov Zhdanov <[hidden email]> wrote:

> Sergi, please take a look at this.
>
> --Yakov
>
> 2015-05-08 6:51 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
>
> > Hi,
> >
> > When I do initial load of data (with a lot of indexed fields) into a
> cache
> > I see GC activity on indexing internal objects.
> >
> > It is possible to turn off reindexing while I'm loading data and do
> reindex
> > after initial load is done?
> >
> > --
> > Alexey Kuznetsov
> > GridGain Systems
> > www.gridgain.com
> >
>

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

[Sergi]

It does not make any sense to disable indexing on data loading and to index
later because it is exactly the same amount of work.
Anyways lets create test and investigate this.

Sergi

2015-05-08 11:39 GMT+03:00 Alexey Kuznetsov <[hidden email]>:

> Issue created: https://issues.apache.org/jira/browse/IGNITE-878
> Vasiliy will create simple test to reproduce issue.
> And after that will assign issue to Sergi.
>
> On Fri, May 8, 2015 at 3:33 PM, Yakov Zhdanov <[hidden email]> wrote:
>
> > Sergi, please take a look at this.
> >
> > --Yakov
> >
> > 2015-05-08 6:51 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
> >
> > > Hi,
> > >
> > > When I do initial load of data (with a lot of indexed fields) into a
> > cache
> > > I see GC activity on indexing internal objects.
> > >
> > > It is possible to turn off reindexing while I'm loading data and do
> > reindex
> > > after initial load is done?
> > >
> > > --
> > > Alexey Kuznetsov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>

[Alexey Kuznetsov-2]

My main point is avoid not needed GC.
Vasiliy and Pavel will prepare test for you.

On Fri, May 8, 2015 at 10:48 PM, Sergi Vladykin <[hidden email]>
wrote:

> It does not make any sense to disable indexing on data loading and to index
> later because it is exactly the same amount of work.
> Anyways lets create test and investigate this.
>
> Sergi
>
> 2015-05-08 11:39 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
>
> > Issue created: https://issues.apache.org/jira/browse/IGNITE-878
> > Vasiliy will create simple test to reproduce issue.
> > And after that will assign issue to Sergi.
> >
> > On Fri, May 8, 2015 at 3:33 PM, Yakov Zhdanov <[hidden email]>
> wrote:
> >
> > > Sergi, please take a look at this.
> > >
> > > --Yakov
> > >
> > > 2015-05-08 6:51 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
> > >
> > > > Hi,
> > > >
> > > > When I do initial load of data (with a lot of indexed fields) into a
> > > cache
> > > > I see GC activity on indexing internal objects.
> > > >
> > > > It is possible to turn off reindexing while I'm loading data and do
> > > reindex
> > > > after initial load is done?
> > > >
> > > > --
> > > > Alexey Kuznetsov
> > > > GridGain Systems
> > > > www.gridgain.com
> > > >
> > >
> >
> >
> >
> > --
> > Alexey Kuznetsov
> > GridGain Systems
> > www.gridgain.com
> >
>

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

[Andrey Kornev]

There is one situation where disabling indexing during the data load might make a difference. When the source of my data is a log-based store (like Kafka) with multiple versions of the same fact, I'd really want to play the entire log back first, and then get the cache indexed...

Andrey

> Date: Fri, 8 May 2015 23:16:02 +0700
> Subject: Re: Initial data load and indexes
> From: [hidden email]
> To: [hidden email]
>
> My main point is avoid not needed GC.
> Vasiliy and Pavel will prepare test for you.
>
> On Fri, May 8, 2015 at 10:48 PM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > It does not make any sense to disable indexing on data loading and to index
> > later because it is exactly the same amount of work.
> > Anyways lets create test and investigate this.
> >
> > Sergi
> >
> > 2015-05-08 11:39 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
> >
> > > Issue created: https://issues.apache.org/jira/browse/IGNITE-878
> > > Vasiliy will create simple test to reproduce issue.
> > > And after that will assign issue to Sergi.
> > >
> > > On Fri, May 8, 2015 at 3:33 PM, Yakov Zhdanov <[hidden email]>
> > wrote:
> > >
> > > > Sergi, please take a look at this.
> > > >
> > > > --Yakov
> > > >
> > > > 2015-05-08 6:51 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
> > > >
> > > > > Hi,
> > > > >
> > > > > When I do initial load of data (with a lot of indexed fields) into a
> > > > cache
> > > > > I see GC activity on indexing internal objects.
> > > > >
> > > > > It is possible to turn off reindexing while I'm loading data and do
> > > > reindex
> > > > > after initial load is done?
> > > > >
> > > > > --
> > > > > Alexey Kuznetsov
> > > > > GridGain Systems
> > > > > www.gridgain.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Alexey Kuznetsov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com

     

[dsetrakyan]

On Fri, May 8, 2015 at 8:48 AM, Sergi Vladykin <[hidden email]>
wrote:

> It does not make any sense to disable indexing on data loading and to index
> later because it is exactly the same amount of work.
>

I agree. If we delay indexing, what do we do with queries in the mean time?


> Anyways lets create test and investigate this.
>

Sounds good.

>
> Sergi
>
> 2015-05-08 11:39 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
>
> > Issue created: https://issues.apache.org/jira/browse/IGNITE-878
> > Vasiliy will create simple test to reproduce issue.
> > And after that will assign issue to Sergi.
> >
> > On Fri, May 8, 2015 at 3:33 PM, Yakov Zhdanov <[hidden email]>
> wrote:
> >
> > > Sergi, please take a look at this.
> > >
> > > --Yakov
> > >
> > > 2015-05-08 6:51 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
> > >
> > > > Hi,
> > > >
> > > > When I do initial load of data (with a lot of indexed fields) into a
> > > cache
> > > > I see GC activity on indexing internal objects.
> > > >
> > > > It is possible to turn off reindexing while I'm loading data and do
> > > reindex
> > > > after initial load is done?
> > > >
> > > > --
> > > > Alexey Kuznetsov
> > > > GridGain Systems
> > > > www.gridgain.com
> > > >
> > >
> >
> >
> >
> > --
> > Alexey Kuznetsov
> > GridGain Systems
> > www.gridgain.com
> >
>

[Alexey Kuznetsov-2]

>> I agree. If we delay indexing, what do we do with queries in the mean
time?
Work slow :)

IMHO, initial load is usually performed in exclusive mode.


On Sat, May 9, 2015 at 12:40 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Fri, May 8, 2015 at 8:48 AM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > It does not make any sense to disable indexing on data loading and to
> index
> > later because it is exactly the same amount of work.
> >
>
> I agree. If we delay indexing, what do we do with queries in the mean time?
>
>
> > Anyways lets create test and investigate this.
> >
>
> Sounds good.
>
>
> >
> > Sergi
> >
> > 2015-05-08 11:39 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
> >
> > > Issue created: https://issues.apache.org/jira/browse/IGNITE-878
> > > Vasiliy will create simple test to reproduce issue.
> > > And after that will assign issue to Sergi.
> > >
> > > On Fri, May 8, 2015 at 3:33 PM, Yakov Zhdanov <[hidden email]>
> > wrote:
> > >
> > > > Sergi, please take a look at this.
> > > >
> > > > --Yakov
> > > >
> > > > 2015-05-08 6:51 GMT+03:00 Alexey Kuznetsov <[hidden email]
> >:
> > > >
> > > > > Hi,
> > > > >
> > > > > When I do initial load of data (with a lot of indexed fields) into
> a
> > > > cache
> > > > > I see GC activity on indexing internal objects.
> > > > >
> > > > > It is possible to turn off reindexing while I'm loading data and do
> > > > reindex
> > > > > after initial load is done?
> > > > >
> > > > > --
> > > > > Alexey Kuznetsov
> > > > > GridGain Systems
> > > > > www.gridgain.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Alexey Kuznetsov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com