How a new index is built in runtime?

[dmagda]

Alex P., Vladimir,

Having CREATE INDEX command we can define indexes in runtime. However, it’s unclear how a new index is built.

Let’s imagine I have a field “name” that was in Person’s model for a while and there are millions of such objects in the cluster. Now I turned the field into the index in runtime. How Ignite is going to built the index? Do we iterate over the millions of objects with the field in background or at the time of the CREATE INDEX execution blocking the latter? Or is there more sophisticated process?

—
Denis

[yzhdanov]

Denis, you are absolutely right. Iterating through millions of objects is
indeed a very complex task. Ignite casts a secret spell instead.

--Yakov

2017-08-17 21:26 GMT+01:00 Denis Magda <[hidden email]>:

> Alex P., Vladimir,
>
> Having CREATE INDEX command we can define indexes in runtime. However,
> it’s unclear how a new index is built.
>
> Let’s imagine I have a field “name” that was in Person’s model for a while
> and there are millions of such objects in the cluster. Now I turned the
> field into the index in runtime. How Ignite is going to built the index? Do
> we iterate over the millions of objects with the field in background or at
> the time of the CREATE INDEX execution blocking the latter? Or is there
> more sophisticated process?
>
> —
> Denis

[dsetrakyan]

On Thu, Aug 17, 2017 at 1:47 PM, Yakov Zhdanov <[hidden email]> wrote:

> Denis, you are absolutely right. Iterating through millions of objects is
> indeed a very complex task. Ignite casts a secret spell instead.\
>

Well, Ignite should cast a secret spell and do it in parallel, across
multiple threads. This will make index creation faster, but will occupy
more cores. Perhaps we should have both, single-threaded and multi-threaded
options.

>
> --Yakov
>
> 2017-08-17 21:26 GMT+01:00 Denis Magda <[hidden email]>:
>
> > Alex P., Vladimir,
> >
> > Having CREATE INDEX command we can define indexes in runtime. However,
> > it’s unclear how a new index is built.
> >
> > Let’s imagine I have a field “name” that was in Person’s model for a
> while
> > and there are millions of such objects in the cluster. Now I turned the
> > field into the index in runtime. How Ignite is going to built the index?
> Do
> > we iterate over the millions of objects with the field in background or
> at
> > the time of the CREATE INDEX execution blocking the latter? Or is there
> > more sophisticated process?
> >
> > —
> > Denis
>

[yzhdanov]

Of course, iteration should have an option to be run from more than 1
thread. What will really help, IMO, is ability to insert presorted batches
in a single tree operation.

--
Yakov Zhdanov

[Vladimir Ozerov]

Denis,

We iterate over the whole cache and build the index entry-by-entry. Control
is returned back to the user when index is ready.

On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <[hidden email]> wrote:

> Of course, iteration should have an option to be run from more than 1
> thread. What will really help, IMO, is ability to insert presorted batches
> in a single tree operation.
>
> --
> Yakov Zhdanov
>

[dsetrakyan]

Vova, how hard is it to make it multi-threaded?

⁣D.​


On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov <[hidden email]> wrote:

>Denis,
>
>We iterate over the whole cache and build the index entry-by-entry.
>Control
>is returned back to the user when index is ready.
>
>On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <[hidden email]>
>wrote:
>
>> Of course, iteration should have an option to be run from more than 1
>> thread. What will really help, IMO, is ability to insert presorted
>batches
>> in a single tree operation.
>>
>> --
>> Yakov Zhdanov
>>

[Sergey Kozlov]

Vova

Is the data available during an index creation?

On Mon, Aug 28, 2017 at 3:46 PM, <[hidden email]> wrote:

> Vova, how hard is it to make it multi-threaded?
>
> ⁣D.​
>
>
> On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov <
> [hidden email]> wrote:
> >Denis,
> >
> >We iterate over the whole cache and build the index entry-by-entry.
> >Control
> >is returned back to the user when index is ready.
> >
> >On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <[hidden email]>
> >wrote:
> >
> >> Of course, iteration should have an option to be run from more than 1
> >> thread. What will really help, IMO, is ability to insert presorted
> >batches
> >> in a single tree operation.
> >>
> >> --
> >> Yakov Zhdanov
> >>
>

--
Sergey Kozlov
GridGain Systems
www.gridgain.com

[Vladimir Ozerov]

Should not be too hard. However, this will not change situation a lot.
Instead, we need to implement more efficient algorithm of index filling.
E.g. fill it with pre-sorted batches. We already have a ticket for that.

On Mon, Aug 28, 2017 at 3:46 PM, <[hidden email]> wrote:

> Vova, how hard is it to make it multi-threaded?
>
> ⁣D.​
>
>
> On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov <
> [hidden email]> wrote:
> >Denis,
> >
> >We iterate over the whole cache and build the index entry-by-entry.
> >Control
> >is returned back to the user when index is ready.
> >
> >On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <[hidden email]>
> >wrote:
> >
> >> Of course, iteration should have an option to be run from more than 1
> >> thread. What will really help, IMO, is ability to insert presorted
> >batches
> >> in a single tree operation.
> >>
> >> --
> >> Yakov Zhdanov
> >>
>

[dmagda]

Vladimir,

Is this a sync or async operation?

—
Denis

> On Aug 28, 2017, at 1:05 AM, Vladimir Ozerov <[hidden email]> wrote:
>
> Denis,
>
> We iterate over the whole cache and build the index entry-by-entry. Control
> is returned back to the user when index is ready.
>
> On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <[hidden email]> wrote:
>
>> Of course, iteration should have an option to be run from more than 1
>> thread. What will really help, IMO, is ability to insert presorted batches
>> in a single tree operation.
>>
>> --
>> Yakov Zhdanov
>>

[Vladimir Ozerov]

Sync.

пн, 28 авг. 2017 г. в 19:34, Denis Magda <[hidden email]>:

> Vladimir,
>
> Is this a sync or async operation?
>
> —
> Denis
>
> > On Aug 28, 2017, at 1:05 AM, Vladimir Ozerov <[hidden email]>
> wrote:
> >
> > Denis,
> >
> > We iterate over the whole cache and build the index entry-by-entry.
> Control
> > is returned back to the user when index is ready.
> >
> > On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <[hidden email]>
> wrote:
> >
> >> Of course, iteration should have an option to be run from more than 1
> >> thread. What will really help, IMO, is ability to insert presorted
> batches
> >> in a single tree operation.
> >>
> >> --
> >> Yakov Zhdanov
> >>
>
>