Geo-spatial and full-text indexes support in Ignite 2.x

[Vladimir Ozerov]

Guys,

AFAIK these two index types are not supported with enabled persistence at
the moment, neither they stored on the disk anyhow. Can someone help with
estimates on how difficult would it be to implement these indexes over
page-memory architecture?

Looks like we will have to write our own implementation of these indexes,
instead of relying on Lucene and H2. Am I right?

Vladimir.

[dmagda]

Vladimir,

We need to consider that these two types of indexes are not stored off-heap either.

It expands the task a bit — the indexes have to be fully integrated with the new durable memory architecture supporting both off-heap and persistence layers.

—
Denis

> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <[hidden email]> wrote:
>
> Guys,
>
> AFAIK these two index types are not supported with enabled persistence at
> the moment, neither they stored on the disk anyhow. Can someone help with
> estimates on how difficult would it be to implement these indexes over
> page-memory architecture?
>
> Looks like we will have to write our own implementation of these indexes,
> instead of relying on Lucene and H2. Am I right?
>
> Vladimir.

[Andrew Mashenkov]

Denis,

Lucene fullText index stores off-heap in old way, not in page memory.
Therefore it is not persistent.

There is a second issue related to FullText index as it is different kind
of index and it has own query type and it can't be use in SQL queries.
Looks like it make sense to integrate FullText indices in our SQL layer as
well.

AFAIK, H2 has some support of FullText indices based on Lucene, so we can
get a hint how we can integrate it.



On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <[hidden email]> wrote:

> Vladimir,
>
> We need to consider that these two types of indexes are not stored
> off-heap either.
>
> It expands the task a bit — the indexes have to be fully integrated with
> the new durable memory architecture supporting both off-heap and
> persistence layers.
>
> —
> Denis
>
> > On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <[hidden email]>
> wrote:
> >
> > Guys,
> >
> > AFAIK these two index types are not supported with enabled persistence at
> > the moment, neither they stored on the disk anyhow. Can someone help with
> > estimates on how difficult would it be to implement these indexes over
> > page-memory architecture?
> >
> > Looks like we will have to write our own implementation of these indexes,
> > instead of relying on Lucene and H2. Am I right?
> >
> > Vladimir.
>
>

--
Best regards,
Andrey V. Mashenkov

[dmagda]

Andrey,

I’ve already wiped the old off-heap memory out of my mind. Sure I assumed the off-heap space managed by the new memory architecture.

—
Denis

> On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov <[hidden email]> wrote:
>
> Denis,
>
> Lucene fullText index stores off-heap in old way, not in page memory.
> Therefore it is not persistent.
>
> There is a second issue related to FullText index as it is different kind
> of index and it has own query type and it can't be use in SQL queries.
> Looks like it make sense to integrate FullText indices in our SQL layer as
> well.
>
> AFAIK, H2 has some support of FullText indices based on Lucene, so we can
> get a hint how we can integrate it.
>
>
>
> On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <[hidden email]> wrote:
>
>> Vladimir,
>>
>> We need to consider that these two types of indexes are not stored
>> off-heap either.
>>
>> It expands the task a bit — the indexes have to be fully integrated with
>> the new durable memory architecture supporting both off-heap and
>> persistence layers.
>>
>> —
>> Denis
>>
>>> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <[hidden email]>
>> wrote:
>>>
>>> Guys,
>>>
>>> AFAIK these two index types are not supported with enabled persistence at
>>> the moment, neither they stored on the disk anyhow. Can someone help with
>>> estimates on how difficult would it be to implement these indexes over
>>> page-memory architecture?
>>>
>>> Looks like we will have to write our own implementation of these indexes,
>>> instead of relying on Lucene and H2. Am I right?
>>>
>>> Vladimir.
>>
>>
>
>
> --
> Best regards,
> Andrey V. Mashenkov

[Andrew Mashenkov]

Denis,

I see we still use GridUnsafeMemory in LuceneIndex [1] in master.
Do you mean a wrapper to new memory manager is used here?

[1]
https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridLuceneIndex.java

On Tue, Aug 1, 2017 at 10:28 PM, Denis Magda <[hidden email]> wrote:

> Andrey,
>
> I’ve already wiped the old off-heap memory out of my mind. Sure I assumed
> the off-heap space managed by the new memory architecture.
>
> —
> Denis
>
> > On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov <
> [hidden email]> wrote:
> >
> > Denis,
> >
> > Lucene fullText index stores off-heap in old way, not in page memory.
> > Therefore it is not persistent.
> >
> > There is a second issue related to FullText index as it is different kind
> > of index and it has own query type and it can't be use in SQL queries.
> > Looks like it make sense to integrate FullText indices in our SQL layer
> as
> > well.
> >
> > AFAIK, H2 has some support of FullText indices based on Lucene, so we can
> > get a hint how we can integrate it.
> >
> >
> >
> > On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <[hidden email]> wrote:
> >
> >> Vladimir,
> >>
> >> We need to consider that these two types of indexes are not stored
> >> off-heap either.
> >>
> >> It expands the task a bit — the indexes have to be fully integrated with
> >> the new durable memory architecture supporting both off-heap and
> >> persistence layers.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <[hidden email]>
> >> wrote:
> >>>
> >>> Guys,
> >>>
> >>> AFAIK these two index types are not supported with enabled persistence
> at
> >>> the moment, neither they stored on the disk anyhow. Can someone help
> with
> >>> estimates on how difficult would it be to implement these indexes over
> >>> page-memory architecture?
> >>>
> >>> Looks like we will have to write our own implementation of these
> indexes,
> >>> instead of relying on Lucene and H2. Am I right?
> >>>
> >>> Vladimir.
> >>
> >>
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
>
>

--
Best regards,
Andrey V. Mashenkov

[dmagda]

Andrey,

What I’m trying to say is that presently both text and geo-spatial indexes are unsupported NOT only for the persistence layer but for the durable (page) memory in general - as you properly pointed out the text indexes are stored in the old off-heap memory but has to be moved to be handled in page memory’s off-heap region directly.

Makes sense?

—
Denis

> On Aug 1, 2017, at 1:05 PM, Andrey Mashenkov <[hidden email]> wrote:
>
> Denis,
>
> I see we still use GridUnsafeMemory in LuceneIndex [1] in master.
> Do you mean a wrapper to new memory manager is used here?
>
> [1]
> https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridLuceneIndex.java
>
> On Tue, Aug 1, 2017 at 10:28 PM, Denis Magda <[hidden email]> wrote:
>
>> Andrey,
>>
>> I’ve already wiped the old off-heap memory out of my mind. Sure I assumed
>> the off-heap space managed by the new memory architecture.
>>
>> —
>> Denis
>>
>>> On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov <
>> [hidden email]> wrote:
>>>
>>> Denis,
>>>
>>> Lucene fullText index stores off-heap in old way, not in page memory.
>>> Therefore it is not persistent.
>>>
>>> There is a second issue related to FullText index as it is different kind
>>> of index and it has own query type and it can't be use in SQL queries.
>>> Looks like it make sense to integrate FullText indices in our SQL layer
>> as
>>> well.
>>>
>>> AFAIK, H2 has some support of FullText indices based on Lucene, so we can
>>> get a hint how we can integrate it.
>>>
>>>
>>>
>>> On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <[hidden email]> wrote:
>>>
>>>> Vladimir,
>>>>
>>>> We need to consider that these two types of indexes are not stored
>>>> off-heap either.
>>>>
>>>> It expands the task a bit — the indexes have to be fully integrated with
>>>> the new durable memory architecture supporting both off-heap and
>>>> persistence layers.
>>>>
>>>> —
>>>> Denis
>>>>
>>>>> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <[hidden email]>
>>>> wrote:
>>>>>
>>>>> Guys,
>>>>>
>>>>> AFAIK these two index types are not supported with enabled persistence
>> at
>>>>> the moment, neither they stored on the disk anyhow. Can someone help
>> with
>>>>> estimates on how difficult would it be to implement these indexes over
>>>>> page-memory architecture?
>>>>>
>>>>> Looks like we will have to write our own implementation of these
>> indexes,
>>>>> instead of relying on Lucene and H2. Am I right?
>>>>>
>>>>> Vladimir.
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>
>>
>
>
> --
> Best regards,
> Andrey V. Mashenkov

[Andrew Mashenkov]

Vladimir,

Is my understanding right you suggest to create a custom Lucene codec which
will be able to store index in ignite page memory?

Lucene javadoc has complete description of codec API [1] and looks like it
is possible to implement custom one.

[1]
https://lucene.apache.org/core/5_5_2/core/org/apache/lucene/codecs/lucene54/package-summary.html#package_description

On Wed, Aug 2, 2017 at 12:02 AM, Denis Magda <[hidden email]> wrote:

> Andrey,
>
> What I’m trying to say is that presently both text and geo-spatial indexes
> are unsupported NOT only for the persistence layer but for the durable
> (page) memory in general - as you properly pointed out the text indexes are
> stored in the old off-heap memory but has to be moved to be handled in page
> memory’s off-heap region directly.
>
> Makes sense?
>
> —
> Denis
>
> > On Aug 1, 2017, at 1:05 PM, Andrey Mashenkov <[hidden email]>
> wrote:
> >
> > Denis,
> >
> > I see we still use GridUnsafeMemory in LuceneIndex [1] in master.
> > Do you mean a wrapper to new memory manager is used here?
> >
> > [1]
> > https://github.com/apache/ignite/blob/master/modules/
> indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/
> GridLuceneIndex.java
> >
> > On Tue, Aug 1, 2017 at 10:28 PM, Denis Magda <[hidden email]> wrote:
> >
> >> Andrey,
> >>
> >> I’ve already wiped the old off-heap memory out of my mind. Sure I
> assumed
> >> the off-heap space managed by the new memory architecture.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov <
> >> [hidden email]> wrote:
> >>>
> >>> Denis,
> >>>
> >>> Lucene fullText index stores off-heap in old way, not in page memory.
> >>> Therefore it is not persistent.
> >>>
> >>> There is a second issue related to FullText index as it is different
> kind
> >>> of index and it has own query type and it can't be use in SQL queries.
> >>> Looks like it make sense to integrate FullText indices in our SQL layer
> >> as
> >>> well.
> >>>
> >>> AFAIK, H2 has some support of FullText indices based on Lucene, so we
> can
> >>> get a hint how we can integrate it.
> >>>
> >>>
> >>>
> >>> On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <[hidden email]> wrote:
> >>>
> >>>> Vladimir,
> >>>>
> >>>> We need to consider that these two types of indexes are not stored
> >>>> off-heap either.
> >>>>
> >>>> It expands the task a bit — the indexes have to be fully integrated
> with
> >>>> the new durable memory architecture supporting both off-heap and
> >>>> persistence layers.
> >>>>
> >>>> —
> >>>> Denis
> >>>>
> >>>>> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <[hidden email]>
> >>>> wrote:
> >>>>>
> >>>>> Guys,
> >>>>>
> >>>>> AFAIK these two index types are not supported with enabled
> persistence
> >> at
> >>>>> the moment, neither they stored on the disk anyhow. Can someone help
> >> with
> >>>>> estimates on how difficult would it be to implement these indexes
> over
> >>>>> page-memory architecture?
> >>>>>
> >>>>> Looks like we will have to write our own implementation of these
> >> indexes,
> >>>>> instead of relying on Lucene and H2. Am I right?
> >>>>>
> >>>>> Vladimir.
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Best regards,
> >>> Andrey V. Mashenkov
> >>
> >>
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
>
>

--
Best regards,
Andrey V. Mashenkov