Basic index infrastructure as a part of core APIs

[gvvinblade]

Igniters,

Some of you knows I'm working on a new Apache Calcite based SQL query
execution engine now[1] and I succeeded a little in this [2].

This activity becomes more important after H2 community decided to remove a
set of interfaces that we are using in our integration[3].

In scope of my current activity I need to separate indexes related system
parts from H2, and reuse them in Calcite.

I prepared an IEP[4] that is aimed to bring basic various index types
support to Ignite.

In addition to solve my issue, It, I hope, will help to integrate full text
search indexes into Ignite better and provide a way to add new index types
support (like geospatial indexes).

Also, as a next step, it allows to implement lightweight sorted bounded
scan queries, this feature, as far as I know, was requested several times.

I'll appreciate it you, guys share your thoughts on that.

1.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
2. https://github.com/apache/ignite/pull/7071
3. https://github.com/h2database/h2database/issues/2226
4.
https://cwiki.apache.org/confluence/display/IGNITE/IEP-49%3A+Basic+index+infrastructure+as+a+part+of+core+Ignite+APIs

Regards,
Igor

[Roman Kondakov]

Hi Igor,

thank you for your effort!

I have a question about the `Index.find(Args)` method. IEP says that it
will return `a cursor over found rows`. But what if we don't need a row
itself? For example in the case of possible optimizations of index scan
that we can implement later. I can name a couple of them:

- index-only scan [1]: we don't need a materialized row itself, we need
only the indexed value from the index page.
- multi-index scan (aka PG's bitmap scan [2]) where we can retrieve only
row links from index(es), then perform union/intersection operations
over them and retrieve from the RowStore only those rows that are
actually needed.

In this case we don't need a row itself, but a row's physical id (aka
link) or the indexed value from the index page.

What do you think? Should we have several `Index.find` methods with
different return value? Or there might be another solution for such
optimizations?

Thank you.

[1] https://www.postgresql.org/docs/10/indexes-index-only-scans.html
[2] https://www.postgresql.org/message-id/12553.1135634231@...

--
Kind Regards
Roman Kondakov


On 22.05.2020 22:37, Igor Seliverstov wrote:

> Igniters,
>
> Some of you knows I'm working on a new Apache Calcite based SQL query
> execution engine now[1] and I succeeded a little in this [2].
>
> This activity becomes more important after H2 community decided to remove a
> set of interfaces that we are using in our integration[3].
>
> In scope of my current activity I need to separate indexes related system
> parts from H2, and reuse them in Calcite.
>
> I prepared an IEP[4] that is aimed to bring basic various index types
> support to Ignite.
>
> In addition to solve my issue, It, I hope, will help to integrate full text
> search indexes into Ignite better and provide a way to add new index types
> support (like geospatial indexes).
>
> Also, as a next step, it allows to implement lightweight sorted bounded
> scan queries, this feature, as far as I know, was requested several times.
>
> I'll appreciate it you, guys share your thoughts on that.
>
> 1.
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
> 2. https://github.com/apache/ignite/pull/7071
> 3. https://github.com/h2database/h2database/issues/2226
> 4.
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-49%3A+Basic+index+infrastructure+as+a+part+of+core+Ignite+APIs
>
> Regards,
> Igor
>

[gvvinblade]

Roman,

I see your point,

Actually Index should be a basic interface, a specific implementation may
be acquired by unwrap() method call (similar to the same method Cache
API[1]).

The specific implementation provides all additional methods.

1.
https://www.javadoc.io/static/javax.cache/cache-api/1.0.0/javax/cache/Cache.html#unwrap(java.lang.Class)

Regards,
Igor

сб, 23 мая 2020 г., 15:59 Roman Kondakov <[hidden email]>:

> Hi Igor,
>
> thank you for your effort!
>
> I have a question about the `Index.find(Args)` method. IEP says that it
> will return `a cursor over found rows`. But what if we don't need a row
> itself? For example in the case of possible optimizations of index scan
> that we can implement later. I can name a couple of them:
>
> - index-only scan [1]: we don't need a materialized row itself, we need
> only the indexed value from the index page.
> - multi-index scan (aka PG's bitmap scan [2]) where we can retrieve only
> row links from index(es), then perform union/intersection operations
> over them and retrieve from the RowStore only those rows that are
> actually needed.
>
> In this case we don't need a row itself, but a row's physical id (aka
> link) or the indexed value from the index page.
>
> What do you think? Should we have several `Index.find` methods with
> different return value? Or there might be another solution for such
> optimizations?
>
> Thank you.
>
> [1] https://www.postgresql.org/docs/10/indexes-index-only-scans.html
> [2] https://www.postgresql.org/message-id/12553.1135634231@...
>
> --
> Kind Regards
> Roman Kondakov
>
>
> On 22.05.2020 22:37, Igor Seliverstov wrote:
> > Igniters,
> >
> > Some of you knows I'm working on a new Apache Calcite based SQL query
> > execution engine now[1] and I succeeded a little in this [2].
> >
> > This activity becomes more important after H2 community decided to
> remove a
> > set of interfaces that we are using in our integration[3].
> >
> > In scope of my current activity I need to separate indexes related system
> > parts from H2, and reuse them in Calcite.
> >
> > I prepared an IEP[4] that is aimed to bring basic various index types
> > support to Ignite.
> >
> > In addition to solve my issue, It, I hope, will help to integrate full
> text
> > search indexes into Ignite better and provide a way to add new index
> types
> > support (like geospatial indexes).
> >
> > Also, as a next step, it allows to implement lightweight sorted bounded
> > scan queries, this feature, as far as I know, was requested several
> times.
> >
> > I'll appreciate it you, guys share your thoughts on that.
> >
> > 1.
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
> > 2. https://github.com/apache/ignite/pull/7071
> > 3. https://github.com/h2database/h2database/issues/2226
> > 4.
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-49%3A+Basic+index+infrastructure+as+a+part+of+core+Ignite+APIs
> >
> > Regards,
> > Igor
> >
>