Dynamic SQL/Fulltext index create/drop.

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Dynamic SQL/Fulltext index create/drop.

Sergi
Yo guys!

I think it is time to start implementing this feature since it became
really relevant lately.

Currently we have static indexing configuration via either
CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata.
Now we want to have IgniteCache API methods like createIndex/dropIndex
which will allow to modify index layout at runtime without cluster restart.

I think on the public API side we have to create index description classes
like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally don't
like approach with weirdly structured maps which we have in
CacheTypeMetadata, but there it was done for simpler Spring XML support.

From the implementation standpoint I think we have to utilize replicated
transactional sys cache with continous queries to listen updates from it.
Put index description objects there and handle these updates on all the
interested nodes.

On start each cache has to attempt load its initial index configuration
transactionally or if it already exists, then utilize runtime configuration
from cluster.

At Indexing level we have to just find needed GridH2Table, take write lock
on it, add index, fill it with data from primary index and release the
write lock. Non-blocking approach we can implement further.

Index drop is basically the same.

Lets discuss API and implementation here and then I'll create jira issue
with final description.

By the way I am a bit busy to implement this stuff myself, so anyone
interested please go ahead!

Sergi
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic SQL/Fulltext index create/drop.

Alexey Kuznetsov-2
Just interesting, will be some sort of SQL supported to create indexes for
example from Visor-console in interactive mode?

On Mon, Jun 1, 2015 at 9:41 PM, Sergi Vladykin <[hidden email]>
wrote:

> Yo guys!
>
> I think it is time to start implementing this feature since it became
> really relevant lately.
>
> Currently we have static indexing configuration via either
> CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata.
> Now we want to have IgniteCache API methods like createIndex/dropIndex
> which will allow to modify index layout at runtime without cluster restart.
>
> I think on the public API side we have to create index description classes
> like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally don't
> like approach with weirdly structured maps which we have in
> CacheTypeMetadata, but there it was done for simpler Spring XML support.
>
> From the implementation standpoint I think we have to utilize replicated
> transactional sys cache with continous queries to listen updates from it.
> Put index description objects there and handle these updates on all the
> interested nodes.
>
> On start each cache has to attempt load its initial index configuration
> transactionally or if it already exists, then utilize runtime configuration
> from cluster.
>
> At Indexing level we have to just find needed GridH2Table, take write lock
> on it, add index, fill it with data from primary index and release the
> write lock. Non-blocking approach we can implement further.
>
> Index drop is basically the same.
>
> Lets discuss API and implementation here and then I'll create jira issue
> with final description.
>
> By the way I am a bit busy to implement this stuff myself, so anyone
> interested please go ahead!
>
> Sergi
>



--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic SQL/Fulltext index create/drop.

yzhdanov
In reply to this post by Sergi
I would do index creation/drop through custom messages in discovery. We
already use this to start and stop caches and start continuous processes.
This approach is much easier from synchronization standpoint. Moreover, you
guarantee that altering happens on single topology version.

--Yakov

2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>:

> Yo guys!
>
> I think it is time to start implementing this feature since it became
> really relevant lately.
>
> Currently we have static indexing configuration via either
> CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata.
> Now we want to have IgniteCache API methods like createIndex/dropIndex
> which will allow to modify index layout at runtime without cluster restart.
>
> I think on the public API side we have to create index description classes
> like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally don't
> like approach with weirdly structured maps which we have in
> CacheTypeMetadata, but there it was done for simpler Spring XML support.
>
> From the implementation standpoint I think we have to utilize replicated
> transactional sys cache with continous queries to listen updates from it.
> Put index description objects there and handle these updates on all the
> interested nodes.
>
> On start each cache has to attempt load its initial index configuration
> transactionally or if it already exists, then utilize runtime configuration
> from cluster.
>
> At Indexing level we have to just find needed GridH2Table, take write lock
> on it, add index, fill it with data from primary index and release the
> write lock. Non-blocking approach we can implement further.
>
> Index drop is basically the same.
>
> Lets discuss API and implementation here and then I'll create jira issue
> with final description.
>
> By the way I am a bit busy to implement this stuff myself, so anyone
> interested please go ahead!
>
> Sergi
>
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic SQL/Fulltext index create/drop.

Sergi
In reply to this post by Alexey Kuznetsov-2
> Just interesting, will be some sort of SQL supported to create indexes for
> example from Visor-console in interactive mode?
>

If visor console starts node and provides whole cache API then yes,
otherwise we have to add special command.
Anyways I think we should do it in a separate step.

Sergi



> On Mon, Jun 1, 2015 at 9:41 PM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > Yo guys!
> >
> > I think it is time to start implementing this feature since it became
> > really relevant lately.
> >
> > Currently we have static indexing configuration via either
> > CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata.
> > Now we want to have IgniteCache API methods like createIndex/dropIndex
> > which will allow to modify index layout at runtime without cluster
> restart.
> >
> > I think on the public API side we have to create index description
> classes
> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally
> don't
> > like approach with weirdly structured maps which we have in
> > CacheTypeMetadata, but there it was done for simpler Spring XML support.
> >
> > From the implementation standpoint I think we have to utilize replicated
> > transactional sys cache with continous queries to listen updates from it.
> > Put index description objects there and handle these updates on all the
> > interested nodes.
> >
> > On start each cache has to attempt load its initial index configuration
> > transactionally or if it already exists, then utilize runtime
> configuration
> > from cluster.
> >
> > At Indexing level we have to just find needed GridH2Table, take write
> lock
> > on it, add index, fill it with data from primary index and release the
> > write lock. Non-blocking approach we can implement further.
> >
> > Index drop is basically the same.
> >
> > Lets discuss API and implementation here and then I'll create jira issue
> > with final description.
> >
> > By the way I am a bit busy to implement this stuff myself, so anyone
> > interested please go ahead!
> >
> > Sergi
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic SQL/Fulltext index create/drop.

Sergi
In reply to this post by yzhdanov
> I would do index creation/drop through custom messages in discovery. We
> already use this to start and stop caches and start continuous processes.
>

Could you please explain in more details how this works?


> This approach is much easier from synchronization standpoint.


I'm more familiar with cache transactions. What exactly is easier? Am I
missing something hard in tx?


> Moreover, you
> guarantee that altering happens on single topology version.
>

I don't see why we need to care about it in this case.

Sergi



> --Yakov
>
> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>:
>
> > Yo guys!
> >
> > I think it is time to start implementing this feature since it became
> > really relevant lately.
> >
> > Currently we have static indexing configuration via either
> > CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata.
> > Now we want to have IgniteCache API methods like createIndex/dropIndex
> > which will allow to modify index layout at runtime without cluster
> restart.
> >
> > I think on the public API side we have to create index description
> classes
> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally
> don't
> > like approach with weirdly structured maps which we have in
> > CacheTypeMetadata, but there it was done for simpler Spring XML support.
> >
> > From the implementation standpoint I think we have to utilize replicated
> > transactional sys cache with continous queries to listen updates from it.
> > Put index description objects there and handle these updates on all the
> > interested nodes.
> >
> > On start each cache has to attempt load its initial index configuration
> > transactionally or if it already exists, then utilize runtime
> configuration
> > from cluster.
> >
> > At Indexing level we have to just find needed GridH2Table, take write
> lock
> > on it, add index, fill it with data from primary index and release the
> > write lock. Non-blocking approach we can implement further.
> >
> > Index drop is basically the same.
> >
> > Lets discuss API and implementation here and then I'll create jira issue
> > with final description.
> >
> > By the way I am a bit busy to implement this stuff myself, so anyone
> > interested please go ahead!
> >
> > Sergi
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic SQL/Fulltext index create/drop.

Sergi
I propose the following API:

IgniteCahe.createIndex(Index idx);
IgniteCahe.dropIndex(String idxName);

abstract class Index {
     Index(String idxName);
}

class SQLSortedIndex extends Index {
    void addField(String fieldName, boolean asc);
}

class SQLGeoSpatialIndex extends Index {
     SQLGeoSpatialIndex(String idxName, String fieldName)
}

class FulltextIndex extends Index {
    void addField(String fieldName);
}

Sergi


2015-06-01 19:13 GMT+03:00 Sergi Vladykin <[hidden email]>:

>
> I would do index creation/drop through custom messages in discovery. We
>> already use this to start and stop caches and start continuous processes.
>>
>
> Could you please explain in more details how this works?
>
>
>> This approach is much easier from synchronization standpoint.
>
>
> I'm more familiar with cache transactions. What exactly is easier? Am I
> missing something hard in tx?
>
>
>> Moreover, you
>> guarantee that altering happens on single topology version.
>>
>
> I don't see why we need to care about it in this case.
>
> Sergi
>
>
>
>> --Yakov
>>
>> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>:
>>
>> > Yo guys!
>> >
>> > I think it is time to start implementing this feature since it became
>> > really relevant lately.
>> >
>> > Currently we have static indexing configuration via either
>> > CacheConfiguration.setIndexedTypes or
>> CacheConfiguration.setTypeMetadata.
>> > Now we want to have IgniteCache API methods like createIndex/dropIndex
>> > which will allow to modify index layout at runtime without cluster
>> restart.
>> >
>> > I think on the public API side we have to create index description
>> classes
>> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally
>> don't
>> > like approach with weirdly structured maps which we have in
>> > CacheTypeMetadata, but there it was done for simpler Spring XML support.
>> >
>> > From the implementation standpoint I think we have to utilize replicated
>> > transactional sys cache with continous queries to listen updates from
>> it.
>> > Put index description objects there and handle these updates on all the
>> > interested nodes.
>> >
>> > On start each cache has to attempt load its initial index configuration
>> > transactionally or if it already exists, then utilize runtime
>> configuration
>> > from cluster.
>> >
>> > At Indexing level we have to just find needed GridH2Table, take write
>> lock
>> > on it, add index, fill it with data from primary index and release the
>> > write lock. Non-blocking approach we can implement further.
>> >
>> > Index drop is basically the same.
>> >
>> > Lets discuss API and implementation here and then I'll create jira issue
>> > with final description.
>> >
>> > By the way I am a bit busy to implement this stuff myself, so anyone
>> > interested please go ahead!
>> >
>> > Sergi
>> >
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic SQL/Fulltext index create/drop.

Alexey Kuznetsov-2
Minor note.
If I correctly remember in CacheTypeMetadata we also define index field
sort direction with boolean and "true" means "desc".

Maybe  SQLSortedIndex  should be:

class SQLSortedIndex extends Index {
    void addField(String fieldName, boolean __desc__);
}

?

On Thu, Jun 4, 2015 at 5:00 AM, Sergi Vladykin <[hidden email]>
wrote:

> I propose the following API:
>
> IgniteCahe.createIndex(Index idx);
> IgniteCahe.dropIndex(String idxName);
>
> abstract class Index {
>      Index(String idxName);
> }
>
> class SQLSortedIndex extends Index {
>     void addField(String fieldName, boolean asc);
> }
>
> class SQLGeoSpatialIndex extends Index {
>      SQLGeoSpatialIndex(String idxName, String fieldName)
> }
>
> class FulltextIndex extends Index {
>     void addField(String fieldName);
> }
>
> Sergi
>
>
> 2015-06-01 19:13 GMT+03:00 Sergi Vladykin <[hidden email]>:
>
> >
> > I would do index creation/drop through custom messages in discovery. We
> >> already use this to start and stop caches and start continuous
> processes.
> >>
> >
> > Could you please explain in more details how this works?
> >
> >
> >> This approach is much easier from synchronization standpoint.
> >
> >
> > I'm more familiar with cache transactions. What exactly is easier? Am I
> > missing something hard in tx?
> >
> >
> >> Moreover, you
> >> guarantee that altering happens on single topology version.
> >>
> >
> > I don't see why we need to care about it in this case.
> >
> > Sergi
> >
> >
> >
> >> --Yakov
> >>
> >> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>:
> >>
> >> > Yo guys!
> >> >
> >> > I think it is time to start implementing this feature since it became
> >> > really relevant lately.
> >> >
> >> > Currently we have static indexing configuration via either
> >> > CacheConfiguration.setIndexedTypes or
> >> CacheConfiguration.setTypeMetadata.
> >> > Now we want to have IgniteCache API methods like createIndex/dropIndex
> >> > which will allow to modify index layout at runtime without cluster
> >> restart.
> >> >
> >> > I think on the public API side we have to create index description
> >> classes
> >> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally
> >> don't
> >> > like approach with weirdly structured maps which we have in
> >> > CacheTypeMetadata, but there it was done for simpler Spring XML
> support.
> >> >
> >> > From the implementation standpoint I think we have to utilize
> replicated
> >> > transactional sys cache with continous queries to listen updates from
> >> it.
> >> > Put index description objects there and handle these updates on all
> the
> >> > interested nodes.
> >> >
> >> > On start each cache has to attempt load its initial index
> configuration
> >> > transactionally or if it already exists, then utilize runtime
> >> configuration
> >> > from cluster.
> >> >
> >> > At Indexing level we have to just find needed GridH2Table, take write
> >> lock
> >> > on it, add index, fill it with data from primary index and release the
> >> > write lock. Non-blocking approach we can implement further.
> >> >
> >> > Index drop is basically the same.
> >> >
> >> > Lets discuss API and implementation here and then I'll create jira
> issue
> >> > with final description.
> >> >
> >> > By the way I am a bit busy to implement this stuff myself, so anyone
> >> > interested please go ahead!
> >> >
> >> > Sergi
> >> >
> >>
> >
> >
>



--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic SQL/Fulltext index create/drop.

dsetrakyan
In reply to this post by Sergi
Sergi,

I see you have plain index and then sorted index. Do we even have indexes
that are not sorted in Ignite?

D.

On Thu, Jun 4, 2015 at 1:00 AM, Sergi Vladykin <[hidden email]>
wrote:

> I propose the following API:
>
> IgniteCahe.createIndex(Index idx);
> IgniteCahe.dropIndex(String idxName);
>
> abstract class Index {
>      Index(String idxName);
> }
>
> class SQLSortedIndex extends Index {
>     void addField(String fieldName, boolean asc);
> }
>
> class SQLGeoSpatialIndex extends Index {
>      SQLGeoSpatialIndex(String idxName, String fieldName)
> }
>
> class FulltextIndex extends Index {
>     void addField(String fieldName);
> }
>
> Sergi
>
>
> 2015-06-01 19:13 GMT+03:00 Sergi Vladykin <[hidden email]>:
>
> >
> > I would do index creation/drop through custom messages in discovery. We
> >> already use this to start and stop caches and start continuous
> processes.
> >>
> >
> > Could you please explain in more details how this works?
> >
> >
> >> This approach is much easier from synchronization standpoint.
> >
> >
> > I'm more familiar with cache transactions. What exactly is easier? Am I
> > missing something hard in tx?
> >
> >
> >> Moreover, you
> >> guarantee that altering happens on single topology version.
> >>
> >
> > I don't see why we need to care about it in this case.
> >
> > Sergi
> >
> >
> >
> >> --Yakov
> >>
> >> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>:
> >>
> >> > Yo guys!
> >> >
> >> > I think it is time to start implementing this feature since it became
> >> > really relevant lately.
> >> >
> >> > Currently we have static indexing configuration via either
> >> > CacheConfiguration.setIndexedTypes or
> >> CacheConfiguration.setTypeMetadata.
> >> > Now we want to have IgniteCache API methods like createIndex/dropIndex
> >> > which will allow to modify index layout at runtime without cluster
> >> restart.
> >> >
> >> > I think on the public API side we have to create index description
> >> classes
> >> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally
> >> don't
> >> > like approach with weirdly structured maps which we have in
> >> > CacheTypeMetadata, but there it was done for simpler Spring XML
> support.
> >> >
> >> > From the implementation standpoint I think we have to utilize
> replicated
> >> > transactional sys cache with continous queries to listen updates from
> >> it.
> >> > Put index description objects there and handle these updates on all
> the
> >> > interested nodes.
> >> >
> >> > On start each cache has to attempt load its initial index
> configuration
> >> > transactionally or if it already exists, then utilize runtime
> >> configuration
> >> > from cluster.
> >> >
> >> > At Indexing level we have to just find needed GridH2Table, take write
> >> lock
> >> > on it, add index, fill it with data from primary index and release the
> >> > write lock. Non-blocking approach we can implement further.
> >> >
> >> > Index drop is basically the same.
> >> >
> >> > Lets discuss API and implementation here and then I'll create jira
> issue
> >> > with final description.
> >> >
> >> > By the way I am a bit busy to implement this stuff myself, so anyone
> >> > interested please go ahead!
> >> >
> >> > Sergi
> >> >
> >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic SQL/Fulltext index create/drop.

Sergi
We don't have "plain index", Index is abstract super class for all the
indexes. Geospatial is an example of unsorted SQL index. In the future we
can decide to implement SQL Hash indexes which will be unordered as well.
You may think that suggested naming is a bit redundant, but I believe it
makes sense here.

Sergi

2015-06-05 1:16 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Sergi,
>
> I see you have plain index and then sorted index. Do we even have indexes
> that are not sorted in Ignite?
>
> D.
>
> On Thu, Jun 4, 2015 at 1:00 AM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > I propose the following API:
> >
> > IgniteCahe.createIndex(Index idx);
> > IgniteCahe.dropIndex(String idxName);
> >
> > abstract class Index {
> >      Index(String idxName);
> > }
> >
> > class SQLSortedIndex extends Index {
> >     void addField(String fieldName, boolean asc);
> > }
> >
> > class SQLGeoSpatialIndex extends Index {
> >      SQLGeoSpatialIndex(String idxName, String fieldName)
> > }
> >
> > class FulltextIndex extends Index {
> >     void addField(String fieldName);
> > }
> >
> > Sergi
> >
> >
> > 2015-06-01 19:13 GMT+03:00 Sergi Vladykin <[hidden email]>:
> >
> > >
> > > I would do index creation/drop through custom messages in discovery. We
> > >> already use this to start and stop caches and start continuous
> > processes.
> > >>
> > >
> > > Could you please explain in more details how this works?
> > >
> > >
> > >> This approach is much easier from synchronization standpoint.
> > >
> > >
> > > I'm more familiar with cache transactions. What exactly is easier? Am I
> > > missing something hard in tx?
> > >
> > >
> > >> Moreover, you
> > >> guarantee that altering happens on single topology version.
> > >>
> > >
> > > I don't see why we need to care about it in this case.
> > >
> > > Sergi
> > >
> > >
> > >
> > >> --Yakov
> > >>
> > >> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>:
> > >>
> > >> > Yo guys!
> > >> >
> > >> > I think it is time to start implementing this feature since it
> became
> > >> > really relevant lately.
> > >> >
> > >> > Currently we have static indexing configuration via either
> > >> > CacheConfiguration.setIndexedTypes or
> > >> CacheConfiguration.setTypeMetadata.
> > >> > Now we want to have IgniteCache API methods like
> createIndex/dropIndex
> > >> > which will allow to modify index layout at runtime without cluster
> > >> restart.
> > >> >
> > >> > I think on the public API side we have to create index description
> > >> classes
> > >> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally
> > >> don't
> > >> > like approach with weirdly structured maps which we have in
> > >> > CacheTypeMetadata, but there it was done for simpler Spring XML
> > support.
> > >> >
> > >> > From the implementation standpoint I think we have to utilize
> > replicated
> > >> > transactional sys cache with continous queries to listen updates
> from
> > >> it.
> > >> > Put index description objects there and handle these updates on all
> > the
> > >> > interested nodes.
> > >> >
> > >> > On start each cache has to attempt load its initial index
> > configuration
> > >> > transactionally or if it already exists, then utilize runtime
> > >> configuration
> > >> > from cluster.
> > >> >
> > >> > At Indexing level we have to just find needed GridH2Table, take
> write
> > >> lock
> > >> > on it, add index, fill it with data from primary index and release
> the
> > >> > write lock. Non-blocking approach we can implement further.
> > >> >
> > >> > Index drop is basically the same.
> > >> >
> > >> > Lets discuss API and implementation here and then I'll create jira
> > issue
> > >> > with final description.
> > >> >
> > >> > By the way I am a bit busy to implement this stuff myself, so anyone
> > >> > interested please go ahead!
> > >> >
> > >> > Sergi
> > >> >
> > >>
> > >
> > >
> >
>