Yo guys!
I think it is time to start implementing this feature since it became really relevant lately. Currently we have static indexing configuration via either CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata. Now we want to have IgniteCache API methods like createIndex/dropIndex which will allow to modify index layout at runtime without cluster restart. I think on the public API side we have to create index description classes like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally don't like approach with weirdly structured maps which we have in CacheTypeMetadata, but there it was done for simpler Spring XML support. From the implementation standpoint I think we have to utilize replicated transactional sys cache with continous queries to listen updates from it. Put index description objects there and handle these updates on all the interested nodes. On start each cache has to attempt load its initial index configuration transactionally or if it already exists, then utilize runtime configuration from cluster. At Indexing level we have to just find needed GridH2Table, take write lock on it, add index, fill it with data from primary index and release the write lock. Non-blocking approach we can implement further. Index drop is basically the same. Lets discuss API and implementation here and then I'll create jira issue with final description. By the way I am a bit busy to implement this stuff myself, so anyone interested please go ahead! Sergi |
Just interesting, will be some sort of SQL supported to create indexes for
example from Visor-console in interactive mode? On Mon, Jun 1, 2015 at 9:41 PM, Sergi Vladykin <[hidden email]> wrote: > Yo guys! > > I think it is time to start implementing this feature since it became > really relevant lately. > > Currently we have static indexing configuration via either > CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata. > Now we want to have IgniteCache API methods like createIndex/dropIndex > which will allow to modify index layout at runtime without cluster restart. > > I think on the public API side we have to create index description classes > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally don't > like approach with weirdly structured maps which we have in > CacheTypeMetadata, but there it was done for simpler Spring XML support. > > From the implementation standpoint I think we have to utilize replicated > transactional sys cache with continous queries to listen updates from it. > Put index description objects there and handle these updates on all the > interested nodes. > > On start each cache has to attempt load its initial index configuration > transactionally or if it already exists, then utilize runtime configuration > from cluster. > > At Indexing level we have to just find needed GridH2Table, take write lock > on it, add index, fill it with data from primary index and release the > write lock. Non-blocking approach we can implement further. > > Index drop is basically the same. > > Lets discuss API and implementation here and then I'll create jira issue > with final description. > > By the way I am a bit busy to implement this stuff myself, so anyone > interested please go ahead! > > Sergi > -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
In reply to this post by Sergi
I would do index creation/drop through custom messages in discovery. We
already use this to start and stop caches and start continuous processes. This approach is much easier from synchronization standpoint. Moreover, you guarantee that altering happens on single topology version. --Yakov 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>: > Yo guys! > > I think it is time to start implementing this feature since it became > really relevant lately. > > Currently we have static indexing configuration via either > CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata. > Now we want to have IgniteCache API methods like createIndex/dropIndex > which will allow to modify index layout at runtime without cluster restart. > > I think on the public API side we have to create index description classes > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally don't > like approach with weirdly structured maps which we have in > CacheTypeMetadata, but there it was done for simpler Spring XML support. > > From the implementation standpoint I think we have to utilize replicated > transactional sys cache with continous queries to listen updates from it. > Put index description objects there and handle these updates on all the > interested nodes. > > On start each cache has to attempt load its initial index configuration > transactionally or if it already exists, then utilize runtime configuration > from cluster. > > At Indexing level we have to just find needed GridH2Table, take write lock > on it, add index, fill it with data from primary index and release the > write lock. Non-blocking approach we can implement further. > > Index drop is basically the same. > > Lets discuss API and implementation here and then I'll create jira issue > with final description. > > By the way I am a bit busy to implement this stuff myself, so anyone > interested please go ahead! > > Sergi > |
In reply to this post by Alexey Kuznetsov-2
> Just interesting, will be some sort of SQL supported to create indexes for
> example from Visor-console in interactive mode? > If visor console starts node and provides whole cache API then yes, otherwise we have to add special command. Anyways I think we should do it in a separate step. Sergi > On Mon, Jun 1, 2015 at 9:41 PM, Sergi Vladykin <[hidden email]> > wrote: > > > Yo guys! > > > > I think it is time to start implementing this feature since it became > > really relevant lately. > > > > Currently we have static indexing configuration via either > > CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata. > > Now we want to have IgniteCache API methods like createIndex/dropIndex > > which will allow to modify index layout at runtime without cluster > restart. > > > > I think on the public API side we have to create index description > classes > > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally > don't > > like approach with weirdly structured maps which we have in > > CacheTypeMetadata, but there it was done for simpler Spring XML support. > > > > From the implementation standpoint I think we have to utilize replicated > > transactional sys cache with continous queries to listen updates from it. > > Put index description objects there and handle these updates on all the > > interested nodes. > > > > On start each cache has to attempt load its initial index configuration > > transactionally or if it already exists, then utilize runtime > configuration > > from cluster. > > > > At Indexing level we have to just find needed GridH2Table, take write > lock > > on it, add index, fill it with data from primary index and release the > > write lock. Non-blocking approach we can implement further. > > > > Index drop is basically the same. > > > > Lets discuss API and implementation here and then I'll create jira issue > > with final description. > > > > By the way I am a bit busy to implement this stuff myself, so anyone > > interested please go ahead! > > > > Sergi > > > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com > |
In reply to this post by yzhdanov
> I would do index creation/drop through custom messages in discovery. We
> already use this to start and stop caches and start continuous processes. > Could you please explain in more details how this works? > This approach is much easier from synchronization standpoint. I'm more familiar with cache transactions. What exactly is easier? Am I missing something hard in tx? > Moreover, you > guarantee that altering happens on single topology version. > I don't see why we need to care about it in this case. Sergi > --Yakov > > 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>: > > > Yo guys! > > > > I think it is time to start implementing this feature since it became > > really relevant lately. > > > > Currently we have static indexing configuration via either > > CacheConfiguration.setIndexedTypes or CacheConfiguration.setTypeMetadata. > > Now we want to have IgniteCache API methods like createIndex/dropIndex > > which will allow to modify index layout at runtime without cluster > restart. > > > > I think on the public API side we have to create index description > classes > > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally > don't > > like approach with weirdly structured maps which we have in > > CacheTypeMetadata, but there it was done for simpler Spring XML support. > > > > From the implementation standpoint I think we have to utilize replicated > > transactional sys cache with continous queries to listen updates from it. > > Put index description objects there and handle these updates on all the > > interested nodes. > > > > On start each cache has to attempt load its initial index configuration > > transactionally or if it already exists, then utilize runtime > configuration > > from cluster. > > > > At Indexing level we have to just find needed GridH2Table, take write > lock > > on it, add index, fill it with data from primary index and release the > > write lock. Non-blocking approach we can implement further. > > > > Index drop is basically the same. > > > > Lets discuss API and implementation here and then I'll create jira issue > > with final description. > > > > By the way I am a bit busy to implement this stuff myself, so anyone > > interested please go ahead! > > > > Sergi > > > |
I propose the following API:
IgniteCahe.createIndex(Index idx); IgniteCahe.dropIndex(String idxName); abstract class Index { Index(String idxName); } class SQLSortedIndex extends Index { void addField(String fieldName, boolean asc); } class SQLGeoSpatialIndex extends Index { SQLGeoSpatialIndex(String idxName, String fieldName) } class FulltextIndex extends Index { void addField(String fieldName); } Sergi 2015-06-01 19:13 GMT+03:00 Sergi Vladykin <[hidden email]>: > > I would do index creation/drop through custom messages in discovery. We >> already use this to start and stop caches and start continuous processes. >> > > Could you please explain in more details how this works? > > >> This approach is much easier from synchronization standpoint. > > > I'm more familiar with cache transactions. What exactly is easier? Am I > missing something hard in tx? > > >> Moreover, you >> guarantee that altering happens on single topology version. >> > > I don't see why we need to care about it in this case. > > Sergi > > > >> --Yakov >> >> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>: >> >> > Yo guys! >> > >> > I think it is time to start implementing this feature since it became >> > really relevant lately. >> > >> > Currently we have static indexing configuration via either >> > CacheConfiguration.setIndexedTypes or >> CacheConfiguration.setTypeMetadata. >> > Now we want to have IgniteCache API methods like createIndex/dropIndex >> > which will allow to modify index layout at runtime without cluster >> restart. >> > >> > I think on the public API side we have to create index description >> classes >> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally >> don't >> > like approach with weirdly structured maps which we have in >> > CacheTypeMetadata, but there it was done for simpler Spring XML support. >> > >> > From the implementation standpoint I think we have to utilize replicated >> > transactional sys cache with continous queries to listen updates from >> it. >> > Put index description objects there and handle these updates on all the >> > interested nodes. >> > >> > On start each cache has to attempt load its initial index configuration >> > transactionally or if it already exists, then utilize runtime >> configuration >> > from cluster. >> > >> > At Indexing level we have to just find needed GridH2Table, take write >> lock >> > on it, add index, fill it with data from primary index and release the >> > write lock. Non-blocking approach we can implement further. >> > >> > Index drop is basically the same. >> > >> > Lets discuss API and implementation here and then I'll create jira issue >> > with final description. >> > >> > By the way I am a bit busy to implement this stuff myself, so anyone >> > interested please go ahead! >> > >> > Sergi >> > >> > > |
Minor note.
If I correctly remember in CacheTypeMetadata we also define index field sort direction with boolean and "true" means "desc". Maybe SQLSortedIndex should be: class SQLSortedIndex extends Index { void addField(String fieldName, boolean __desc__); } ? On Thu, Jun 4, 2015 at 5:00 AM, Sergi Vladykin <[hidden email]> wrote: > I propose the following API: > > IgniteCahe.createIndex(Index idx); > IgniteCahe.dropIndex(String idxName); > > abstract class Index { > Index(String idxName); > } > > class SQLSortedIndex extends Index { > void addField(String fieldName, boolean asc); > } > > class SQLGeoSpatialIndex extends Index { > SQLGeoSpatialIndex(String idxName, String fieldName) > } > > class FulltextIndex extends Index { > void addField(String fieldName); > } > > Sergi > > > 2015-06-01 19:13 GMT+03:00 Sergi Vladykin <[hidden email]>: > > > > > I would do index creation/drop through custom messages in discovery. We > >> already use this to start and stop caches and start continuous > processes. > >> > > > > Could you please explain in more details how this works? > > > > > >> This approach is much easier from synchronization standpoint. > > > > > > I'm more familiar with cache transactions. What exactly is easier? Am I > > missing something hard in tx? > > > > > >> Moreover, you > >> guarantee that altering happens on single topology version. > >> > > > > I don't see why we need to care about it in this case. > > > > Sergi > > > > > > > >> --Yakov > >> > >> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>: > >> > >> > Yo guys! > >> > > >> > I think it is time to start implementing this feature since it became > >> > really relevant lately. > >> > > >> > Currently we have static indexing configuration via either > >> > CacheConfiguration.setIndexedTypes or > >> CacheConfiguration.setTypeMetadata. > >> > Now we want to have IgniteCache API methods like createIndex/dropIndex > >> > which will allow to modify index layout at runtime without cluster > >> restart. > >> > > >> > I think on the public API side we have to create index description > >> classes > >> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally > >> don't > >> > like approach with weirdly structured maps which we have in > >> > CacheTypeMetadata, but there it was done for simpler Spring XML > support. > >> > > >> > From the implementation standpoint I think we have to utilize > replicated > >> > transactional sys cache with continous queries to listen updates from > >> it. > >> > Put index description objects there and handle these updates on all > the > >> > interested nodes. > >> > > >> > On start each cache has to attempt load its initial index > configuration > >> > transactionally or if it already exists, then utilize runtime > >> configuration > >> > from cluster. > >> > > >> > At Indexing level we have to just find needed GridH2Table, take write > >> lock > >> > on it, add index, fill it with data from primary index and release the > >> > write lock. Non-blocking approach we can implement further. > >> > > >> > Index drop is basically the same. > >> > > >> > Lets discuss API and implementation here and then I'll create jira > issue > >> > with final description. > >> > > >> > By the way I am a bit busy to implement this stuff myself, so anyone > >> > interested please go ahead! > >> > > >> > Sergi > >> > > >> > > > > > -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
In reply to this post by Sergi
Sergi,
I see you have plain index and then sorted index. Do we even have indexes that are not sorted in Ignite? D. On Thu, Jun 4, 2015 at 1:00 AM, Sergi Vladykin <[hidden email]> wrote: > I propose the following API: > > IgniteCahe.createIndex(Index idx); > IgniteCahe.dropIndex(String idxName); > > abstract class Index { > Index(String idxName); > } > > class SQLSortedIndex extends Index { > void addField(String fieldName, boolean asc); > } > > class SQLGeoSpatialIndex extends Index { > SQLGeoSpatialIndex(String idxName, String fieldName) > } > > class FulltextIndex extends Index { > void addField(String fieldName); > } > > Sergi > > > 2015-06-01 19:13 GMT+03:00 Sergi Vladykin <[hidden email]>: > > > > > I would do index creation/drop through custom messages in discovery. We > >> already use this to start and stop caches and start continuous > processes. > >> > > > > Could you please explain in more details how this works? > > > > > >> This approach is much easier from synchronization standpoint. > > > > > > I'm more familiar with cache transactions. What exactly is easier? Am I > > missing something hard in tx? > > > > > >> Moreover, you > >> guarantee that altering happens on single topology version. > >> > > > > I don't see why we need to care about it in this case. > > > > Sergi > > > > > > > >> --Yakov > >> > >> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>: > >> > >> > Yo guys! > >> > > >> > I think it is time to start implementing this feature since it became > >> > really relevant lately. > >> > > >> > Currently we have static indexing configuration via either > >> > CacheConfiguration.setIndexedTypes or > >> CacheConfiguration.setTypeMetadata. > >> > Now we want to have IgniteCache API methods like createIndex/dropIndex > >> > which will allow to modify index layout at runtime without cluster > >> restart. > >> > > >> > I think on the public API side we have to create index description > >> classes > >> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally > >> don't > >> > like approach with weirdly structured maps which we have in > >> > CacheTypeMetadata, but there it was done for simpler Spring XML > support. > >> > > >> > From the implementation standpoint I think we have to utilize > replicated > >> > transactional sys cache with continous queries to listen updates from > >> it. > >> > Put index description objects there and handle these updates on all > the > >> > interested nodes. > >> > > >> > On start each cache has to attempt load its initial index > configuration > >> > transactionally or if it already exists, then utilize runtime > >> configuration > >> > from cluster. > >> > > >> > At Indexing level we have to just find needed GridH2Table, take write > >> lock > >> > on it, add index, fill it with data from primary index and release the > >> > write lock. Non-blocking approach we can implement further. > >> > > >> > Index drop is basically the same. > >> > > >> > Lets discuss API and implementation here and then I'll create jira > issue > >> > with final description. > >> > > >> > By the way I am a bit busy to implement this stuff myself, so anyone > >> > interested please go ahead! > >> > > >> > Sergi > >> > > >> > > > > > |
We don't have "plain index", Index is abstract super class for all the
indexes. Geospatial is an example of unsorted SQL index. In the future we can decide to implement SQL Hash indexes which will be unordered as well. You may think that suggested naming is a bit redundant, but I believe it makes sense here. Sergi 2015-06-05 1:16 GMT+03:00 Dmitriy Setrakyan <[hidden email]>: > Sergi, > > I see you have plain index and then sorted index. Do we even have indexes > that are not sorted in Ignite? > > D. > > On Thu, Jun 4, 2015 at 1:00 AM, Sergi Vladykin <[hidden email]> > wrote: > > > I propose the following API: > > > > IgniteCahe.createIndex(Index idx); > > IgniteCahe.dropIndex(String idxName); > > > > abstract class Index { > > Index(String idxName); > > } > > > > class SQLSortedIndex extends Index { > > void addField(String fieldName, boolean asc); > > } > > > > class SQLGeoSpatialIndex extends Index { > > SQLGeoSpatialIndex(String idxName, String fieldName) > > } > > > > class FulltextIndex extends Index { > > void addField(String fieldName); > > } > > > > Sergi > > > > > > 2015-06-01 19:13 GMT+03:00 Sergi Vladykin <[hidden email]>: > > > > > > > > I would do index creation/drop through custom messages in discovery. We > > >> already use this to start and stop caches and start continuous > > processes. > > >> > > > > > > Could you please explain in more details how this works? > > > > > > > > >> This approach is much easier from synchronization standpoint. > > > > > > > > > I'm more familiar with cache transactions. What exactly is easier? Am I > > > missing something hard in tx? > > > > > > > > >> Moreover, you > > >> guarantee that altering happens on single topology version. > > >> > > > > > > I don't see why we need to care about it in this case. > > > > > > Sergi > > > > > > > > > > > >> --Yakov > > >> > > >> 2015-06-01 17:41 GMT+03:00 Sergi Vladykin <[hidden email]>: > > >> > > >> > Yo guys! > > >> > > > >> > I think it is time to start implementing this feature since it > became > > >> > really relevant lately. > > >> > > > >> > Currently we have static indexing configuration via either > > >> > CacheConfiguration.setIndexedTypes or > > >> CacheConfiguration.setTypeMetadata. > > >> > Now we want to have IgniteCache API methods like > createIndex/dropIndex > > >> > which will allow to modify index layout at runtime without cluster > > >> restart. > > >> > > > >> > I think on the public API side we have to create index description > > >> classes > > >> > like SQLSortedIndex, FulltextIndex, SQLGeospatialIndex. I personally > > >> don't > > >> > like approach with weirdly structured maps which we have in > > >> > CacheTypeMetadata, but there it was done for simpler Spring XML > > support. > > >> > > > >> > From the implementation standpoint I think we have to utilize > > replicated > > >> > transactional sys cache with continous queries to listen updates > from > > >> it. > > >> > Put index description objects there and handle these updates on all > > the > > >> > interested nodes. > > >> > > > >> > On start each cache has to attempt load its initial index > > configuration > > >> > transactionally or if it already exists, then utilize runtime > > >> configuration > > >> > from cluster. > > >> > > > >> > At Indexing level we have to just find needed GridH2Table, take > write > > >> lock > > >> > on it, add index, fill it with data from primary index and release > the > > >> > write lock. Non-blocking approach we can implement further. > > >> > > > >> > Index drop is basically the same. > > >> > > > >> > Lets discuss API and implementation here and then I'll create jira > > issue > > >> > with final description. > > >> > > > >> > By the way I am a bit busy to implement this stuff myself, so anyone > > >> > interested please go ahead! > > >> > > > >> > Sergi > > >> > > > >> > > > > > > > > > |
Free forum by Nabble | Edit this page |