[GitHub] ignite pull request #3719: IGNITE-8048 merge query entities for dynamic cach...

[andrey-kuznetsov]

GitHub user akalash opened a pull request:

    https://github.com/apache/ignite/pull/3719

    IGNITE-8048 merge query entities for dynamic cache during joining node

   

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/gridgain/apache-ignite ignite-8048

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/ignite/pull/3719.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3719
   
----
commit f80023baa15e67e0ab42fafafa759667176cb32c
Author: Anton Kalashnikov <kaa.dev@...>
Date:   2018-03-29T14:56:51Z

    IGNITE-8048 merge query entities for dynamic cache during joining node

----


---

[andrey-kuznetsov]

Github user asfgit closed the pull request at:

    https://github.com/apache/ignite/pull/3719


---

[Manu]

Hi!

I have taken a look at this new functionality and it is very interesting.
But I have a couple of doubts about the performance.

For each modification of the same QueryEntitity, a new
SchemaAbstractOperation is generated, in the case of adding columns the
performance is not affected, but in the case of creating new indexes I think
there is a problem:

- If the modification in the QueryEntitity contains the creation of N
indexes, N SchemaIndexCreateOperation are generated for the same table, this
causes the underline cache to be scanned N times to populate each indice (by
SchemaIndexCacheVisitorClosure). For caches with a few data is not a
problem, but for caches with millions of records is not optimal.

A possible solution would be to create a new type of SchemaAbstractOperation
(for example SchemaUpgradeQueryEntityOperation with the new QueryEntity,
boolean forceRebuild, boolean forceMutateQueryEntity), to register on one
block this new QueryEntitity for:

- Create new indices at same time (populate all indices with a single scan)
(auto, like now)
- With on demand forceMutateQueryEntity (ignore QueryEntity patch checks),
ability to eliminate indices (and fully release resources) and columns. Over
time the QueryEntity will evolve and at some point it is possible that
certain columns and indices are not necessary any more... I think it would
be interesting to allow trim indexed data and eliminate unused columns.

Hope it helps!

Regards




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Vladimir Ozerov]

Hi,

What is the problem being addressed by this change? We do not allow for
multiple indexes creation at a time. The only possible way to create an
index is CREATE INDEX command when only one index is created.

On Tue, Apr 17, 2018 at 10:08 PM, Manu <[hidden email]> wrote:

> Hi!
>
> I have taken a look at this new functionality and it is very interesting.
> But I have a couple of doubts about the performance.
>
> For each modification of the same QueryEntitity, a new
> SchemaAbstractOperation is generated, in the case of adding columns the
> performance is not affected, but in the case of creating new indexes I
> think
> there is a problem:
>
> - If the modification in the QueryEntitity contains the creation of N
> indexes, N SchemaIndexCreateOperation are generated for the same table,
> this
> causes the underline cache to be scanned N times to populate each indice
> (by
> SchemaIndexCacheVisitorClosure). For caches with a few data is not a
> problem, but for caches with millions of records is not optimal.
>
> A possible solution would be to create a new type of
> SchemaAbstractOperation
> (for example SchemaUpgradeQueryEntityOperation with the new QueryEntity,
> boolean forceRebuild, boolean forceMutateQueryEntity), to register on one
> block this new QueryEntitity for:
>
> - Create new indices at same time (populate all indices with a single scan)
> (auto, like now)
> - With on demand forceMutateQueryEntity (ignore QueryEntity patch checks),
> ability to eliminate indices (and fully release resources) and columns.
> Over
> time the QueryEntity will evolve and at some point it is possible that
> certain columns and indices are not necessary any more... I think it would
> be interesting to allow trim indexed data and eliminate unused columns.
>
> Hope it helps!
>
> Regards
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[Manu]

Hi,

Sorry, I assumed that QueryEntities merge process was always applied (on
active or inactive cluster) and that it was responsible for auto creating
and populate new indexes and columns on the fly... my fault.

Then, in addition to updating CacheConfiguration and QuerySchema, does it do
something else?

One more question, when the cluster is re-activated, will new columns and
new indexes be created and populated?

Thanks!



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Vladimir Ozerov]

Hi Manu,

Activation/deactivation process does not trigger creation of new columns or
indexes. They are only created either on cache create (dynamic or on node
startup), or during DDL command execution.

Vladimir.

On Wed, Apr 18, 2018 at 6:57 PM, Manu <[hidden email]> wrote:

> Hi,
>
> Sorry, I assumed that QueryEntities merge process was always applied (on
> active or inactive cluster) and that it was responsible for auto creating
> and populate new indexes and columns on the fly... my fault.
>
> Then, in addition to updating CacheConfiguration and QuerySchema, does it
> do
> something else?
>
> One more question, when the cluster is re-activated, will new columns and
> new indexes be created and populated?
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>