Continuous Queries with several remote filter on the same cache

[Roman.Koriakov]

Hi Community,

I ask this question here because I haven't found the answer in the documentation.

Could you please clarify how Continuous Queries work? What the behavior of Continuous Queries if we have several clients with different Remote Filters on the same cache? For example, if we have: one server node with cache and we have up to 20 client nodes each of them will execute Continuous Query on the same cache but with different Remote Filters. Will each client get the data according to its remote filter? Or it is supposed to have only one Remote Filter for all clients and every client should filter data in its local event listener?
I would be grateful if you send some link which describes the behavior of Continuous Queries more thoroughly.
Best regards,
Roman

[dmagda]

Hi Roman,

Every continuous query is a unique entity that is processed by servers
independently. With your example, the server node will execute all 20
filters for every cache insert/update operation. The server will notify
through local listeners only those clients whose remote filters returned
'true'.

-
Denis


On Thu, Jun 4, 2020 at 8:44 PM <[hidden email]> wrote:

> Hi Community,
>
> I ask this question here because I haven't found the answer in the
> documentation.
>
> Could you please clarify how Continuous Queries work? What the behavior of
> Continuous Queries if we have several clients with different Remote Filters
> on the same cache? For example, if we have: one server node with cache and
> we have up to 20 client nodes each of them will execute Continuous Query on
> the same cache but with different Remote Filters. Will each client get the
> data according to its remote filter? Or it is supposed to have only one
> Remote Filter for all clients and every client should filter data in its
> local event listener?
> I would be grateful if you send some link which describes the behavior of
> Continuous Queries more thoroughly.
> Best regards,
> Roman
>

[Roman.Koriakov]

Hi Denis,
A big thank you for the answer.
Could you please tell me where can I find this logic in the sources. Which package should I look into?

-----Original Message-----
From: Denis Magda <[hidden email]>
Sent: Saturday, June 6, 2020 2:07 AM
To: dev <[hidden email]>
Subject: Re: Continuous Queries with several remote filter on the same cache

Hi Roman,

Every continuous query is a unique entity that is processed by servers independently. With your example, the server node will execute all 20 filters for every cache insert/update operation. The server will notify through local listeners only those clients whose remote filters returned 'true'.

-
Denis


On Thu, Jun 4, 2020 at 8:44 PM <[hidden email]> wrote:

> Hi Community,
>
> I ask this question here because I haven't found the answer in the
> documentation.
>
> Could you please clarify how Continuous Queries work? What the
> behavior of Continuous Queries if we have several clients with
> different Remote Filters on the same cache? For example, if we have:
> one server node with cache and we have up to 20 client nodes each of
> them will execute Continuous Query on the same cache but with
> different Remote Filters. Will each client get the data according to
> its remote filter? Or it is supposed to have only one Remote Filter
> for all clients and every client should filter data in its local event listener?
> I would be grateful if you send some link which describes the behavior
> of Continuous Queries more thoroughly.
> Best regards,
> Roman
>

[dmagda]

Roman,

Please check the following methods:
* CacheContiniousQueryHandler (the filter usage):
https://github.com/apache/ignite/blob/6955ac291352dd67c1f84a006cda512ee54f38bb/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryHandler.java#L994
* CacheContinuousQueryManager (the listener execution):
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryManager.java#L376

-
Denis


On Sun, Jun 7, 2020 at 12:19 AM <[hidden email]> wrote:

> Hi Denis,
> A big thank you for the answer.
> Could you please tell me where can I find this logic in the sources. Which
> package should I look into?
>
> -----Original Message-----
> From: Denis Magda <[hidden email]>
> Sent: Saturday, June 6, 2020 2:07 AM
> To: dev <[hidden email]>
> Subject: Re: Continuous Queries with several remote filter on the same
> cache
>
> Hi Roman,
>
> Every continuous query is a unique entity that is processed by servers
> independently. With your example, the server node will execute all 20
> filters for every cache insert/update operation. The server will notify
> through local listeners only those clients whose remote filters returned
> 'true'.
>
> -
> Denis
>
>
> On Thu, Jun 4, 2020 at 8:44 PM <[hidden email]> wrote:
>
> > Hi Community,
> >
> > I ask this question here because I haven't found the answer in the
> > documentation.
> >
> > Could you please clarify how Continuous Queries work? What the
> > behavior of Continuous Queries if we have several clients with
> > different Remote Filters on the same cache? For example, if we have:
> > one server node with cache and we have up to 20 client nodes each of
> > them will execute Continuous Query on the same cache but with
> > different Remote Filters. Will each client get the data according to
> > its remote filter? Or it is supposed to have only one Remote Filter
> > for all clients and every client should filter data in its local event
> listener?
> > I would be grateful if you send some link which describes the behavior
> > of Continuous Queries more thoroughly.
> > Best regards,
> > Roman
> >
>

[Roman.Koriakov]

Hi Denis,

Thanks! Is there some guarantee about the order of the updates? Even when we have multiple cache nodes.

Best regards,
Roman

-----Original Message-----
From: Denis Magda <[hidden email]>
Sent: Monday, June 8, 2020 10:20 PM
To: dev <[hidden email]>
Subject: Re: Continuous Queries with several remote filter on the same cache

Roman,

Please check the following methods:
* CacheContiniousQueryHandler (the filter usage):
https://github.com/apache/ignite/blob/6955ac291352dd67c1f84a006cda512ee54f38bb/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryHandler.java#L994
* CacheContinuousQueryManager (the listener execution):
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryManager.java#L376

-
Denis


On Sun, Jun 7, 2020 at 12:19 AM <[hidden email]> wrote:

> Hi Denis,
> A big thank you for the answer.
> Could you please tell me where can I find this logic in the sources.
> Which package should I look into?
>
> -----Original Message-----
> From: Denis Magda <[hidden email]>
> Sent: Saturday, June 6, 2020 2:07 AM
> To: dev <[hidden email]>
> Subject: Re: Continuous Queries with several remote filter on the same
> cache
>
> Hi Roman,
>
> Every continuous query is a unique entity that is processed by servers
> independently. With your example, the server node will execute all 20
> filters for every cache insert/update operation. The server will
> notify through local listeners only those clients whose remote filters
> returned 'true'.
>
> -
> Denis
>
>
> On Thu, Jun 4, 2020 at 8:44 PM <[hidden email]> wrote:
>
> > Hi Community,
> >
> > I ask this question here because I haven't found the answer in the
> > documentation.
> >
> > Could you please clarify how Continuous Queries work? What the
> > behavior of Continuous Queries if we have several clients with
> > different Remote Filters on the same cache? For example, if we have:
> > one server node with cache and we have up to 20 client nodes each of
> > them will execute Continuous Query on the same cache but with
> > different Remote Filters. Will each client get the data according to
> > its remote filter? Or it is supposed to have only one Remote Filter
> > for all clients and every client should filter data in its local
> > event
> listener?
> > I would be grateful if you send some link which describes the
> > behavior of Continuous Queries more thoroughly.
> > Best regards,
> > Roman
> >
>

[dmagda]

Roman,

The updates are ordered per partition. Let's take this example of an
application updating several records:

put (k1, val1) => mapped to partition_10 => node_A
put (k2, val2) => mapped to partition_5 => node_B
put (k3, val3) => mapped to partition_10 => node_A

It's guaranteed that a continuous query listener will be notified about k1
and k3 updates in this order - k1 first and k3 after. As for the k2 update,
it can arrive at any time (i.e., before k1, after k3 or in the middle).




-
Denis


On Fri, Jun 26, 2020 at 12:58 AM <[hidden email]> wrote:

> Hi Denis,
>
> Thanks! Is there some guarantee about the order of the updates? Even when
> we have multiple cache nodes.
>
> Best regards,
> Roman
>
> -----Original Message-----
> From: Denis Magda <[hidden email]>
> Sent: Monday, June 8, 2020 10:20 PM
> To: dev <[hidden email]>
> Subject: Re: Continuous Queries with several remote filter on the same
> cache
>
> Roman,
>
> Please check the following methods:
> * CacheContiniousQueryHandler (the filter usage):
>
> https://github.com/apache/ignite/blob/6955ac291352dd67c1f84a006cda512ee54f38bb/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryHandler.java#L994
> * CacheContinuousQueryManager (the listener execution):
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryManager.java#L376
>
> -
> Denis
>
>
> On Sun, Jun 7, 2020 at 12:19 AM <[hidden email]> wrote:
>
> > Hi Denis,
> > A big thank you for the answer.
> > Could you please tell me where can I find this logic in the sources.
> > Which package should I look into?
> >
> > -----Original Message-----
> > From: Denis Magda <[hidden email]>
> > Sent: Saturday, June 6, 2020 2:07 AM
> > To: dev <[hidden email]>
> > Subject: Re: Continuous Queries with several remote filter on the same
> > cache
> >
> > Hi Roman,
> >
> > Every continuous query is a unique entity that is processed by servers
> > independently. With your example, the server node will execute all 20
> > filters for every cache insert/update operation. The server will
> > notify through local listeners only those clients whose remote filters
> > returned 'true'.
> >
> > -
> > Denis
> >
> >
> > On Thu, Jun 4, 2020 at 8:44 PM <[hidden email]> wrote:
> >
> > > Hi Community,
> > >
> > > I ask this question here because I haven't found the answer in the
> > > documentation.
> > >
> > > Could you please clarify how Continuous Queries work? What the
> > > behavior of Continuous Queries if we have several clients with
> > > different Remote Filters on the same cache? For example, if we have:
> > > one server node with cache and we have up to 20 client nodes each of
> > > them will execute Continuous Query on the same cache but with
> > > different Remote Filters. Will each client get the data according to
> > > its remote filter? Or it is supposed to have only one Remote Filter
> > > for all clients and every client should filter data in its local
> > > event
> > listener?
> > > I would be grateful if you send some link which describes the
> > > behavior of Continuous Queries more thoroughly.
> > > Best regards,
> > > Roman
> > >
> >
>

[Roman.Koriakov]

Hi Denis,

What do you think about some improvements in @IgniteAsyncCallback regarding usability? What if we add a number of threads parameter in @IgniteAsyncCallback?

Best regards,
Roman

-----Original Message-----
From: Denis Magda <[hidden email]>
Sent: Friday, June 26, 2020 7:38 PM
To: dev <[hidden email]>
Subject: Re: Continuous Queries with several remote filter on the same cache

Roman,

The updates are ordered per partition. Let's take this example of an application updating several records:

put (k1, val1) => mapped to partition_10 => node_A put (k2, val2) => mapped to partition_5 => node_B put (k3, val3) => mapped to partition_10 => node_A

It's guaranteed that a continuous query listener will be notified about k1 and k3 updates in this order - k1 first and k3 after. As for the k2 update, it can arrive at any time (i.e., before k1, after k3 or in the middle).




-
Denis


On Fri, Jun 26, 2020 at 12:58 AM <[hidden email]> wrote:

> Hi Denis,
>
> Thanks! Is there some guarantee about the order of the updates? Even
> when we have multiple cache nodes.
>
> Best regards,
> Roman
>
> -----Original Message-----
> From: Denis Magda <[hidden email]>
> Sent: Monday, June 8, 2020 10:20 PM
> To: dev <[hidden email]>
> Subject: Re: Continuous Queries with several remote filter on the same
> cache
>
> Roman,
>
> Please check the following methods:
> * CacheContiniousQueryHandler (the filter usage):
>
> https://github.com/apache/ignite/blob/6955ac291352dd67c1f84a006cda512e
> e54f38bb/modules/core/src/main/java/org/apache/ignite/internal/process
> ors/cache/query/continuous/CacheContinuousQueryHandler.java#L994
> * CacheContinuousQueryManager (the listener execution):
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/jav
> a/org/apache/ignite/internal/processors/cache/query/continuous/CacheCo
> ntinuousQueryManager.java#L376
>
> -
> Denis
>
>
> On Sun, Jun 7, 2020 at 12:19 AM <[hidden email]> wrote:
>
> > Hi Denis,
> > A big thank you for the answer.
> > Could you please tell me where can I find this logic in the sources.
> > Which package should I look into?
> >
> > -----Original Message-----
> > From: Denis Magda <[hidden email]>
> > Sent: Saturday, June 6, 2020 2:07 AM
> > To: dev <[hidden email]>
> > Subject: Re: Continuous Queries with several remote filter on the
> > same cache
> >
> > Hi Roman,
> >
> > Every continuous query is a unique entity that is processed by
> > servers independently. With your example, the server node will
> > execute all 20 filters for every cache insert/update operation. The
> > server will notify through local listeners only those clients whose
> > remote filters returned 'true'.
> >
> > -
> > Denis
> >
> >
> > On Thu, Jun 4, 2020 at 8:44 PM <[hidden email]> wrote:
> >
> > > Hi Community,
> > >
> > > I ask this question here because I haven't found the answer in the
> > > documentation.
> > >
> > > Could you please clarify how Continuous Queries work? What the
> > > behavior of Continuous Queries if we have several clients with
> > > different Remote Filters on the same cache? For example, if we have:
> > > one server node with cache and we have up to 20 client nodes each
> > > of them will execute Continuous Query on the same cache but with
> > > different Remote Filters. Will each client get the data according
> > > to its remote filter? Or it is supposed to have only one Remote
> > > Filter for all clients and every client should filter data in its
> > > local event
> > listener?
> > > I would be grateful if you send some link which describes the
> > > behavior of Continuous Queries more thoroughly.
> > > Best regards,
> > > Roman
> > >
> >
>

[dmagda]

Hi Roman,

Apologize for the late reply. Could you please clarify why it's useful to
the threads count parameter? How are you planning to use it?

-
Denis


On Tue, Jul 7, 2020 at 5:25 AM <[hidden email]> wrote:

> Hi Denis,
>
> What do you think about some improvements in @IgniteAsyncCallback
> regarding usability? What if we add a number of threads parameter in
> @IgniteAsyncCallback?
>
> Best regards,
> Roman
>
> -----Original Message-----
> From: Denis Magda <[hidden email]>
> Sent: Friday, June 26, 2020 7:38 PM
> To: dev <[hidden email]>
> Subject: Re: Continuous Queries with several remote filter on the same
> cache
>
> Roman,
>
> The updates are ordered per partition. Let's take this example of an
> application updating several records:
>
> put (k1, val1) => mapped to partition_10 => node_A put (k2, val2) =>
> mapped to partition_5 => node_B put (k3, val3) => mapped to partition_10 =>
> node_A
>
> It's guaranteed that a continuous query listener will be notified about k1
> and k3 updates in this order - k1 first and k3 after. As for the k2 update,
> it can arrive at any time (i.e., before k1, after k3 or in the middle).
>
>
>
>
> -
> Denis
>
>
> On Fri, Jun 26, 2020 at 12:58 AM <[hidden email]> wrote:
>
> > Hi Denis,
> >
> > Thanks! Is there some guarantee about the order of the updates? Even
> > when we have multiple cache nodes.
> >
> > Best regards,
> > Roman
> >
> > -----Original Message-----
> > From: Denis Magda <[hidden email]>
> > Sent: Monday, June 8, 2020 10:20 PM
> > To: dev <[hidden email]>
> > Subject: Re: Continuous Queries with several remote filter on the same
> > cache
> >
> > Roman,
> >
> > Please check the following methods:
> > * CacheContiniousQueryHandler (the filter usage):
> >
> > https://github.com/apache/ignite/blob/6955ac291352dd67c1f84a006cda512e
> > e54f38bb/modules/core/src/main/java/org/apache/ignite/internal/process
> > ors/cache/query/continuous/CacheContinuousQueryHandler.java#L994
> > * CacheContinuousQueryManager (the listener execution):
> >
> > https://github.com/apache/ignite/blob/master/modules/core/src/main/jav
> > a/org/apache/ignite/internal/processors/cache/query/continuous/CacheCo
> > ntinuousQueryManager.java#L376
> >
> > -
> > Denis
> >
> >
> > On Sun, Jun 7, 2020 at 12:19 AM <[hidden email]> wrote:
> >
> > > Hi Denis,
> > > A big thank you for the answer.
> > > Could you please tell me where can I find this logic in the sources.
> > > Which package should I look into?
> > >
> > > -----Original Message-----
> > > From: Denis Magda <[hidden email]>
> > > Sent: Saturday, June 6, 2020 2:07 AM
> > > To: dev <[hidden email]>
> > > Subject: Re: Continuous Queries with several remote filter on the
> > > same cache
> > >
> > > Hi Roman,
> > >
> > > Every continuous query is a unique entity that is processed by
> > > servers independently. With your example, the server node will
> > > execute all 20 filters for every cache insert/update operation. The
> > > server will notify through local listeners only those clients whose
> > > remote filters returned 'true'.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Jun 4, 2020 at 8:44 PM <[hidden email]> wrote:
> > >
> > > > Hi Community,
> > > >
> > > > I ask this question here because I haven't found the answer in the
> > > > documentation.
> > > >
> > > > Could you please clarify how Continuous Queries work? What the
> > > > behavior of Continuous Queries if we have several clients with
> > > > different Remote Filters on the same cache? For example, if we have:
> > > > one server node with cache and we have up to 20 client nodes each
> > > > of them will execute Continuous Query on the same cache but with
> > > > different Remote Filters. Will each client get the data according
> > > > to its remote filter? Or it is supposed to have only one Remote
> > > > Filter for all clients and every client should filter data in its
> > > > local event
> > > listener?
> > > > I would be grateful if you send some link which describes the
> > > > behavior of Continuous Queries more thoroughly.
> > > > Best regards,
> > > > Roman
> > > >
> > >
> >
>