Continuous queries changes
Locked
Continuous queries changes
 
|
Igniters,
Based on discussions with our users I came to conclusion that our continuous query implementation is not good enough for use cases with strong consistency requirements, because there is a possibility to lose updates in case of topology change. So I started working on https://issues.apache.org/jira/browse/IGNITE-426 and I hope to finish in in couple of weeks so that we can include it in next release. I have the following design in mind: - Maintain updates queue on backup node(s) in addition to primary node. - If primary node crushes, this queue is flushed to listening clients. - To avoid notification duplicates we will have a per-partition update counter. Once an entry in some partition is updated, counter for this partition is incremented on both primary and backups. The value of this counter is also sent along with the update to the client, which also maintains the copy of this mapping. If at some moment it receives an update with the counter less than in its local map, this update is a duplicate and can be discarded. - Also need to figure out the best way to clean the backup queue if topology is stable. Will be happy to hear any suggestions :) To make all this work we also need to implement thread-per-partition mode in atomic cache, because now updates order on backup nodes can differ from the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm already working on this. Feel free to share your thoughts! -Val |
|
|
Val,
I have idea on how to clean up backup queue. 1. Our communication uses acks. So, you can determine [on server node] whether client received the update from local server or not. I think you can easily change existing code to get notifications on ack receiving (this way you dont need to introduce your own acks). 2. How do you know when evict from backup? Each message that client acks corresponds to some per-partition long value you talked above (great idea, btw!). Servers can exchange per-partition long value that corresponds to the latest acked message and that's the way how backups can safely evict from the queue. Let me know if you have questions. --Yakov 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko <[hidden email] >: > Igniters, > > Based on discussions with our users I came to conclusion that our > continuous query implementation is not good enough for use cases with > strong consistency requirements, because there is a possibility to lose > updates in case of topology change. > > So I started working on https://issues.apache.org/jira/browse/IGNITE-426 > and I hope to finish in in couple of weeks so that we can include it in > next release. > > I have the following design in mind: > > - Maintain updates queue on backup node(s) in addition to primary node. > - If primary node crushes, this queue is flushed to listening clients. > - To avoid notification duplicates we will have a per-partition update > counter. Once an entry in some partition is updated, counter for this > partition is incremented on both primary and backups. The value of this > counter is also sent along with the update to the client, which also > maintains the copy of this mapping. If at some moment it receives an > update > with the counter less than in its local map, this update is a duplicate > and > can be discarded. > - Also need to figure out the best way to clean the backup queue if > topology is stable. Will be happy to hear any suggestions :) > > To make all this work we also need to implement thread-per-partition mode > in atomic cache, because now updates order on backup nodes can differ from > the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm > already working on this. > > Feel free to share your thoughts! > > -Val > |
|
|
Val,
Could you please elaborate what you mean by "updates queue" you plan to maintain on the primary and backup nodes? Thanks Andrey > Date: Fri, 24 Jul 2015 17:51:48 +0300 > Subject: Re: Continuous queries changes > From: [hidden email] > To: [hidden email] > > Val, > > I have idea on how to clean up backup queue. > > 1. Our communication uses acks. So, you can determine [on server node] > whether client received the update from local server or not. I think you > can easily change existing code to get notifications on ack receiving (this > way you dont need to introduce your own acks). > 2. How do you know when evict from backup? Each message that client acks > corresponds to some per-partition long value you talked above (great idea, > btw!). Servers can exchange per-partition long value that corresponds to > the latest acked message and that's the way how backups can safely evict > from the queue. > > Let me know if you have questions. > > --Yakov > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko <[hidden email] > >: > > > Igniters, > > > > Based on discussions with our users I came to conclusion that our > > continuous query implementation is not good enough for use cases with > > strong consistency requirements, because there is a possibility to lose > > updates in case of topology change. > > > > So I started working on https://issues.apache.org/jira/browse/IGNITE-426 > > and I hope to finish in in couple of weeks so that we can include it in > > next release. > > > > I have the following design in mind: > > > > - Maintain updates queue on backup node(s) in addition to primary node. > > - If primary node crushes, this queue is flushed to listening clients. > > - To avoid notification duplicates we will have a per-partition update > > counter. Once an entry in some partition is updated, counter for this > > partition is incremented on both primary and backups. The value of this > > counter is also sent along with the update to the client, which also > > maintains the copy of this mapping. If at some moment it receives an > > update > > with the counter less than in its local map, this update is a duplicate > > and > > can be discarded. > > - Also need to figure out the best way to clean the backup queue if > > topology is stable. Will be happy to hear any suggestions :) > > > > To make all this work we also need to implement thread-per-partition mode > > in atomic cache, because now updates order on backup nodes can differ from > > the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm > > already working on this. > > > > Feel free to share your thoughts! > > > > -Val > > |
Re: Continuous queries changes
|
|
Andrey,
I mean the queue of update events that is collected on backup nodes and flushed to listening clients in case of topology change. -Val On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev <[hidden email]> wrote: > Val, > > Could you please elaborate what you mean by "updates queue" you plan to > maintain on the primary and backup nodes? > > Thanks > Andrey > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > Subject: Re: Continuous queries changes > > From: [hidden email] > > To: [hidden email] > > > > Val, > > > > I have idea on how to clean up backup queue. > > > > 1. Our communication uses acks. So, you can determine [on server node] > > whether client received the update from local server or not. I think you > > can easily change existing code to get notifications on ack receiving > (this > > way you dont need to introduce your own acks). > > 2. How do you know when evict from backup? Each message that client acks > > corresponds to some per-partition long value you talked above (great > idea, > > btw!). Servers can exchange per-partition long value that corresponds to > > the latest acked message and that's the way how backups can safely evict > > from the queue. > > > > Let me know if you have questions. > > > > --Yakov > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > [hidden email] > > >: > > > > > Igniters, > > > > > > Based on discussions with our users I came to conclusion that our > > > continuous query implementation is not good enough for use cases with > > > strong consistency requirements, because there is a possibility to lose > > > updates in case of topology change. > > > > > > So I started working on > https://issues.apache.org/jira/browse/IGNITE-426 > > > and I hope to finish in in couple of weeks so that we can include it in > > > next release. > > > > > > I have the following design in mind: > > > > > > - Maintain updates queue on backup node(s) in addition to primary > node. > > > - If primary node crushes, this queue is flushed to listening > clients. > > > - To avoid notification duplicates we will have a per-partition > update > > > counter. Once an entry in some partition is updated, counter for > this > > > partition is incremented on both primary and backups. The value of > this > > > counter is also sent along with the update to the client, which also > > > maintains the copy of this mapping. If at some moment it receives an > > > update > > > with the counter less than in its local map, this update is a > duplicate > > > and > > > can be discarded. > > > - Also need to figure out the best way to clean the backup queue if > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > To make all this work we also need to implement thread-per-partition > mode > > > in atomic cache, because now updates order on backup nodes can differ > from > > > the primary node: https://issues.apache.org/jira/browse/IGNITE-104. > I'm > > > already working on this. > > > > > > Feel free to share your thoughts! > > > > > > -Val > > > > > |
|
|
Val,
I'm sorry for being obtuse. :) I was simply wondering if the queue is going to be holding all unfiltered events per partition or will there be a queue per continuous query instance per partition? Or, is it going to be arranged some other way? Also, in order to know when it's ok to remove an event from the backup queue, wouldn't this approach require maintaining a queue for each connected client and having to deal with potentially unbounded queue growth if a client struggles to keep up or simply stops acking? Isn't this feature getting Ignite into the murky waters of the message brokers and guaranteed once-only message delivery with all the complexity and overhead that come with it? Besides in some cases, it's doesn't really matter if some updates are missing, while in others it is only necessary to be able to detect a missing update. I wouldn't want to have to pay for something I don't need... Thanks Andrey > Date: Fri, 24 Jul 2015 23:40:15 -0700 > Subject: Re: Continuous queries changes > From: [hidden email] > To: [hidden email] > > Andrey, > > I mean the queue of update events that is collected on backup nodes and > flushed to listening clients in case of topology change. > > -Val > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev <[hidden email]> > wrote: > > > Val, > > > > Could you please elaborate what you mean by "updates queue" you plan to > > maintain on the primary and backup nodes? > > > > Thanks > > Andrey > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > Subject: Re: Continuous queries changes > > > From: [hidden email] > > > To: [hidden email] > > > > > > Val, > > > > > > I have idea on how to clean up backup queue. > > > > > > 1. Our communication uses acks. So, you can determine [on server node] > > > whether client received the update from local server or not. I think you > > > can easily change existing code to get notifications on ack receiving > > (this > > > way you dont need to introduce your own acks). > > > 2. How do you know when evict from backup? Each message that client acks > > > corresponds to some per-partition long value you talked above (great > > idea, > > > btw!). Servers can exchange per-partition long value that corresponds to > > > the latest acked message and that's the way how backups can safely evict > > > from the queue. > > > > > > Let me know if you have questions. > > > > > > --Yakov > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > [hidden email] > > > >: > > > > > > > Igniters, > > > > > > > > Based on discussions with our users I came to conclusion that our > > > > continuous query implementation is not good enough for use cases with > > > > strong consistency requirements, because there is a possibility to lose > > > > updates in case of topology change. > > > > > > > > So I started working on > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > and I hope to finish in in couple of weeks so that we can include it in > > > > next release. > > > > > > > > I have the following design in mind: > > > > > > > > - Maintain updates queue on backup node(s) in addition to primary > > node. > > > > - If primary node crushes, this queue is flushed to listening > > clients. > > > > - To avoid notification duplicates we will have a per-partition > > update > > > > counter. Once an entry in some partition is updated, counter for > > this > > > > partition is incremented on both primary and backups. The value of > > this > > > > counter is also sent along with the update to the client, which also > > > > maintains the copy of this mapping. If at some moment it receives an > > > > update > > > > with the counter less than in its local map, this update is a > > duplicate > > > > and > > > > can be discarded. > > > > - Also need to figure out the best way to clean the backup queue if > > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > > > To make all this work we also need to implement thread-per-partition > > mode > > > > in atomic cache, because now updates order on backup nodes can differ > > from > > > > the primary node: https://issues.apache.org/jira/browse/IGNITE-104. > > I'm > > > > already working on this. > > > > > > > > Feel free to share your thoughts! > > > > > > > > -Val > > > > > > > > |
|
On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev <[hidden email]>
wrote: > Val, > > I'm sorry for being obtuse. :) > > I was simply wondering if the queue is going to be holding all unfiltered > events per partition or will there be a queue per continuous query instance > per partition? Or, is it going to be arranged some other way? > I believe that backup queues will have the same filters as primary queues. > Also, in order to know when it's ok to remove an event from the backup > queue, wouldn't this approach require maintaining a queue for each > connected client and having to deal with potentially unbounded queue > growth if a client struggles to keep up or simply stops acking? > I think the policy for backups should be no different as for the primaries. As far as slow clients, Ignite is capable to automatically disconnect them: http://s.apache.org/managing-slow-clients Isn't this feature getting Ignite into the murky waters of the message > brokers and guaranteed once-only message delivery with all the complexity > and overhead that come with it? Besides in some cases, it's doesn't really > matter if some updates are missing, while in others it is only necessary to > be able to detect a missing update. I wouldn't want to have to pay for > something I don't need... > I believe that the new proposed approach will be optional and you will still be able to get event notifications in non-fault-tolerant manner the old way. > > Thanks > Andrey > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > Subject: Re: Continuous queries changes > > From: [hidden email] > > To: [hidden email] > > > > Andrey, > > > > I mean the queue of update events that is collected on backup nodes and > > flushed to listening clients in case of topology change. > > > > -Val > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev <[hidden email] > > > > wrote: > > > > > Val, > > > > > > Could you please elaborate what you mean by "updates queue" you plan to > > > maintain on the primary and backup nodes? > > > > > > Thanks > > > Andrey > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > Subject: Re: Continuous queries changes > > > > From: [hidden email] > > > > To: [hidden email] > > > > > > > > Val, > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > 1. Our communication uses acks. So, you can determine [on server > node] > > > > whether client received the update from local server or not. I think > you > > > > can easily change existing code to get notifications on ack receiving > > > (this > > > > way you dont need to introduce your own acks). > > > > 2. How do you know when evict from backup? Each message that client > acks > > > > corresponds to some per-partition long value you talked above (great > > > idea, > > > > btw!). Servers can exchange per-partition long value that > corresponds to > > > > the latest acked message and that's the way how backups can safely > evict > > > > from the queue. > > > > > > > > Let me know if you have questions. > > > > > > > > --Yakov > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > [hidden email] > > > > >: > > > > > > > > > Igniters, > > > > > > > > > > Based on discussions with our users I came to conclusion that our > > > > > continuous query implementation is not good enough for use cases > with > > > > > strong consistency requirements, because there is a possibility to > lose > > > > > updates in case of topology change. > > > > > > > > > > So I started working on > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > and I hope to finish in in couple of weeks so that we can include > it in > > > > > next release. > > > > > > > > > > I have the following design in mind: > > > > > > > > > > - Maintain updates queue on backup node(s) in addition to > primary > > > node. > > > > > - If primary node crushes, this queue is flushed to listening > > > clients. > > > > > - To avoid notification duplicates we will have a per-partition > > > update > > > > > counter. Once an entry in some partition is updated, counter for > > > this > > > > > partition is incremented on both primary and backups. The value > of > > > this > > > > > counter is also sent along with the update to the client, which > also > > > > > maintains the copy of this mapping. If at some moment it > receives an > > > > > update > > > > > with the counter less than in its local map, this update is a > > > duplicate > > > > > and > > > > > can be discarded. > > > > > - Also need to figure out the best way to clean the backup > queue if > > > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > > > > > To make all this work we also need to implement > thread-per-partition > > > mode > > > > > in atomic cache, because now updates order on backup nodes can > differ > > > from > > > > > the primary node: https://issues.apache.org/jira/browse/IGNITE-104 > . > > > I'm > > > > > already working on this. > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > -Val > > > > > > > > > > > > |
|
|
I wonder if the same result (guaranteed delivery of CQ notifications) can be achieved entirely in the "user space" using the public Ignite API only?
For example: - start a server-side CQ and have the listener push the notifications into an IgniteQueue. - have the client connect to the queue and start receiving the notifications. Regards Andrey > From: [hidden email] > Date: Sun, 26 Jul 2015 22:15:09 -0700 > Subject: Re: Continuous queries changes > To: [hidden email] > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev <[hidden email]> > wrote: > > > Val, > > > > I'm sorry for being obtuse. :) > > > > I was simply wondering if the queue is going to be holding all unfiltered > > events per partition or will there be a queue per continuous query instance > > per partition? Or, is it going to be arranged some other way? > > > > I believe that backup queues will have the same filters as primary queues. > > > > Also, in order to know when it's ok to remove an event from the backup > > queue, wouldn't this approach require maintaining a queue for each > > connected client and having to deal with potentially unbounded queue > > growth if a client struggles to keep up or simply stops acking? > > > > I think the policy for backups should be no different as for the primaries. > As far as slow clients, Ignite is capable to automatically disconnect them: > http://s.apache.org/managing-slow-clients > > Isn't this feature getting Ignite into the murky waters of the message > > brokers and guaranteed once-only message delivery with all the complexity > > and overhead that come with it? Besides in some cases, it's doesn't really > > matter if some updates are missing, while in others it is only necessary to > > be able to detect a missing update. I wouldn't want to have to pay for > > something I don't need... > > > > I believe that the new proposed approach will be optional and you will > still be able to get event notifications in non-fault-tolerant manner the > old way. > > > > > > Thanks > > Andrey > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > Subject: Re: Continuous queries changes > > > From: [hidden email] > > > To: [hidden email] > > > > > > Andrey, > > > > > > I mean the queue of update events that is collected on backup nodes and > > > flushed to listening clients in case of topology change. > > > > > > -Val > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev <[hidden email] > > > > > > wrote: > > > > > > > Val, > > > > > > > > Could you please elaborate what you mean by "updates queue" you plan to > > > > maintain on the primary and backup nodes? > > > > > > > > Thanks > > > > Andrey > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > Subject: Re: Continuous queries changes > > > > > From: [hidden email] > > > > > To: [hidden email] > > > > > > > > > > Val, > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on server > > node] > > > > > whether client received the update from local server or not. I think > > you > > > > > can easily change existing code to get notifications on ack receiving > > > > (this > > > > > way you dont need to introduce your own acks). > > > > > 2. How do you know when evict from backup? Each message that client > > acks > > > > > corresponds to some per-partition long value you talked above (great > > > > idea, > > > > > btw!). Servers can exchange per-partition long value that > > corresponds to > > > > > the latest acked message and that's the way how backups can safely > > evict > > > > > from the queue. > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > --Yakov > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > [hidden email] > > > > > >: > > > > > > > > > > > Igniters, > > > > > > > > > > > > Based on discussions with our users I came to conclusion that our > > > > > > continuous query implementation is not good enough for use cases > > with > > > > > > strong consistency requirements, because there is a possibility to > > lose > > > > > > updates in case of topology change. > > > > > > > > > > > > So I started working on > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > and I hope to finish in in couple of weeks so that we can include > > it in > > > > > > next release. > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition to > > primary > > > > node. > > > > > > - If primary node crushes, this queue is flushed to listening > > > > clients. > > > > > > - To avoid notification duplicates we will have a per-partition > > > > update > > > > > > counter. Once an entry in some partition is updated, counter for > > > > this > > > > > > partition is incremented on both primary and backups. The value > > of > > > > this > > > > > > counter is also sent along with the update to the client, which > > also > > > > > > maintains the copy of this mapping. If at some moment it > > receives an > > > > > > update > > > > > > with the counter less than in its local map, this update is a > > > > duplicate > > > > > > and > > > > > > can be discarded. > > > > > > - Also need to figure out the best way to clean the backup > > queue if > > > > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > > > > > > > To make all this work we also need to implement > > thread-per-partition > > > > mode > > > > > > in atomic cache, because now updates order on backup nodes can > > differ > > > > from > > > > > > the primary node: https://issues.apache.org/jira/browse/IGNITE-104 > > . > > > > I'm > > > > > > already working on this. > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > |
|
|
On Mon, Jul 27, 2015 at 7:35 AM, Andrey Kornev <[hidden email]>
wrote: > I wonder if the same result (guaranteed delivery of CQ notifications) can > be achieved entirely in the "user space" using the public Ignite API only? > > For example: > - start a server-side CQ and have the listener push the notifications into > an IgniteQueue. > - have the client connect to the queue and start receiving the > notifications. > Hm... Do you mean that in this approach we will have 1 CQ queue per server, instead of 1 queue per subscription, as we planned before? > > Regards > Andrey > > > From: [hidden email] > > Date: Sun, 26 Jul 2015 22:15:09 -0700 > > Subject: Re: Continuous queries changes > > To: [hidden email] > > > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev <[hidden email] > > > > wrote: > > > > > Val, > > > > > > I'm sorry for being obtuse. :) > > > > > > I was simply wondering if the queue is going to be holding all > unfiltered > > > events per partition or will there be a queue per continuous query > instance > > > per partition? Or, is it going to be arranged some other way? > > > > > > > I believe that backup queues will have the same filters as primary > queues. > > > > > > > Also, in order to know when it's ok to remove an event from the backup > > > queue, wouldn't this approach require maintaining a queue for each > > > connected client and having to deal with potentially unbounded queue > > > growth if a client struggles to keep up or simply stops acking? > > > > > > > I think the policy for backups should be no different as for the > primaries. > > As far as slow clients, Ignite is capable to automatically disconnect > them: > > http://s.apache.org/managing-slow-clients > > > > Isn't this feature getting Ignite into the murky waters of the message > > > brokers and guaranteed once-only message delivery with all the > complexity > > > and overhead that come with it? Besides in some cases, it's doesn't > really > > > matter if some updates are missing, while in others it is only > necessary to > > > be able to detect a missing update. I wouldn't want to have to pay for > > > something I don't need... > > > > > > > I believe that the new proposed approach will be optional and you will > > still be able to get event notifications in non-fault-tolerant manner the > > old way. > > > > > > > > > > Thanks > > > Andrey > > > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > > Subject: Re: Continuous queries changes > > > > From: [hidden email] > > > > To: [hidden email] > > > > > > > > Andrey, > > > > > > > > I mean the queue of update events that is collected on backup nodes > and > > > > flushed to listening clients in case of topology change. > > > > > > > > -Val > > > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > [hidden email] > > > > > > > > wrote: > > > > > > > > > Val, > > > > > > > > > > Could you please elaborate what you mean by "updates queue" you > plan to > > > > > maintain on the primary and backup nodes? > > > > > > > > > > Thanks > > > > > Andrey > > > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > > Subject: Re: Continuous queries changes > > > > > > From: [hidden email] > > > > > > To: [hidden email] > > > > > > > > > > > > Val, > > > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on server > > > node] > > > > > > whether client received the update from local server or not. I > think > > > you > > > > > > can easily change existing code to get notifications on ack > receiving > > > > > (this > > > > > > way you dont need to introduce your own acks). > > > > > > 2. How do you know when evict from backup? Each message that > client > > > acks > > > > > > corresponds to some per-partition long value you talked above > (great > > > > > idea, > > > > > > btw!). Servers can exchange per-partition long value that > > > corresponds to > > > > > > the latest acked message and that's the way how backups can > safely > > > evict > > > > > > from the queue. > > > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > > > --Yakov > > > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > Based on discussions with our users I came to conclusion that > our > > > > > > > continuous query implementation is not good enough for use > cases > > > with > > > > > > > strong consistency requirements, because there is a > possibility to > > > lose > > > > > > > updates in case of topology change. > > > > > > > > > > > > > > So I started working on > > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > and I hope to finish in in couple of weeks so that we can > include > > > it in > > > > > > > next release. > > > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition to > > > primary > > > > > node. > > > > > > > - If primary node crushes, this queue is flushed to > listening > > > > > clients. > > > > > > > - To avoid notification duplicates we will have a > per-partition > > > > > update > > > > > > > counter. Once an entry in some partition is updated, > counter for > > > > > this > > > > > > > partition is incremented on both primary and backups. The > value > > > of > > > > > this > > > > > > > counter is also sent along with the update to the client, > which > > > also > > > > > > > maintains the copy of this mapping. If at some moment it > > > receives an > > > > > > > update > > > > > > > with the counter less than in its local map, this update is > a > > > > > duplicate > > > > > > > and > > > > > > > can be discarded. > > > > > > > - Also need to figure out the best way to clean the backup > > > queue if > > > > > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > > > > > > > > > To make all this work we also need to implement > > > thread-per-partition > > > > > mode > > > > > > > in atomic cache, because now updates order on backup nodes can > > > differ > > > > > from > > > > > > > the primary node: > https://issues.apache.org/jira/browse/IGNITE-104 > > > . > > > > > I'm > > > > > > > already working on this. > > > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > |
Re: Continuous queries changes
|
|
In reply to this post by dsetrakyan
I have nothing to add. Thanks, Dmitry :)
On Sun, Jul 26, 2015 at 10:15 PM, Dmitriy Setrakyan <[hidden email]> wrote: > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev <[hidden email]> > wrote: > > > Val, > > > > I'm sorry for being obtuse. :) > > > > I was simply wondering if the queue is going to be holding all unfiltered > > events per partition or will there be a queue per continuous query > instance > > per partition? Or, is it going to be arranged some other way? > > > > I believe that backup queues will have the same filters as primary queues. > > > > Also, in order to know when it's ok to remove an event from the backup > > queue, wouldn't this approach require maintaining a queue for each > > connected client and having to deal with potentially unbounded queue > > growth if a client struggles to keep up or simply stops acking? > > > > I think the policy for backups should be no different as for the primaries. > As far as slow clients, Ignite is capable to automatically disconnect them: > http://s.apache.org/managing-slow-clients > > Isn't this feature getting Ignite into the murky waters of the message > > brokers and guaranteed once-only message delivery with all the complexity > > and overhead that come with it? Besides in some cases, it's doesn't > really > > matter if some updates are missing, while in others it is only necessary > to > > be able to detect a missing update. I wouldn't want to have to pay for > > something I don't need... > > > > I believe that the new proposed approach will be optional and you will > still be able to get event notifications in non-fault-tolerant manner the > old way. > > > > > > Thanks > > Andrey > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > Subject: Re: Continuous queries changes > > > From: [hidden email] > > > To: [hidden email] > > > > > > Andrey, > > > > > > I mean the queue of update events that is collected on backup nodes and > > > flushed to listening clients in case of topology change. > > > > > > -Val > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > [hidden email] > > > > > > wrote: > > > > > > > Val, > > > > > > > > Could you please elaborate what you mean by "updates queue" you plan > to > > > > maintain on the primary and backup nodes? > > > > > > > > Thanks > > > > Andrey > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > Subject: Re: Continuous queries changes > > > > > From: [hidden email] > > > > > To: [hidden email] > > > > > > > > > > Val, > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on server > > node] > > > > > whether client received the update from local server or not. I > think > > you > > > > > can easily change existing code to get notifications on ack > receiving > > > > (this > > > > > way you dont need to introduce your own acks). > > > > > 2. How do you know when evict from backup? Each message that client > > acks > > > > > corresponds to some per-partition long value you talked above > (great > > > > idea, > > > > > btw!). Servers can exchange per-partition long value that > > corresponds to > > > > > the latest acked message and that's the way how backups can safely > > evict > > > > > from the queue. > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > --Yakov > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > [hidden email] > > > > > >: > > > > > > > > > > > Igniters, > > > > > > > > > > > > Based on discussions with our users I came to conclusion that our > > > > > > continuous query implementation is not good enough for use cases > > with > > > > > > strong consistency requirements, because there is a possibility > to > > lose > > > > > > updates in case of topology change. > > > > > > > > > > > > So I started working on > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > and I hope to finish in in couple of weeks so that we can include > > it in > > > > > > next release. > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition to > > primary > > > > node. > > > > > > - If primary node crushes, this queue is flushed to listening > > > > clients. > > > > > > - To avoid notification duplicates we will have a > per-partition > > > > update > > > > > > counter. Once an entry in some partition is updated, counter > for > > > > this > > > > > > partition is incremented on both primary and backups. The > value > > of > > > > this > > > > > > counter is also sent along with the update to the client, > which > > also > > > > > > maintains the copy of this mapping. If at some moment it > > receives an > > > > > > update > > > > > > with the counter less than in its local map, this update is a > > > > duplicate > > > > > > and > > > > > > can be discarded. > > > > > > - Also need to figure out the best way to clean the backup > > queue if > > > > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > > > > > > > To make all this work we also need to implement > > thread-per-partition > > > > mode > > > > > > in atomic cache, because now updates order on backup nodes can > > differ > > > > from > > > > > > the primary node: > https://issues.apache.org/jira/browse/IGNITE-104 > > . > > > > I'm > > > > > > already working on this. > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > |
Re: Continuous queries changes
|
|
In reply to this post by Andrey Kornev
Andrey,
I think your approach works, but it requires periodical polling from queue. Continuous queries provide an ability to get push notifications for updates, which from my experience is critical for some use cases. -Val On Mon, Jul 27, 2015 at 7:35 AM, Andrey Kornev <[hidden email]> wrote: > I wonder if the same result (guaranteed delivery of CQ notifications) can > be achieved entirely in the "user space" using the public Ignite API only? > > For example: > - start a server-side CQ and have the listener push the notifications into > an IgniteQueue. > - have the client connect to the queue and start receiving the > notifications. > > Regards > Andrey > > > From: [hidden email] > > Date: Sun, 26 Jul 2015 22:15:09 -0700 > > Subject: Re: Continuous queries changes > > To: [hidden email] > > > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev <[hidden email] > > > > wrote: > > > > > Val, > > > > > > I'm sorry for being obtuse. :) > > > > > > I was simply wondering if the queue is going to be holding all > unfiltered > > > events per partition or will there be a queue per continuous query > instance > > > per partition? Or, is it going to be arranged some other way? > > > > > > > I believe that backup queues will have the same filters as primary > queues. > > > > > > > Also, in order to know when it's ok to remove an event from the backup > > > queue, wouldn't this approach require maintaining a queue for each > > > connected client and having to deal with potentially unbounded queue > > > growth if a client struggles to keep up or simply stops acking? > > > > > > > I think the policy for backups should be no different as for the > primaries. > > As far as slow clients, Ignite is capable to automatically disconnect > them: > > http://s.apache.org/managing-slow-clients > > > > Isn't this feature getting Ignite into the murky waters of the message > > > brokers and guaranteed once-only message delivery with all the > complexity > > > and overhead that come with it? Besides in some cases, it's doesn't > really > > > matter if some updates are missing, while in others it is only > necessary to > > > be able to detect a missing update. I wouldn't want to have to pay for > > > something I don't need... > > > > > > > I believe that the new proposed approach will be optional and you will > > still be able to get event notifications in non-fault-tolerant manner the > > old way. > > > > > > > > > > Thanks > > > Andrey > > > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > > Subject: Re: Continuous queries changes > > > > From: [hidden email] > > > > To: [hidden email] > > > > > > > > Andrey, > > > > > > > > I mean the queue of update events that is collected on backup nodes > and > > > > flushed to listening clients in case of topology change. > > > > > > > > -Val > > > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > [hidden email] > > > > > > > > wrote: > > > > > > > > > Val, > > > > > > > > > > Could you please elaborate what you mean by "updates queue" you > plan to > > > > > maintain on the primary and backup nodes? > > > > > > > > > > Thanks > > > > > Andrey > > > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > > Subject: Re: Continuous queries changes > > > > > > From: [hidden email] > > > > > > To: [hidden email] > > > > > > > > > > > > Val, > > > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on server > > > node] > > > > > > whether client received the update from local server or not. I > think > > > you > > > > > > can easily change existing code to get notifications on ack > receiving > > > > > (this > > > > > > way you dont need to introduce your own acks). > > > > > > 2. How do you know when evict from backup? Each message that > client > > > acks > > > > > > corresponds to some per-partition long value you talked above > (great > > > > > idea, > > > > > > btw!). Servers can exchange per-partition long value that > > > corresponds to > > > > > > the latest acked message and that's the way how backups can > safely > > > evict > > > > > > from the queue. > > > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > > > --Yakov > > > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > Based on discussions with our users I came to conclusion that > our > > > > > > > continuous query implementation is not good enough for use > cases > > > with > > > > > > > strong consistency requirements, because there is a > possibility to > > > lose > > > > > > > updates in case of topology change. > > > > > > > > > > > > > > So I started working on > > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > and I hope to finish in in couple of weeks so that we can > include > > > it in > > > > > > > next release. > > > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition to > > > primary > > > > > node. > > > > > > > - If primary node crushes, this queue is flushed to > listening > > > > > clients. > > > > > > > - To avoid notification duplicates we will have a > per-partition > > > > > update > > > > > > > counter. Once an entry in some partition is updated, > counter for > > > > > this > > > > > > > partition is incremented on both primary and backups. The > value > > > of > > > > > this > > > > > > > counter is also sent along with the update to the client, > which > > > also > > > > > > > maintains the copy of this mapping. If at some moment it > > > receives an > > > > > > > update > > > > > > > with the counter less than in its local map, this update is > a > > > > > duplicate > > > > > > > and > > > > > > > can be discarded. > > > > > > > - Also need to figure out the best way to clean the backup > > > queue if > > > > > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > > > > > > > > > To make all this work we also need to implement > > > thread-per-partition > > > > > mode > > > > > > > in atomic cache, because now updates order on backup nodes can > > > differ > > > > > from > > > > > > > the primary node: > https://issues.apache.org/jira/browse/IGNITE-104 > > > . > > > > > I'm > > > > > > > already working on this. > > > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > |
|
|
Hi Val,
I was hoping to be able to use the blocking IgniteQueue.take() to achieve the desirable "push" semantics. Andrey > Date: Mon, 27 Jul 2015 11:43:13 -0700 > Subject: Re: Continuous queries changes > From: [hidden email] > To: [hidden email] > > Andrey, > > I think your approach works, but it requires periodical polling from queue. > Continuous queries provide an ability to get push notifications for > updates, which from my experience is critical for some use cases. > > -Val > > On Mon, Jul 27, 2015 at 7:35 AM, Andrey Kornev <[hidden email]> > wrote: > > > I wonder if the same result (guaranteed delivery of CQ notifications) can > > be achieved entirely in the "user space" using the public Ignite API only? > > > > For example: > > - start a server-side CQ and have the listener push the notifications into > > an IgniteQueue. > > - have the client connect to the queue and start receiving the > > notifications. > > > > Regards > > Andrey > > > > > From: [hidden email] > > > Date: Sun, 26 Jul 2015 22:15:09 -0700 > > > Subject: Re: Continuous queries changes > > > To: [hidden email] > > > > > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev <[hidden email] > > > > > > wrote: > > > > > > > Val, > > > > > > > > I'm sorry for being obtuse. :) > > > > > > > > I was simply wondering if the queue is going to be holding all > > unfiltered > > > > events per partition or will there be a queue per continuous query > > instance > > > > per partition? Or, is it going to be arranged some other way? > > > > > > > > > > I believe that backup queues will have the same filters as primary > > queues. > > > > > > > > > > Also, in order to know when it's ok to remove an event from the backup > > > > queue, wouldn't this approach require maintaining a queue for each > > > > connected client and having to deal with potentially unbounded queue > > > > growth if a client struggles to keep up or simply stops acking? > > > > > > > > > > I think the policy for backups should be no different as for the > > primaries. > > > As far as slow clients, Ignite is capable to automatically disconnect > > them: > > > http://s.apache.org/managing-slow-clients > > > > > > Isn't this feature getting Ignite into the murky waters of the message > > > > brokers and guaranteed once-only message delivery with all the > > complexity > > > > and overhead that come with it? Besides in some cases, it's doesn't > > really > > > > matter if some updates are missing, while in others it is only > > necessary to > > > > be able to detect a missing update. I wouldn't want to have to pay for > > > > something I don't need... > > > > > > > > > > I believe that the new proposed approach will be optional and you will > > > still be able to get event notifications in non-fault-tolerant manner the > > > old way. > > > > > > > > > > > > > > Thanks > > > > Andrey > > > > > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > > > Subject: Re: Continuous queries changes > > > > > From: [hidden email] > > > > > To: [hidden email] > > > > > > > > > > Andrey, > > > > > > > > > > I mean the queue of update events that is collected on backup nodes > > and > > > > > flushed to listening clients in case of topology change. > > > > > > > > > > -Val > > > > > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > Val, > > > > > > > > > > > > Could you please elaborate what you mean by "updates queue" you > > plan to > > > > > > maintain on the primary and backup nodes? > > > > > > > > > > > > Thanks > > > > > > Andrey > > > > > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > > > Subject: Re: Continuous queries changes > > > > > > > From: [hidden email] > > > > > > > To: [hidden email] > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on server > > > > node] > > > > > > > whether client received the update from local server or not. I > > think > > > > you > > > > > > > can easily change existing code to get notifications on ack > > receiving > > > > > > (this > > > > > > > way you dont need to introduce your own acks). > > > > > > > 2. How do you know when evict from backup? Each message that > > client > > > > acks > > > > > > > corresponds to some per-partition long value you talked above > > (great > > > > > > idea, > > > > > > > btw!). Servers can exchange per-partition long value that > > > > corresponds to > > > > > > > the latest acked message and that's the way how backups can > > safely > > > > evict > > > > > > > from the queue. > > > > > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > > > > > --Yakov > > > > > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > Based on discussions with our users I came to conclusion that > > our > > > > > > > > continuous query implementation is not good enough for use > > cases > > > > with > > > > > > > > strong consistency requirements, because there is a > > possibility to > > > > lose > > > > > > > > updates in case of topology change. > > > > > > > > > > > > > > > > So I started working on > > > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > > and I hope to finish in in couple of weeks so that we can > > include > > > > it in > > > > > > > > next release. > > > > > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition to > > > > primary > > > > > > node. > > > > > > > > - If primary node crushes, this queue is flushed to > > listening > > > > > > clients. > > > > > > > > - To avoid notification duplicates we will have a > > per-partition > > > > > > update > > > > > > > > counter. Once an entry in some partition is updated, > > counter for > > > > > > this > > > > > > > > partition is incremented on both primary and backups. The > > value > > > > of > > > > > > this > > > > > > > > counter is also sent along with the update to the client, > > which > > > > also > > > > > > > > maintains the copy of this mapping. If at some moment it > > > > receives an > > > > > > > > update > > > > > > > > with the counter less than in its local map, this update is > > a > > > > > > duplicate > > > > > > > > and > > > > > > > > can be discarded. > > > > > > > > - Also need to figure out the best way to clean the backup > > > > queue if > > > > > > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > > > > > > > > > > > To make all this work we also need to implement > > > > thread-per-partition > > > > > > mode > > > > > > > > in atomic cache, because now updates order on backup nodes can > > > > differ > > > > > > from > > > > > > > > the primary node: > > https://issues.apache.org/jira/browse/IGNITE-104 > > > > . > > > > > > I'm > > > > > > > > already working on this. > > > > > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
In reply to this post by dsetrakyan
Dmitriy,
What I had in mind was to have each client create a dedicated (replicated) queue for each CQ it's about to start. Once started, the server-side CQ listener would then simply push the notifications onto the queue. With such setup, the CQ notifications should be able to survive crashes of the server nodes as well as the client node reconnects. It would also allow the users handle the queue growth in a more graceful (or application specific) way than a client disconnect. Andrey > From: [hidden email] > Date: Mon, 27 Jul 2015 07:47:39 -0700 > Subject: Re: Continuous queries changes > To: [hidden email] > > On Mon, Jul 27, 2015 at 7:35 AM, Andrey Kornev <[hidden email]> > wrote: > > > I wonder if the same result (guaranteed delivery of CQ notifications) can > > be achieved entirely in the "user space" using the public Ignite API only? > > > > For example: > > - start a server-side CQ and have the listener push the notifications into > > an IgniteQueue. > > - have the client connect to the queue and start receiving the > > notifications. > > > > Hm... Do you mean that in this approach we will have 1 CQ queue per server, > instead of 1 queue per subscription, as we planned before? > > > > > > Regards > > Andrey > > > > > From: [hidden email] > > > Date: Sun, 26 Jul 2015 22:15:09 -0700 > > > Subject: Re: Continuous queries changes > > > To: [hidden email] > > > > > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev <[hidden email] > > > > > > wrote: > > > > > > > Val, > > > > > > > > I'm sorry for being obtuse. :) > > > > > > > > I was simply wondering if the queue is going to be holding all > > unfiltered > > > > events per partition or will there be a queue per continuous query > > instance > > > > per partition? Or, is it going to be arranged some other way? > > > > > > > > > > I believe that backup queues will have the same filters as primary > > queues. > > > > > > > > > > Also, in order to know when it's ok to remove an event from the backup > > > > queue, wouldn't this approach require maintaining a queue for each > > > > connected client and having to deal with potentially unbounded queue > > > > growth if a client struggles to keep up or simply stops acking? > > > > > > > > > > I think the policy for backups should be no different as for the > > primaries. > > > As far as slow clients, Ignite is capable to automatically disconnect > > them: > > > http://s.apache.org/managing-slow-clients > > > > > > Isn't this feature getting Ignite into the murky waters of the message > > > > brokers and guaranteed once-only message delivery with all the > > complexity > > > > and overhead that come with it? Besides in some cases, it's doesn't > > really > > > > matter if some updates are missing, while in others it is only > > necessary to > > > > be able to detect a missing update. I wouldn't want to have to pay for > > > > something I don't need... > > > > > > > > > > I believe that the new proposed approach will be optional and you will > > > still be able to get event notifications in non-fault-tolerant manner the > > > old way. > > > > > > > > > > > > > > Thanks > > > > Andrey > > > > > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > > > Subject: Re: Continuous queries changes > > > > > From: [hidden email] > > > > > To: [hidden email] > > > > > > > > > > Andrey, > > > > > > > > > > I mean the queue of update events that is collected on backup nodes > > and > > > > > flushed to listening clients in case of topology change. > > > > > > > > > > -Val > > > > > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > Val, > > > > > > > > > > > > Could you please elaborate what you mean by "updates queue" you > > plan to > > > > > > maintain on the primary and backup nodes? > > > > > > > > > > > > Thanks > > > > > > Andrey > > > > > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > > > Subject: Re: Continuous queries changes > > > > > > > From: [hidden email] > > > > > > > To: [hidden email] > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on server > > > > node] > > > > > > > whether client received the update from local server or not. I > > think > > > > you > > > > > > > can easily change existing code to get notifications on ack > > receiving > > > > > > (this > > > > > > > way you dont need to introduce your own acks). > > > > > > > 2. How do you know when evict from backup? Each message that > > client > > > > acks > > > > > > > corresponds to some per-partition long value you talked above > > (great > > > > > > idea, > > > > > > > btw!). Servers can exchange per-partition long value that > > > > corresponds to > > > > > > > the latest acked message and that's the way how backups can > > safely > > > > evict > > > > > > > from the queue. > > > > > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > > > > > --Yakov > > > > > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > Based on discussions with our users I came to conclusion that > > our > > > > > > > > continuous query implementation is not good enough for use > > cases > > > > with > > > > > > > > strong consistency requirements, because there is a > > possibility to > > > > lose > > > > > > > > updates in case of topology change. > > > > > > > > > > > > > > > > So I started working on > > > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > > and I hope to finish in in couple of weeks so that we can > > include > > > > it in > > > > > > > > next release. > > > > > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition to > > > > primary > > > > > > node. > > > > > > > > - If primary node crushes, this queue is flushed to > > listening > > > > > > clients. > > > > > > > > - To avoid notification duplicates we will have a > > per-partition > > > > > > update > > > > > > > > counter. Once an entry in some partition is updated, > > counter for > > > > > > this > > > > > > > > partition is incremented on both primary and backups. The > > value > > > > of > > > > > > this > > > > > > > > counter is also sent along with the update to the client, > > which > > > > also > > > > > > > > maintains the copy of this mapping. If at some moment it > > > > receives an > > > > > > > > update > > > > > > > > with the counter less than in its local map, this update is > > a > > > > > > duplicate > > > > > > > > and > > > > > > > > can be discarded. > > > > > > > > - Also need to figure out the best way to clean the backup > > > > queue if > > > > > > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > > > > > > > > > > > To make all this work we also need to implement > > > > thread-per-partition > > > > > > mode > > > > > > > > in atomic cache, because now updates order on backup nodes can > > > > differ > > > > > > from > > > > > > > > the primary node: > > https://issues.apache.org/jira/browse/IGNITE-104 > > > > . > > > > > > I'm > > > > > > > > already working on this. > > > > > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > |
Re: Continuous queries changes
|
|
In reply to this post by Andrey Kornev
In my opinion the approach with the IgniteQueue does not work well: if the
events were pushed to the IgniteQueue asynchronously, the same problem with missing events would exist since node might crash in the window when update is already completed, but event is not yet added to the queue. If events were pushed to the IgniteQueue synchronously, it would mean that we add a huge performance hit to each update since queue.add() for this use-case would require transactional semantics. On the other hand, I do not see anything wrong with the backup queue - it should not add any performance drawbacks and the design suggested by Valentin/Yakov seems to guarantee the absence of duplicates. 2015-07-27 15:19 GMT-07:00 Andrey Kornev <[hidden email]>: > Hi Val, > > I was hoping to be able to use the blocking IgniteQueue.take() to achieve > the desirable "push" semantics. > > Andrey > > > Date: Mon, 27 Jul 2015 11:43:13 -0700 > > Subject: Re: Continuous queries changes > > From: [hidden email] > > To: [hidden email] > > > > Andrey, > > > > I think your approach works, but it requires periodical polling from > queue. > > Continuous queries provide an ability to get push notifications for > > updates, which from my experience is critical for some use cases. > > > > -Val > > > > On Mon, Jul 27, 2015 at 7:35 AM, Andrey Kornev <[hidden email] > > > > wrote: > > > > > I wonder if the same result (guaranteed delivery of CQ notifications) > can > > > be achieved entirely in the "user space" using the public Ignite API > only? > > > > > > For example: > > > - start a server-side CQ and have the listener push the notifications > into > > > an IgniteQueue. > > > - have the client connect to the queue and start receiving the > > > notifications. > > > > > > Regards > > > Andrey > > > > > > > From: [hidden email] > > > > Date: Sun, 26 Jul 2015 22:15:09 -0700 > > > > Subject: Re: Continuous queries changes > > > > To: [hidden email] > > > > > > > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev < > [hidden email] > > > > > > > > wrote: > > > > > > > > > Val, > > > > > > > > > > I'm sorry for being obtuse. :) > > > > > > > > > > I was simply wondering if the queue is going to be holding all > > > unfiltered > > > > > events per partition or will there be a queue per continuous query > > > instance > > > > > per partition? Or, is it going to be arranged some other way? > > > > > > > > > > > > > I believe that backup queues will have the same filters as primary > > > queues. > > > > > > > > > > > > > Also, in order to know when it's ok to remove an event from the > backup > > > > > queue, wouldn't this approach require maintaining a queue for each > > > > > connected client and having to deal with potentially unbounded > queue > > > > > growth if a client struggles to keep up or simply stops acking? > > > > > > > > > > > > > I think the policy for backups should be no different as for the > > > primaries. > > > > As far as slow clients, Ignite is capable to automatically disconnect > > > them: > > > > http://s.apache.org/managing-slow-clients > > > > > > > > Isn't this feature getting Ignite into the murky waters of the > message > > > > > brokers and guaranteed once-only message delivery with all the > > > complexity > > > > > and overhead that come with it? Besides in some cases, it's doesn't > > > really > > > > > matter if some updates are missing, while in others it is only > > > necessary to > > > > > be able to detect a missing update. I wouldn't want to have to pay > for > > > > > something I don't need... > > > > > > > > > > > > > I believe that the new proposed approach will be optional and you > will > > > > still be able to get event notifications in non-fault-tolerant > manner the > > > > old way. > > > > > > > > > > > > > > > > > > Thanks > > > > > Andrey > > > > > > > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > > > > Subject: Re: Continuous queries changes > > > > > > From: [hidden email] > > > > > > To: [hidden email] > > > > > > > > > > > > Andrey, > > > > > > > > > > > > I mean the queue of update events that is collected on backup > nodes > > > and > > > > > > flushed to listening clients in case of topology change. > > > > > > > > > > > > -Val > > > > > > > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > Could you please elaborate what you mean by "updates queue" you > > > plan to > > > > > > > maintain on the primary and backup nodes? > > > > > > > > > > > > > > Thanks > > > > > > > Andrey > > > > > > > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > > > > Subject: Re: Continuous queries changes > > > > > > > > From: [hidden email] > > > > > > > > To: [hidden email] > > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on > server > > > > > node] > > > > > > > > whether client received the update from local server or not. > I > > > think > > > > > you > > > > > > > > can easily change existing code to get notifications on ack > > > receiving > > > > > > > (this > > > > > > > > way you dont need to introduce your own acks). > > > > > > > > 2. How do you know when evict from backup? Each message that > > > client > > > > > acks > > > > > > > > corresponds to some per-partition long value you talked above > > > (great > > > > > > > idea, > > > > > > > > btw!). Servers can exchange per-partition long value that > > > > > corresponds to > > > > > > > > the latest acked message and that's the way how backups can > > > safely > > > > > evict > > > > > > > > from the queue. > > > > > > > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > > > > > > > --Yakov > > > > > > > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > Based on discussions with our users I came to conclusion > that > > > our > > > > > > > > > continuous query implementation is not good enough for use > > > cases > > > > > with > > > > > > > > > strong consistency requirements, because there is a > > > possibility to > > > > > lose > > > > > > > > > updates in case of topology change. > > > > > > > > > > > > > > > > > > So I started working on > > > > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > > > and I hope to finish in in couple of weeks so that we can > > > include > > > > > it in > > > > > > > > > next release. > > > > > > > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition > to > > > > > primary > > > > > > > node. > > > > > > > > > - If primary node crushes, this queue is flushed to > > > listening > > > > > > > clients. > > > > > > > > > - To avoid notification duplicates we will have a > > > per-partition > > > > > > > update > > > > > > > > > counter. Once an entry in some partition is updated, > > > counter for > > > > > > > this > > > > > > > > > partition is incremented on both primary and backups. > The > > > value > > > > > of > > > > > > > this > > > > > > > > > counter is also sent along with the update to the > client, > > > which > > > > > also > > > > > > > > > maintains the copy of this mapping. If at some moment it > > > > > receives an > > > > > > > > > update > > > > > > > > > with the counter less than in its local map, this > update is > > > a > > > > > > > duplicate > > > > > > > > > and > > > > > > > > > can be discarded. > > > > > > > > > - Also need to figure out the best way to clean the > backup > > > > > queue if > > > > > > > > > topology is stable. Will be happy to hear any > suggestions :) > > > > > > > > > > > > > > > > > > To make all this work we also need to implement > > > > > thread-per-partition > > > > > > > mode > > > > > > > > > in atomic cache, because now updates order on backup nodes > can > > > > > differ > > > > > > > from > > > > > > > > > the primary node: > > > https://issues.apache.org/jira/browse/IGNITE-104 > > > > > . > > > > > > > I'm > > > > > > > > > already working on this. > > > > > > > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
I thought transaction was a fair price to pay for such strong delivery guarantees. But if the originally proposed approach works without requiring anything that is just the 2PC protocol in disguise -- great!
It's not so much the performance hit that I was concerned about, but the amount of state that each node will have to maintain and non-triviality (in my opinion) of having the nodes agree on which queue items can be safely discarded. Thanks! Andrey > Date: Mon, 27 Jul 2015 16:02:49 -0700 > Subject: Re: Continuous queries changes > From: [hidden email] > To: [hidden email] > > In my opinion the approach with the IgniteQueue does not work well: if the > events were pushed to the IgniteQueue asynchronously, the same problem with > missing events would exist since node might crash in the window when update > is already completed, but event is not yet added to the queue. If events > were pushed to the IgniteQueue synchronously, it would mean that we add a > huge performance hit to each update since queue.add() for this use-case > would require transactional semantics. > > On the other hand, I do not see anything wrong with the backup queue - it > should not add any performance drawbacks and the design suggested by > Valentin/Yakov seems to guarantee the absence of duplicates. > > 2015-07-27 15:19 GMT-07:00 Andrey Kornev <[hidden email]>: > > > Hi Val, > > > > I was hoping to be able to use the blocking IgniteQueue.take() to achieve > > the desirable "push" semantics. > > > > Andrey > > > > > Date: Mon, 27 Jul 2015 11:43:13 -0700 > > > Subject: Re: Continuous queries changes > > > From: [hidden email] > > > To: [hidden email] > > > > > > Andrey, > > > > > > I think your approach works, but it requires periodical polling from > > queue. > > > Continuous queries provide an ability to get push notifications for > > > updates, which from my experience is critical for some use cases. > > > > > > -Val > > > > > > On Mon, Jul 27, 2015 at 7:35 AM, Andrey Kornev <[hidden email] > > > > > > wrote: > > > > > > > I wonder if the same result (guaranteed delivery of CQ notifications) > > can > > > > be achieved entirely in the "user space" using the public Ignite API > > only? > > > > > > > > For example: > > > > - start a server-side CQ and have the listener push the notifications > > into > > > > an IgniteQueue. > > > > - have the client connect to the queue and start receiving the > > > > notifications. > > > > > > > > Regards > > > > Andrey > > > > > > > > > From: [hidden email] > > > > > Date: Sun, 26 Jul 2015 22:15:09 -0700 > > > > > Subject: Re: Continuous queries changes > > > > > To: [hidden email] > > > > > > > > > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > Val, > > > > > > > > > > > > I'm sorry for being obtuse. :) > > > > > > > > > > > > I was simply wondering if the queue is going to be holding all > > > > unfiltered > > > > > > events per partition or will there be a queue per continuous query > > > > instance > > > > > > per partition? Or, is it going to be arranged some other way? > > > > > > > > > > > > > > > > I believe that backup queues will have the same filters as primary > > > > queues. > > > > > > > > > > > > > > > > Also, in order to know when it's ok to remove an event from the > > backup > > > > > > queue, wouldn't this approach require maintaining a queue for each > > > > > > connected client and having to deal with potentially unbounded > > queue > > > > > > growth if a client struggles to keep up or simply stops acking? > > > > > > > > > > > > > > > > I think the policy for backups should be no different as for the > > > > primaries. > > > > > As far as slow clients, Ignite is capable to automatically disconnect > > > > them: > > > > > http://s.apache.org/managing-slow-clients > > > > > > > > > > Isn't this feature getting Ignite into the murky waters of the > > message > > > > > > brokers and guaranteed once-only message delivery with all the > > > > complexity > > > > > > and overhead that come with it? Besides in some cases, it's doesn't > > > > really > > > > > > matter if some updates are missing, while in others it is only > > > > necessary to > > > > > > be able to detect a missing update. I wouldn't want to have to pay > > for > > > > > > something I don't need... > > > > > > > > > > > > > > > > I believe that the new proposed approach will be optional and you > > will > > > > > still be able to get event notifications in non-fault-tolerant > > manner the > > > > > old way. > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > Andrey > > > > > > > > > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > > > > > Subject: Re: Continuous queries changes > > > > > > > From: [hidden email] > > > > > > > To: [hidden email] > > > > > > > > > > > > > > Andrey, > > > > > > > > > > > > > > I mean the queue of update events that is collected on backup > > nodes > > > > and > > > > > > > flushed to listening clients in case of topology change. > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > Could you please elaborate what you mean by "updates queue" you > > > > plan to > > > > > > > > maintain on the primary and backup nodes? > > > > > > > > > > > > > > > > Thanks > > > > > > > > Andrey > > > > > > > > > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > > > > > Subject: Re: Continuous queries changes > > > > > > > > > From: [hidden email] > > > > > > > > > To: [hidden email] > > > > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on > > server > > > > > > node] > > > > > > > > > whether client received the update from local server or not. > > I > > > > think > > > > > > you > > > > > > > > > can easily change existing code to get notifications on ack > > > > receiving > > > > > > > > (this > > > > > > > > > way you dont need to introduce your own acks). > > > > > > > > > 2. How do you know when evict from backup? Each message that > > > > client > > > > > > acks > > > > > > > > > corresponds to some per-partition long value you talked above > > > > (great > > > > > > > > idea, > > > > > > > > > btw!). Servers can exchange per-partition long value that > > > > > > corresponds to > > > > > > > > > the latest acked message and that's the way how backups can > > > > safely > > > > > > evict > > > > > > > > > from the queue. > > > > > > > > > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > > > > > > > > > --Yakov > > > > > > > > > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > Based on discussions with our users I came to conclusion > > that > > > > our > > > > > > > > > > continuous query implementation is not good enough for use > > > > cases > > > > > > with > > > > > > > > > > strong consistency requirements, because there is a > > > > possibility to > > > > > > lose > > > > > > > > > > updates in case of topology change. > > > > > > > > > > > > > > > > > > > > So I started working on > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > > > > and I hope to finish in in couple of weeks so that we can > > > > include > > > > > > it in > > > > > > > > > > next release. > > > > > > > > > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition > > to > > > > > > primary > > > > > > > > node. > > > > > > > > > > - If primary node crushes, this queue is flushed to > > > > listening > > > > > > > > clients. > > > > > > > > > > - To avoid notification duplicates we will have a > > > > per-partition > > > > > > > > update > > > > > > > > > > counter. Once an entry in some partition is updated, > > > > counter for > > > > > > > > this > > > > > > > > > > partition is incremented on both primary and backups. > > The > > > > value > > > > > > of > > > > > > > > this > > > > > > > > > > counter is also sent along with the update to the > > client, > > > > which > > > > > > also > > > > > > > > > > maintains the copy of this mapping. If at some moment it > > > > > > receives an > > > > > > > > > > update > > > > > > > > > > with the counter less than in its local map, this > > update is > > > > a > > > > > > > > duplicate > > > > > > > > > > and > > > > > > > > > > can be discarded. > > > > > > > > > > - Also need to figure out the best way to clean the > > backup > > > > > > queue if > > > > > > > > > > topology is stable. Will be happy to hear any > > suggestions :) > > > > > > > > > > > > > > > > > > > > To make all this work we also need to implement > > > > > > thread-per-partition > > > > > > > > mode > > > > > > > > > > in atomic cache, because now updates order on backup nodes > > can > > > > > > differ > > > > > > > > from > > > > > > > > > > the primary node: > > > > https://issues.apache.org/jira/browse/IGNITE-104 > > > > > > . > > > > > > > > I'm > > > > > > > > > > already working on this. > > > > > > > > > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Re: Continuous queries changes
|
|
Agree with Alexey. Performance is always our first priority :)
On Mon, Jul 27, 2015 at 5:08 PM, Andrey Kornev <[hidden email]> wrote: > I thought transaction was a fair price to pay for such strong delivery > guarantees. But if the originally proposed approach works without requiring > anything that is just the 2PC protocol in disguise -- great! > > It's not so much the performance hit that I was concerned about, but the > amount of state that each node will have to maintain and non-triviality (in > my opinion) of having the nodes agree on which queue items can be safely > discarded. > > Thanks! > Andrey > > > Date: Mon, 27 Jul 2015 16:02:49 -0700 > > Subject: Re: Continuous queries changes > > From: [hidden email] > > To: [hidden email] > > > > In my opinion the approach with the IgniteQueue does not work well: if > the > > events were pushed to the IgniteQueue asynchronously, the same problem > with > > missing events would exist since node might crash in the window when > update > > is already completed, but event is not yet added to the queue. If events > > were pushed to the IgniteQueue synchronously, it would mean that we add a > > huge performance hit to each update since queue.add() for this use-case > > would require transactional semantics. > > > > On the other hand, I do not see anything wrong with the backup queue - it > > should not add any performance drawbacks and the design suggested by > > Valentin/Yakov seems to guarantee the absence of duplicates. > > > > 2015-07-27 15:19 GMT-07:00 Andrey Kornev <[hidden email]>: > > > > > Hi Val, > > > > > > I was hoping to be able to use the blocking IgniteQueue.take() to > achieve > > > the desirable "push" semantics. > > > > > > Andrey > > > > > > > Date: Mon, 27 Jul 2015 11:43:13 -0700 > > > > Subject: Re: Continuous queries changes > > > > From: [hidden email] > > > > To: [hidden email] > > > > > > > > Andrey, > > > > > > > > I think your approach works, but it requires periodical polling from > > > queue. > > > > Continuous queries provide an ability to get push notifications for > > > > updates, which from my experience is critical for some use cases. > > > > > > > > -Val > > > > > > > > On Mon, Jul 27, 2015 at 7:35 AM, Andrey Kornev < > [hidden email] > > > > > > > > wrote: > > > > > > > > > I wonder if the same result (guaranteed delivery of CQ > notifications) > > > can > > > > > be achieved entirely in the "user space" using the public Ignite > API > > > only? > > > > > > > > > > For example: > > > > > - start a server-side CQ and have the listener push the > notifications > > > into > > > > > an IgniteQueue. > > > > > - have the client connect to the queue and start receiving the > > > > > notifications. > > > > > > > > > > Regards > > > > > Andrey > > > > > > > > > > > From: [hidden email] > > > > > > Date: Sun, 26 Jul 2015 22:15:09 -0700 > > > > > > Subject: Re: Continuous queries changes > > > > > > To: [hidden email] > > > > > > > > > > > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > I'm sorry for being obtuse. :) > > > > > > > > > > > > > > I was simply wondering if the queue is going to be holding all > > > > > unfiltered > > > > > > > events per partition or will there be a queue per continuous > query > > > > > instance > > > > > > > per partition? Or, is it going to be arranged some other way? > > > > > > > > > > > > > > > > > > > I believe that backup queues will have the same filters as > primary > > > > > queues. > > > > > > > > > > > > > > > > > > > Also, in order to know when it's ok to remove an event from the > > > backup > > > > > > > queue, wouldn't this approach require maintaining a queue for > each > > > > > > > connected client and having to deal with potentially unbounded > > > queue > > > > > > > growth if a client struggles to keep up or simply stops acking? > > > > > > > > > > > > > > > > > > > I think the policy for backups should be no different as for the > > > > > primaries. > > > > > > As far as slow clients, Ignite is capable to automatically > disconnect > > > > > them: > > > > > > http://s.apache.org/managing-slow-clients > > > > > > > > > > > > Isn't this feature getting Ignite into the murky waters of the > > > message > > > > > > > brokers and guaranteed once-only message delivery with all the > > > > > complexity > > > > > > > and overhead that come with it? Besides in some cases, it's > doesn't > > > > > really > > > > > > > matter if some updates are missing, while in others it is only > > > > > necessary to > > > > > > > be able to detect a missing update. I wouldn't want to have to > pay > > > for > > > > > > > something I don't need... > > > > > > > > > > > > > > > > > > > I believe that the new proposed approach will be optional and you > > > will > > > > > > still be able to get event notifications in non-fault-tolerant > > > manner the > > > > > > old way. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > Andrey > > > > > > > > > > > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > > > > > > Subject: Re: Continuous queries changes > > > > > > > > From: [hidden email] > > > > > > > > To: [hidden email] > > > > > > > > > > > > > > > > Andrey, > > > > > > > > > > > > > > > > I mean the queue of update events that is collected on backup > > > nodes > > > > > and > > > > > > > > flushed to listening clients in case of topology change. > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > > > Could you please elaborate what you mean by "updates > queue" you > > > > > plan to > > > > > > > > > maintain on the primary and backup nodes? > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > > > > > > Subject: Re: Continuous queries changes > > > > > > > > > > From: [hidden email] > > > > > > > > > > To: [hidden email] > > > > > > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on > > > server > > > > > > > node] > > > > > > > > > > whether client received the update from local server or > not. > > > I > > > > > think > > > > > > > you > > > > > > > > > > can easily change existing code to get notifications on > ack > > > > > receiving > > > > > > > > > (this > > > > > > > > > > way you dont need to introduce your own acks). > > > > > > > > > > 2. How do you know when evict from backup? Each message > that > > > > > client > > > > > > > acks > > > > > > > > > > corresponds to some per-partition long value you talked > above > > > > > (great > > > > > > > > > idea, > > > > > > > > > > btw!). Servers can exchange per-partition long value that > > > > > > > corresponds to > > > > > > > > > > the latest acked message and that's the way how backups > can > > > > > safely > > > > > > > evict > > > > > > > > > > from the queue. > > > > > > > > > > > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > > > > > > > > > > > --Yakov > > > > > > > > > > > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > > > Based on discussions with our users I came to > conclusion > > > that > > > > > our > > > > > > > > > > > continuous query implementation is not good enough for > use > > > > > cases > > > > > > > with > > > > > > > > > > > strong consistency requirements, because there is a > > > > > possibility to > > > > > > > lose > > > > > > > > > > > updates in case of topology change. > > > > > > > > > > > > > > > > > > > > > > So I started working on > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > > > > > and I hope to finish in in couple of weeks so that we > can > > > > > include > > > > > > > it in > > > > > > > > > > > next release. > > > > > > > > > > > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > > > > > > > > > > > - Maintain updates queue on backup node(s) in > addition > > > to > > > > > > > primary > > > > > > > > > node. > > > > > > > > > > > - If primary node crushes, this queue is flushed to > > > > > listening > > > > > > > > > clients. > > > > > > > > > > > - To avoid notification duplicates we will have a > > > > > per-partition > > > > > > > > > update > > > > > > > > > > > counter. Once an entry in some partition is updated, > > > > > counter for > > > > > > > > > this > > > > > > > > > > > partition is incremented on both primary and > backups. > > > The > > > > > value > > > > > > > of > > > > > > > > > this > > > > > > > > > > > counter is also sent along with the update to the > > > client, > > > > > which > > > > > > > also > > > > > > > > > > > maintains the copy of this mapping. If at some > moment it > > > > > > > receives an > > > > > > > > > > > update > > > > > > > > > > > with the counter less than in its local map, this > > > update is > > > > > a > > > > > > > > > duplicate > > > > > > > > > > > and > > > > > > > > > > > can be discarded. > > > > > > > > > > > - Also need to figure out the best way to clean the > > > backup > > > > > > > queue if > > > > > > > > > > > topology is stable. Will be happy to hear any > > > suggestions :) > > > > > > > > > > > > > > > > > > > > > > To make all this work we also need to implement > > > > > > > thread-per-partition > > > > > > > > > mode > > > > > > > > > > > in atomic cache, because now updates order on backup > nodes > > > can > > > > > > > differ > > > > > > > > > from > > > > > > > > > > > the primary node: > > > > > https://issues.apache.org/jira/browse/IGNITE-104 > > > > > > > . > > > > > > > > > I'm > > > > > > > > > > > already working on this. > > > > > > > > > > > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
In reply to this post by Andrey Kornev
On Mon, Jul 27, 2015 at 3:56 PM, Andrey Kornev <[hidden email]>
wrote: > Dmitriy, > > What I had in mind was to have each client create a dedicated (replicated) > queue for each CQ it's about to start. Once started, the server-side CQ > listener would then simply push the notifications onto the queue. With such > setup, the CQ notifications should be able to survive crashes of the server > nodes as well as the client node reconnects. It would also allow the users > handle the queue growth in a more graceful (or application specific) way > than a client disconnect. > Andrey, I don't think this approach would scale. I think the most scalable approach is that each key notification should be backed on the node responsible for storing a backup copy for this key. Makes sense? > > Andrey > > > From: [hidden email] > > Date: Mon, 27 Jul 2015 07:47:39 -0700 > > Subject: Re: Continuous queries changes > > To: [hidden email] > > > > On Mon, Jul 27, 2015 at 7:35 AM, Andrey Kornev <[hidden email] > > > > wrote: > > > > > I wonder if the same result (guaranteed delivery of CQ notifications) > can > > > be achieved entirely in the "user space" using the public Ignite API > only? > > > > > > For example: > > > - start a server-side CQ and have the listener push the notifications > into > > > an IgniteQueue. > > > - have the client connect to the queue and start receiving the > > > notifications. > > > > > > > Hm... Do you mean that in this approach we will have 1 CQ queue per > server, > > instead of 1 queue per subscription, as we planned before? > > > > > > > > > > Regards > > > Andrey > > > > > > > From: [hidden email] > > > > Date: Sun, 26 Jul 2015 22:15:09 -0700 > > > > Subject: Re: Continuous queries changes > > > > To: [hidden email] > > > > > > > > On Sat, Jul 25, 2015 at 8:07 AM, Andrey Kornev < > [hidden email] > > > > > > > > wrote: > > > > > > > > > Val, > > > > > > > > > > I'm sorry for being obtuse. :) > > > > > > > > > > I was simply wondering if the queue is going to be holding all > > > unfiltered > > > > > events per partition or will there be a queue per continuous query > > > instance > > > > > per partition? Or, is it going to be arranged some other way? > > > > > > > > > > > > > I believe that backup queues will have the same filters as primary > > > queues. > > > > > > > > > > > > > Also, in order to know when it's ok to remove an event from the > backup > > > > > queue, wouldn't this approach require maintaining a queue for each > > > > > connected client and having to deal with potentially unbounded > queue > > > > > growth if a client struggles to keep up or simply stops acking? > > > > > > > > > > > > > I think the policy for backups should be no different as for the > > > primaries. > > > > As far as slow clients, Ignite is capable to automatically disconnect > > > them: > > > > http://s.apache.org/managing-slow-clients > > > > > > > > Isn't this feature getting Ignite into the murky waters of the > message > > > > > brokers and guaranteed once-only message delivery with all the > > > complexity > > > > > and overhead that come with it? Besides in some cases, it's doesn't > > > really > > > > > matter if some updates are missing, while in others it is only > > > necessary to > > > > > be able to detect a missing update. I wouldn't want to have to pay > for > > > > > something I don't need... > > > > > > > > > > > > > I believe that the new proposed approach will be optional and you > will > > > > still be able to get event notifications in non-fault-tolerant > manner the > > > > old way. > > > > > > > > > > > > > > > > > > Thanks > > > > > Andrey > > > > > > > > > > > Date: Fri, 24 Jul 2015 23:40:15 -0700 > > > > > > Subject: Re: Continuous queries changes > > > > > > From: [hidden email] > > > > > > To: [hidden email] > > > > > > > > > > > > Andrey, > > > > > > > > > > > > I mean the queue of update events that is collected on backup > nodes > > > and > > > > > > flushed to listening clients in case of topology change. > > > > > > > > > > > > -Val > > > > > > > > > > > > On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > Could you please elaborate what you mean by "updates queue" you > > > plan to > > > > > > > maintain on the primary and backup nodes? > > > > > > > > > > > > > > Thanks > > > > > > > Andrey > > > > > > > > > > > > > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > > > > > > > Subject: Re: Continuous queries changes > > > > > > > > From: [hidden email] > > > > > > > > To: [hidden email] > > > > > > > > > > > > > > > > Val, > > > > > > > > > > > > > > > > I have idea on how to clean up backup queue. > > > > > > > > > > > > > > > > 1. Our communication uses acks. So, you can determine [on > server > > > > > node] > > > > > > > > whether client received the update from local server or not. > I > > > think > > > > > you > > > > > > > > can easily change existing code to get notifications on ack > > > receiving > > > > > > > (this > > > > > > > > way you dont need to introduce your own acks). > > > > > > > > 2. How do you know when evict from backup? Each message that > > > client > > > > > acks > > > > > > > > corresponds to some per-partition long value you talked above > > > (great > > > > > > > idea, > > > > > > > > btw!). Servers can exchange per-partition long value that > > > > > corresponds to > > > > > > > > the latest acked message and that's the way how backups can > > > safely > > > > > evict > > > > > > > > from the queue. > > > > > > > > > > > > > > > > Let me know if you have questions. > > > > > > > > > > > > > > > > --Yakov > > > > > > > > > > > > > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > Based on discussions with our users I came to conclusion > that > > > our > > > > > > > > > continuous query implementation is not good enough for use > > > cases > > > > > with > > > > > > > > > strong consistency requirements, because there is a > > > possibility to > > > > > lose > > > > > > > > > updates in case of topology change. > > > > > > > > > > > > > > > > > > So I started working on > > > > > > > https://issues.apache.org/jira/browse/IGNITE-426 > > > > > > > > > and I hope to finish in in couple of weeks so that we can > > > include > > > > > it in > > > > > > > > > next release. > > > > > > > > > > > > > > > > > > I have the following design in mind: > > > > > > > > > > > > > > > > > > - Maintain updates queue on backup node(s) in addition > to > > > > > primary > > > > > > > node. > > > > > > > > > - If primary node crushes, this queue is flushed to > > > listening > > > > > > > clients. > > > > > > > > > - To avoid notification duplicates we will have a > > > per-partition > > > > > > > update > > > > > > > > > counter. Once an entry in some partition is updated, > > > counter for > > > > > > > this > > > > > > > > > partition is incremented on both primary and backups. > The > > > value > > > > > of > > > > > > > this > > > > > > > > > counter is also sent along with the update to the > client, > > > which > > > > > also > > > > > > > > > maintains the copy of this mapping. If at some moment it > > > > > receives an > > > > > > > > > update > > > > > > > > > with the counter less than in its local map, this > update is > > > a > > > > > > > duplicate > > > > > > > > > and > > > > > > > > > can be discarded. > > > > > > > > > - Also need to figure out the best way to clean the > backup > > > > > queue if > > > > > > > > > topology is stable. Will be happy to hear any > suggestions :) > > > > > > > > > > > > > > > > > > To make all this work we also need to implement > > > > > thread-per-partition > > > > > > > mode > > > > > > > > > in atomic cache, because now updates order on backup nodes > can > > > > > differ > > > > > > > from > > > > > > > > > the primary node: > > > https://issues.apache.org/jira/browse/IGNITE-104 > > > > > . > > > > > > > I'm > > > > > > > > > already working on this. > > > > > > > > > > > > > > > > > > Feel free to share your thoughts! > > > > > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
In reply to this post by Valentin Kulichenko
Hi,
I started to work on ignite-426 and see that current implementation of thread-per-partition approach required for this feature has serious impact on performance. As I understand thread-per-partition is required to avoid issue with update order on backup nodes: now it is possible that on primary node updates were executed 'update 1', 'update 2' and on backup nodes update messages can be handled like <update 2>, <update 1>. But now backup will eventually have correct value due to the fact that cache entry version grows, so 'update 2' will always have version greater than 'update 1'. What if on backup we will sort continuous query events by cache update version? In this case when backup queue is flushed messages will be correctly ordered and thread-per-partition mode is not needed. Do you see any problems with this approach, does it worth to try this? Thanks On Thu, Jul 23, 2015 at 8:53 AM, Valentin Kulichenko < [hidden email]> wrote: > Igniters, > > Based on discussions with our users I came to conclusion that our > continuous query implementation is not good enough for use cases with > strong consistency requirements, because there is a possibility to lose > updates in case of topology change. > > So I started working on https://issues.apache.org/jira/browse/IGNITE-426 > and I hope to finish in in couple of weeks so that we can include it in > next release. > > I have the following design in mind: > > - Maintain updates queue on backup node(s) in addition to primary node. > - If primary node crushes, this queue is flushed to listening clients. > - To avoid notification duplicates we will have a per-partition update > counter. Once an entry in some partition is updated, counter for this > partition is incremented on both primary and backups. The value of this > counter is also sent along with the update to the client, which also > maintains the copy of this mapping. If at some moment it receives an > update > with the counter less than in its local map, this update is a duplicate > and > can be discarded. > - Also need to figure out the best way to clean the backup queue if > topology is stable. Will be happy to hear any suggestions :) > > To make all this work we also need to implement thread-per-partition mode > in atomic cache, because now updates order on backup nodes can differ from > the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm > already working on this. > > Feel free to share your thoughts! > > -Val > |
|
|
After more thinking about ordering on backups instead of
thread-per-partition I believe this approach won't work. First it is possible that primary executes for some key <update 1>, <update 2> (from different threads), then crashes and only <update 1> message is sent to backup, so backup won't be able to restore correct ordering. Also if the same partition can be updated concurrently it is not possible to associate per-partition update counter with each update so it is not clear how to deal with duplicated messages and backup acknowledgement. But since there is no thread-per-partition mode for transactional cache do these issues affect transactional cache as well? On Wed, Aug 12, 2015 at 4:49 PM, Semyon Boikov <[hidden email]> wrote: > Hi, > > I started to work on ignite-426 and see that current implementation of > thread-per-partition approach required for this feature has serious impact > on performance. > > As I understand thread-per-partition is required to avoid issue with > update order on backup nodes: now it is possible that on primary node > updates were executed 'update 1', 'update 2' and on backup nodes update > messages can be handled like <update 2>, <update 1>. > > But now backup will eventually have correct value due to the fact that > cache entry version grows, so 'update 2' will always have version greater > than 'update 1'. > > What if on backup we will sort continuous query events by cache update > version? In this case when backup queue is flushed messages will be > correctly ordered and thread-per-partition mode is not needed. > > Do you see any problems with this approach, does it worth to try this? > > Thanks > > > On Thu, Jul 23, 2015 at 8:53 AM, Valentin Kulichenko < > [hidden email]> wrote: > >> Igniters, >> >> Based on discussions with our users I came to conclusion that our >> continuous query implementation is not good enough for use cases with >> strong consistency requirements, because there is a possibility to lose >> updates in case of topology change. >> >> So I started working on https://issues.apache.org/jira/browse/IGNITE-426 >> and I hope to finish in in couple of weeks so that we can include it in >> next release. >> >> I have the following design in mind: >> >> - Maintain updates queue on backup node(s) in addition to primary node. >> - If primary node crushes, this queue is flushed to listening clients. >> - To avoid notification duplicates we will have a per-partition update >> counter. Once an entry in some partition is updated, counter for this >> partition is incremented on both primary and backups. The value of this >> counter is also sent along with the update to the client, which also >> maintains the copy of this mapping. If at some moment it receives an >> update >> with the counter less than in its local map, this update is a >> duplicate and >> can be discarded. >> - Also need to figure out the best way to clean the backup queue if >> topology is stable. Will be happy to hear any suggestions :) >> >> To make all this work we also need to implement thread-per-partition mode >> in atomic cache, because now updates order on backup nodes can differ from >> the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm >> already working on this. >> >> Feel free to share your thoughts! >> >> -Val >> > > |
Re: Continuous queries changes
|
|
Guys,
I added preliminary version of API for continuous queries to this ticket: https://issues.apache.org/jira/browse/IGNITE-1191 I would like to ask everyone interested in this topic to look at it and provide comments. -Val On Thu, Aug 13, 2015 at 4:48 AM, Semyon Boikov <[hidden email]> wrote: > After more thinking about ordering on backups instead of > thread-per-partition I believe this approach won't work. > > First it is possible that primary executes for some key <update 1>, <update > 2> (from different threads), then crashes and only <update 1> message is > sent to backup, so backup won't be able to restore correct ordering. > > Also if the same partition can be updated concurrently it is not possible > to associate per-partition update counter with each update so it is not > clear how to deal with duplicated messages and backup acknowledgement. > > But since there is no thread-per-partition mode for transactional cache do > these issues affect transactional cache as well? > > > On Wed, Aug 12, 2015 at 4:49 PM, Semyon Boikov <[hidden email]> > wrote: > > > Hi, > > > > I started to work on ignite-426 and see that current implementation of > > thread-per-partition approach required for this feature has serious > impact > > on performance. > > > > As I understand thread-per-partition is required to avoid issue with > > update order on backup nodes: now it is possible that on primary node > > updates were executed 'update 1', 'update 2' and on backup nodes update > > messages can be handled like <update 2>, <update 1>. > > > > But now backup will eventually have correct value due to the fact that > > cache entry version grows, so 'update 2' will always have version greater > > than 'update 1'. > > > > What if on backup we will sort continuous query events by cache update > > version? In this case when backup queue is flushed messages will be > > correctly ordered and thread-per-partition mode is not needed. > > > > Do you see any problems with this approach, does it worth to try this? > > > > Thanks > > > > > > On Thu, Jul 23, 2015 at 8:53 AM, Valentin Kulichenko < > > [hidden email]> wrote: > > > >> Igniters, > >> > >> Based on discussions with our users I came to conclusion that our > >> continuous query implementation is not good enough for use cases with > >> strong consistency requirements, because there is a possibility to lose > >> updates in case of topology change. > >> > >> So I started working on > https://issues.apache.org/jira/browse/IGNITE-426 > >> and I hope to finish in in couple of weeks so that we can include it in > >> next release. > >> > >> I have the following design in mind: > >> > >> - Maintain updates queue on backup node(s) in addition to primary > node. > >> - If primary node crushes, this queue is flushed to listening > clients. > >> - To avoid notification duplicates we will have a per-partition > update > >> counter. Once an entry in some partition is updated, counter for this > >> partition is incremented on both primary and backups. The value of > this > >> counter is also sent along with the update to the client, which also > >> maintains the copy of this mapping. If at some moment it receives an > >> update > >> with the counter less than in its local map, this update is a > >> duplicate and > >> can be discarded. > >> - Also need to figure out the best way to clean the backup queue if > >> topology is stable. Will be happy to hear any suggestions :) > >> > >> To make all this work we also need to implement thread-per-partition > mode > >> in atomic cache, because now updates order on backup nodes can differ > from > >> the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm > >> already working on this. > >> > >> Feel free to share your thoughts! > >> > >> -Val > >> > > > > > |
«
Return to Apache Ignite Developers - Legacy Mail Archive
|
1 view|%1 views
| Free forum by Nabble | Edit this page |