Continuous queries and MVCC

[Vladimir Ozerov]

Folks,

As you know we are developing multi-version concurrency control for Ignite
caches, which is essentially new mode for transactional cache with snapshot
semantics. One of the most important applications of this mode would be
fully transactional SQL The question is how to implement continuous query
semantics with MVCC. Several interesting things here.

1) *Event ordering*. Currently we guarantee order of updates for specific
key. But for fair transactional mode user may want to get transactional
ordering instead of key ordering. If there are several dependent
transactions, I may want to receive their updates in order. E.g.:
TX1: transfer(A -> B)
TX2: transfer(B -> C)
TX3: transfer(C -> D)

If user receives update on key D (TX3), should we guarantee that he already
received updates for all keys in TX1 and TX2? My opinion is that we'd
better to leave things as is and guarantee only per-key ordering. Only one
reason - implementation complexity.

2) *Initial query*. We implemented it so that user can get some initial
data snapshot and then start receiving events. Without MVCC we have no
guarantees of visibility. E.g. if key is updated from V1 to V2, it is
possible to see V2 in initial query and in event. With MVCC it is now
technically possible to query data on certain snapshot and then receive
only events happened after this snapshot. So that we never see V2 twice. Do
you think we this feature will be interesting for our users?

Please share your thoughts.

Vladimir.

[Nikolay Izhikov-2]

Hello, Vladimir.

> 1) *Event ordering*.

I vote to keep things as is.

> 2) *Initial query*
> With MVCC it is now
> technically possible to query data on certain snapshot and then receive
> only events happened after this snapshot

I think it a usefull extension for current implementation.

В Пт, 17/08/2018 в 13:18 +0300, Vladimir Ozerov пишет:

> Folks,
>
> As you know we are developing multi-version concurrency control for Ignite
> caches, which is essentially new mode for transactional cache with snapshot
> semantics. One of the most important applications of this mode would be
> fully transactional SQL The question is how to implement continuous query
> semantics with MVCC. Several interesting things here.
>
> 1) *Event ordering*. Currently we guarantee order of updates for specific
> key. But for fair transactional mode user may want to get transactional
> ordering instead of key ordering. If there are several dependent
> transactions, I may want to receive their updates in order. E.g.:
> TX1: transfer(A -> B)
> TX2: transfer(B -> C)
> TX3: transfer(C -> D)
>
> If user receives update on key D (TX3), should we guarantee that he already
> received updates for all keys in TX1 and TX2? My opinion is that we'd
> better to leave things as is and guarantee only per-key ordering. Only one
> reason - implementation complexity.
>
> 2) *Initial query*. We implemented it so that user can get some initial
> data snapshot and then start receiving events. Without MVCC we have no
> guarantees of visibility. E.g. if key is updated from V1 to V2, it is
> possible to see V2 in initial query and in event. With MVCC it is now
> technically possible to query data on certain snapshot and then receive
> only events happened after this snapshot. So that we never see V2 twice. Do
> you think we this feature will be interesting for our users?
>
> Please share your thoughts.
>
> Vladimir.

[Valentin Kulichenko]

Vladimir,

1. Continuous queries are asynchronous in general case, so I don't think
it's even possible to provide transactional ordering, especially for the
case of distributed transactions. I would leave current guarantees as-is.

2. This one might be pretty useful. If it's not very hard to do, let's do
this improvement.

-Val

On Fri, Aug 17, 2018 at 7:10 AM Nikolay Izhikov <[hidden email]> wrote:

> Hello, Vladimir.
>
> > 1) *Event ordering*.
>
> I vote to keep things as is.
>
> > 2) *Initial query*
> > With MVCC it is now
> > technically possible to query data on certain snapshot and then receive
> > only events happened after this snapshot
>
> I think it a usefull extension for current implementation.
>
> В Пт, 17/08/2018 в 13:18 +0300, Vladimir Ozerov пишет:
> > Folks,
> >
> > As you know we are developing multi-version concurrency control for
> Ignite
> > caches, which is essentially new mode for transactional cache with
> snapshot
> > semantics. One of the most important applications of this mode would be
> > fully transactional SQL The question is how to implement continuous query
> > semantics with MVCC. Several interesting things here.
> >
> > 1) *Event ordering*. Currently we guarantee order of updates for specific
> > key. But for fair transactional mode user may want to get transactional
> > ordering instead of key ordering. If there are several dependent
> > transactions, I may want to receive their updates in order. E.g.:
> > TX1: transfer(A -> B)
> > TX2: transfer(B -> C)
> > TX3: transfer(C -> D)
> >
> > If user receives update on key D (TX3), should we guarantee that he
> already
> > received updates for all keys in TX1 and TX2? My opinion is that we'd
> > better to leave things as is and guarantee only per-key ordering. Only
> one
> > reason - implementation complexity.
> >
> > 2) *Initial query*. We implemented it so that user can get some initial
> > data snapshot and then start receiving events. Without MVCC we have no
> > guarantees of visibility. E.g. if key is updated from V1 to V2, it is
> > possible to see V2 in initial query and in event. With MVCC it is now
> > technically possible to query data on certain snapshot and then receive
> > only events happened after this snapshot. So that we never see V2 twice.
> Do
> > you think we this feature will be interesting for our users?
> >
> > Please share your thoughts.
> >
> > Vladimir.

[dsetrakyan]

On Fri, Aug 17, 2018 at 3:18 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Folks,
>
> As you know we are developing multi-version concurrency control for Ignite
> caches, which is essentially new mode for transactional cache with snapshot
> semantics. One of the most important applications of this mode would be
> fully transactional SQL The question is how to implement continuous query
> semantics with MVCC. Several interesting things here.
>
> 1) *Event ordering*. Currently we guarantee order of updates for specific
> key. But for fair transactional mode user may want to get transactional
> ordering instead of key ordering. If there are several dependent
> transactions, I may want to receive their updates in order. E.g.:
> TX1: transfer(A -> B)
> TX2: transfer(B -> C)
> TX3: transfer(C -> D)
>
> If user receives update on key D (TX3), should we guarantee that he already
> received updates for all keys in TX1 and TX2? My opinion is that we'd
> better to leave things as is and guarantee only per-key ordering. Only one
> reason - implementation complexity.
>

I would preserve at least the current guarantees. However, many users
expressed interested in receiving events in proper order. If it is possible
to do, I would definitely do it, or at lease provide a configuration option
to have such behavior.


>
> 2) *Initial query*. We implemented it so that user can get some initial
> data snapshot and then start receiving events. Without MVCC we have no
> guarantees of visibility. E.g. if key is updated from V1 to V2, it is
> possible to see V2 in initial query and in event. With MVCC it is now
> technically possible to query data on certain snapshot and then receive
> only events happened after this snapshot. So that we never see V2 twice. Do
> you think we this feature will be interesting for our users?
>

Absolutely! It will CQ usage much cleaner and less error-prone.