PRIMARY_SYNC+readFromBackup semantics

[Valentin Kulichenko]

Igniters,

I noticed that combination of PRIMARY_SYNC mode and readFromBackup=true
(both are default values BTW) introduces weird semantics when reading *on a
backup node*. Basically, if I do put and then get for the same key in the
same thread, I can get previous value. In my understanding, this happens
because even local backup is updated asynchronously in this case.

First of all, this is obviously confusing and would be considered as a bug
by most of the users (I just updated the key with some value, why would I
get another value when reading it?).

Second of all, it seems that we send a network message from primary node to
local backup, which doesn't make much sense to me and looks like
unnecessary performance overhead.

Is it possible to update local backup synchronously in this scenario?

-Val

[V.Pyatkov]

Hi Val,

If we update local backup immediate synchronously when sending commit to
primary, this only partly removes questions about consistence view.
But we always can to get other (unpredictable) value, because another
transaction will be executed simultaneously from other threads.

At the same time this is good place for optimization, probably reduce
network overhead.
I think, need to create a ticket in Jira for the improvement.


On Tue, Oct 3, 2017 at 12:27 AM, Valentin Kulichenko <
[hidden email]> wrote:

> Igniters,
>
> I noticed that combination of PRIMARY_SYNC mode and readFromBackup=true
> (both are default values BTW) introduces weird semantics when reading *on a
> backup node*. Basically, if I do put and then get for the same key in the
> same thread, I can get previous value. In my understanding, this happens
> because even local backup is updated asynchronously in this case.
>
> First of all, this is obviously confusing and would be considered as a bug
> by most of the users (I just updated the key with some value, why would I
> get another value when reading it?).
>
> Second of all, it seems that we send a network message from primary node to
> local backup, which doesn't make much sense to me and looks like
> unnecessary performance overhead.
>
> Is it possible to update local backup synchronously in this scenario?
>
> -Val
>

--
Vladislav Pyatkov

[Alexei Scherbakov]

Val,

Totally agreed.

I think this can be easily fixed by setting FULL_SYNC as default sync mode.

Actually, I do not understand why this is not done yet.

2017-10-04 12:09 GMT+03:00 Vladislav Pyatkov <[hidden email]>:

> Hi Val,
>
> If we update local backup immediate synchronously when sending commit to
> primary, this only partly removes questions about consistence view.
> But we always can to get other (unpredictable) value, because another
> transaction will be executed simultaneously from other threads.
>
> At the same time this is good place for optimization, probably reduce
> network overhead.
> I think, need to create a ticket in Jira for the improvement.
>
>
> On Tue, Oct 3, 2017 at 12:27 AM, Valentin Kulichenko <
> [hidden email]> wrote:
>
> > Igniters,
> >
> > I noticed that combination of PRIMARY_SYNC mode and readFromBackup=true
> > (both are default values BTW) introduces weird semantics when reading
> *on a
> > backup node*. Basically, if I do put and then get for the same key in the
> > same thread, I can get previous value. In my understanding, this happens
> > because even local backup is updated asynchronously in this case.
> >
> > First of all, this is obviously confusing and would be considered as a
> bug
> > by most of the users (I just updated the key with some value, why would I
> > get another value when reading it?).
> >
> > Second of all, it seems that we send a network message from primary node
> to
> > local backup, which doesn't make much sense to me and looks like
> > unnecessary performance overhead.
> >
> > Is it possible to update local backup synchronously in this scenario?
> >
> > -Val
> >
>
>
>
> --
> Vladislav Pyatkov
>

--

Best regards,
Alexei Scherbakov

[Valentin Kulichenko]

FULL_SYNC as default is not a solution. PRIMARY_SYNC/readFromBackup is
valid combination of parameters and useful in many scenarios. For example,
what if I have a replicated cache and don't want to update all nodes
synchronously?

The point is that get() invoked after the put() in the same thread should
always return the latest value. Now this behavior is not consistent and
depends on which nodes these operations are executed on.

-Val

On Wed, Oct 4, 2017 at 3:27 AM, Alexei Scherbakov <
[hidden email]> wrote:

> Val,
>
> Totally agreed.
>
> I think this can be easily fixed by setting FULL_SYNC as default sync mode.
>
> Actually, I do not understand why this is not done yet.
>
> 2017-10-04 12:09 GMT+03:00 Vladislav Pyatkov <[hidden email]>:
>
> > Hi Val,
> >
> > If we update local backup immediate synchronously when sending commit to
> > primary, this only partly removes questions about consistence view.
> > But we always can to get other (unpredictable) value, because another
> > transaction will be executed simultaneously from other threads.
> >
> > At the same time this is good place for optimization, probably reduce
> > network overhead.
> > I think, need to create a ticket in Jira for the improvement.
> >
> >
> > On Tue, Oct 3, 2017 at 12:27 AM, Valentin Kulichenko <
> > [hidden email]> wrote:
> >
> > > Igniters,
> > >
> > > I noticed that combination of PRIMARY_SYNC mode and readFromBackup=true
> > > (both are default values BTW) introduces weird semantics when reading
> > *on a
> > > backup node*. Basically, if I do put and then get for the same key in
> the
> > > same thread, I can get previous value. In my understanding, this
> happens
> > > because even local backup is updated asynchronously in this case.
> > >
> > > First of all, this is obviously confusing and would be considered as a
> > bug
> > > by most of the users (I just updated the key with some value, why
> would I
> > > get another value when reading it?).
> > >
> > > Second of all, it seems that we send a network message from primary
> node
> > to
> > > local backup, which doesn't make much sense to me and looks like
> > > unnecessary performance overhead.
> > >
> > > Is it possible to update local backup synchronously in this scenario?
> > >
> > > -Val
> > >
> >
> >
> >
> > --
> > Vladislav Pyatkov
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

[dsetrakyan]

Val, do you have a fix in mind?

On Thu, Oct 5, 2017 at 2:50 AM, Valentin Kulichenko <
[hidden email]> wrote:

> FULL_SYNC as default is not a solution. PRIMARY_SYNC/readFromBackup is
> valid combination of parameters and useful in many scenarios. For example,
> what if I have a replicated cache and don't want to update all nodes
> synchronously?
>
> The point is that get() invoked after the put() in the same thread should
> always return the latest value. Now this behavior is not consistent and
> depends on which nodes these operations are executed on.
>
> -Val
>
> On Wed, Oct 4, 2017 at 3:27 AM, Alexei Scherbakov <
> [hidden email]> wrote:
>
> > Val,
> >
> > Totally agreed.
> >
> > I think this can be easily fixed by setting FULL_SYNC as default sync
> mode.
> >
> > Actually, I do not understand why this is not done yet.
> >
> > 2017-10-04 12:09 GMT+03:00 Vladislav Pyatkov <[hidden email]>:
> >
> > > Hi Val,
> > >
> > > If we update local backup immediate synchronously when sending commit
> to
> > > primary, this only partly removes questions about consistence view.
> > > But we always can to get other (unpredictable) value, because another
> > > transaction will be executed simultaneously from other threads.
> > >
> > > At the same time this is good place for optimization, probably reduce
> > > network overhead.
> > > I think, need to create a ticket in Jira for the improvement.
> > >
> > >
> > > On Tue, Oct 3, 2017 at 12:27 AM, Valentin Kulichenko <
> > > [hidden email]> wrote:
> > >
> > > > Igniters,
> > > >
> > > > I noticed that combination of PRIMARY_SYNC mode and
> readFromBackup=true
> > > > (both are default values BTW) introduces weird semantics when reading
> > > *on a
> > > > backup node*. Basically, if I do put and then get for the same key in
> > the
> > > > same thread, I can get previous value. In my understanding, this
> > happens
> > > > because even local backup is updated asynchronously in this case.
> > > >
> > > > First of all, this is obviously confusing and would be considered as
> a
> > > bug
> > > > by most of the users (I just updated the key with some value, why
> > would I
> > > > get another value when reading it?).
> > > >
> > > > Second of all, it seems that we send a network message from primary
> > node
> > > to
> > > > local backup, which doesn't make much sense to me and looks like
> > > > unnecessary performance overhead.
> > > >
> > > > Is it possible to update local backup synchronously in this scenario?
> > > >
> > > > -Val
> > > >
> > >
> > >
> > >
> > > --
> > > Vladislav Pyatkov
> > >
> >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>

[yzhdanov]

>>Second of all, it seems that we send a network message from primary node
to
local backup, which doesn't make much sense to me and looks like
unnecessary performance overhead.

This does not work. Remember CLOCK mode for atomic updates? We removed that
mode. You need globally growing version to make it this way.

As a solution I would suggest to update local copy before completing
operation future, i.e. on primary node response processing. Val, please
create a ticket if there is no objections (I would also check if we already
have this ticket).

Another point is that user can go to another backup (send compute closure)
and find old value there. To my mind, it would be cool to have an
opportunity to set write sync mode per operation using
"withWriteSynchronization()" syntax.

--Yakov

[Valentin Kulichenko]

Created a ticket: https://issues.apache.org/jira/browse/IGNITE-6575

-Val

On Thu, Oct 5, 2017 at 8:55 AM, Yakov Zhdanov <[hidden email]> wrote:

> >>Second of all, it seems that we send a network message from primary node
> to
> local backup, which doesn't make much sense to me and looks like
> unnecessary performance overhead.
>
> This does not work. Remember CLOCK mode for atomic updates? We removed that
> mode. You need globally growing version to make it this way.
>
> As a solution I would suggest to update local copy before completing
> operation future, i.e. on primary node response processing. Val, please
> create a ticket if there is no objections (I would also check if we already
> have this ticket).
>
> Another point is that user can go to another backup (send compute closure)
> and find old value there. To my mind, it would be cool to have an
> opportunity to set write sync mode per operation using
> "withWriteSynchronization()" syntax.
>
> --Yakov
>