Future of Ignite transactions

[Vladimir Ozerov]

Igniters,

We are moving towards DBMS system. None of them has a notion of
OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
1) Reads (SELECT) do not acquire exclusive row locks
2) Exclusive lock on read could be forced explicitly (SELECT ... FOR UPDATE)
3) Writes do acuire explicit row locks
4) Locks are always acquired immediately once statement is executed
5) The strictest concurrency level - typically SERIALIZABLE - rely on
so-called *range locks* (or *predicate locks*) to track dependencies
between transactions. Some vendors throw an exception in case of conflict -
these are ones where snapshot-based MVCC is used - PostgreSQL, Oracle.
Others do aggressive locking - ones where two-phase locking algorithm is
used - SQL Server, MySQL.

As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes. Instead,
all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could become
"PESSIMISTIC" if requested explicitly, and for snapshot-based vendors (we
are going in this direction) read-write conflicts are resolved in manner
somewhat similar to our OPTIMISTIC/SERIALIZABLE.

That said, I would propose to think on how transactions could look like in
future Ignite versions (say, 3.0). My rough vision:

1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's only
advantage is deadlock-freedom when combined with SERIALIZABLE. If we have
good deadlock detector and nice administrative capabilities, this would not
be a problem for us.

2) Behavior of reads could be controlled through "with" facade:
V val1 = cache.get(key1);                 // Shared lock or no lock
V val2 = cache.withForUpdate().get(key2); // Exclusive lock

3) REPEATABLE_READ - throw exception in case of write-write conflict

4) SERIALIZABLE - throw exception in case of write-write and write-read
confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it doesn't
support predicates)

5) Add READ_ONLY isolation mode where updates will not be allowed at all.
Such transacrtons would be able to bypass some Ignite internals to achieve
greater performance, what could be valuable for mostly-read use cases (e.g.
OLAP).

Thoughts?

Vladimir.

[dsetrakyan]

Vova,

Thanks for doing the research. The changes you are suggesting are a bit too
bold, so let's discuss them in some more detail...

On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Igniters,
>
> We are moving towards DBMS system. None of them has a notion of
> OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> 1) Reads (SELECT) do not acquire exclusive row locks
> 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> UPDATE)
> 3) Writes do acuire explicit row locks
> 4) Locks are always acquired immediately once statement is executed
> 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> so-called *range locks* (or *predicate locks*) to track dependencies
> between transactions. Some vendors throw an exception in case of conflict -
> these are ones where snapshot-based MVCC is used - PostgreSQL, Oracle.
> Others do aggressive locking - ones where two-phase locking algorithm is
> used - SQL Server, MySQL.
>
> As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes. Instead,
> all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could become
> "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors (we
> are going in this direction) read-write conflicts are resolved in manner
> somewhat similar to our OPTIMISTIC/SERIALIZABLE.
>
> That said, I would propose to think on how transactions could look like in
> future Ignite versions (say, 3.0). My rough vision:
>
> 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's only
> advantage is deadlock-freedom when combined with SERIALIZABLE. If we have
> good deadlock detector and nice administrative capabilities, this would not
> be a problem for us.

Hm... The advantage of Optimistic Serialiazable mode is actually lock-free
transactions. The deadlock is impossible in this case. I doubt any deadlock
detector would match the performance advantage we get from lock-free
transactions.


>
> 2) Behavior of reads could be controlled through "with" facade:
> V val1 = cache.get(key1);                 // Shared lock or no lock
> V val2 = cache.withForUpdate().get(key2); // Exclusive lock
>

Don't like the API. We are not trying to abandon the data grid use-case or
API, we are trying to add the database use case.


> 3) REPEATABLE_READ - throw exception in case of write-write conflict
>

Well, I would like to preserve the PESSIMISTIC mode. I find it more
convenient than the "withForUpdate" API. It almost seems like you are
trying to force the pendulum too far in the opposite direction.


> 4) SERIALIZABLE - throw exception in case of write-write and write-read
> confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it doesn't
> support predicates)
>

So, no change here? Good :)


> 5) Add READ_ONLY isolation mode where updates will not be allowed at all.
> Such transacrtons would be able to bypass some Ignite internals to achieve
> greater performance, what could be valuable for mostly-read use cases (e.g.
> OLAP).
>

Love the idea. We have already seen many use cases that could benefit from
it.
How hard is it to implement?


>
> Thoughts?
>
> Vladimir.
>

[Vladimir Ozerov]

Folks,

Sorry for late reply. I had a chat with several Ignite veterans today. We
tried to design transactional SQL for Ignite. One of our questions was how
to align SQL transactions with current Ignite transactions. And we failed.
And then we came to conclusion that current transaction API is unusable. We
have 6 pairs of modes from API standpoint and 4 real modes. This is very
counterintuitive and cannot be mapped to any transactional framework our
users are familiar with.

So we thought how new tx API might looks like, and here is the draft.

1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
current enum) with three standard modes - READ_COMMITTED, REPEATABLE_READ,
SERIALIZABLE.
2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
4) Reads never acuire locks
5) Writes always acquire locks
6) *IgniteCache.withReadForUpdate()* will return special facade which will
obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
on any update
8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
SERIALIZABLE isolation level.
9) Define new API methods:
- IgniteTransactions.txStart(TransactionIsolationLevel isolation)
- IgniteTransactions.txStart(TransactionIsolationLevel isolation,
TransactionHint... hints)
10) Deprecate old TX start methods

As a result we will have simple, clean and extensible API. Which can be
explained to users in 5 minutes, instead of current half an hour. And which
is perfectly aligned with upcoming transactional SQL.

Thoughts?


On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Vova,
>
> Thanks for doing the research. The changes you are suggesting are a bit too
> bold, so let's discuss them in some more detail...
>
> On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > We are moving towards DBMS system. None of them has a notion of
> > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > 1) Reads (SELECT) do not acquire exclusive row locks
> > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > UPDATE)
> > 3) Writes do acuire explicit row locks
> > 4) Locks are always acquired immediately once statement is executed
> > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > so-called *range locks* (or *predicate locks*) to track dependencies
> > between transactions. Some vendors throw an exception in case of
> conflict -
> > these are ones where snapshot-based MVCC is used - PostgreSQL, Oracle.
> > Others do aggressive locking - ones where two-phase locking algorithm is
> > used - SQL Server, MySQL.
> >
> > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes. Instead,
> > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could become
> > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors (we
> > are going in this direction) read-write conflicts are resolved in manner
> > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> >
> > That said, I would propose to think on how transactions could look like
> in
> > future Ignite versions (say, 3.0). My rough vision:
> >
> > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> only
> > advantage is deadlock-freedom when combined with SERIALIZABLE. If we have
> > good deadlock detector and nice administrative capabilities, this would
> not
> > be a problem for us.
>
>
> Hm... The advantage of Optimistic Serialiazable mode is actually lock-free
> transactions. The deadlock is impossible in this case. I doubt any deadlock
> detector would match the performance advantage we get from lock-free
> transactions.
>
>
> >
> > 2) Behavior of reads could be controlled through "with" facade:
> > V val1 = cache.get(key1);                 // Shared lock or no lock
> > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> >
>
> Don't like the API. We are not trying to abandon the data grid use-case or
> API, we are trying to add the database use case.
>
>
> > 3) REPEATABLE_READ - throw exception in case of write-write conflict
> >
>
> Well, I would like to preserve the PESSIMISTIC mode. I find it more
> convenient than the "withForUpdate" API. It almost seems like you are
> trying to force the pendulum too far in the opposite direction.
>
>
> > 4) SERIALIZABLE - throw exception in case of write-write and write-read
> > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it
> doesn't
> > support predicates)
> >
>
> So, no change here? Good :)
>
>
> > 5) Add READ_ONLY isolation mode where updates will not be allowed at all.
> > Such transacrtons would be able to bypass some Ignite internals to
> achieve
> > greater performance, what could be valuable for mostly-read use cases
> (e.g.
> > OLAP).
> >
>
> Love the idea. We have already seen many use cases that could benefit from
> it.
> How hard is it to implement?
>
>
> >
> > Thoughts?
> >
> > Vladimir.
> >
>

[dsetrakyan]

Vladimir,

This looks like a complete overhaul of our transactional behavior and I do
not think we are at a liberty to make such drastic changes. Can you please
explain why the current behavior would not map to the SQL transactions?

D.

On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Folks,
>
> Sorry for late reply. I had a chat with several Ignite veterans today. We
> tried to design transactional SQL for Ignite. One of our questions was how
> to align SQL transactions with current Ignite transactions. And we failed.
> And then we came to conclusion that current transaction API is unusable. We
> have 6 pairs of modes from API standpoint and 4 real modes. This is very
> counterintuitive and cannot be mapped to any transactional framework our
> users are familiar with.
>
> So we thought how new tx API might looks like, and here is the draft.
>
> 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> current enum) with three standard modes - READ_COMMITTED, REPEATABLE_READ,
> SERIALIZABLE.
> 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> 4) Reads never acuire locks
> 5) Writes always acquire locks
> 6) *IgniteCache.withReadForUpdate()* will return special facade which will
> obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> 7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
> on any update
> 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> SERIALIZABLE isolation level.
> 9) Define new API methods:
> - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> TransactionHint... hints)
> 10) Deprecate old TX start methods
>
> As a result we will have simple, clean and extensible API. Which can be
> explained to users in 5 minutes, instead of current half an hour. And which
> is perfectly aligned with upcoming transactional SQL.
>
> Thoughts?
>
>
> On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Vova,
> >
> > Thanks for doing the research. The changes you are suggesting are a bit
> too
> > bold, so let's discuss them in some more detail...
> >
> > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Igniters,
> > >
> > > We are moving towards DBMS system. None of them has a notion of
> > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > > UPDATE)
> > > 3) Writes do acuire explicit row locks
> > > 4) Locks are always acquired immediately once statement is executed
> > > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > > so-called *range locks* (or *predicate locks*) to track dependencies
> > > between transactions. Some vendors throw an exception in case of
> > conflict -
> > > these are ones where snapshot-based MVCC is used - PostgreSQL, Oracle.
> > > Others do aggressive locking - ones where two-phase locking algorithm
> is
> > > used - SQL Server, MySQL.
> > >
> > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> Instead,
> > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could become
> > > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors
> (we
> > > are going in this direction) read-write conflicts are resolved in
> manner
> > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > >
> > > That said, I would propose to think on how transactions could look like
> > in
> > > future Ignite versions (say, 3.0). My rough vision:
> > >
> > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> > only
> > > advantage is deadlock-freedom when combined with SERIALIZABLE. If we
> have
> > > good deadlock detector and nice administrative capabilities, this would
> > not
> > > be a problem for us.
> >
> >
> > Hm... The advantage of Optimistic Serialiazable mode is actually
> lock-free
> > transactions. The deadlock is impossible in this case. I doubt any
> deadlock
> > detector would match the performance advantage we get from lock-free
> > transactions.
> >
> >
> > >
> > > 2) Behavior of reads could be controlled through "with" facade:
> > > V val1 = cache.get(key1);                 // Shared lock or no lock
> > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > >
> >
> > Don't like the API. We are not trying to abandon the data grid use-case
> or
> > API, we are trying to add the database use case.
> >
> >
> > > 3) REPEATABLE_READ - throw exception in case of write-write conflict
> > >
> >
> > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > convenient than the "withForUpdate" API. It almost seems like you are
> > trying to force the pendulum too far in the opposite direction.
> >
> >
> > > 4) SERIALIZABLE - throw exception in case of write-write and write-read
> > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it
> > doesn't
> > > support predicates)
> > >
> >
> > So, no change here? Good :)
> >
> >
> > > 5) Add READ_ONLY isolation mode where updates will not be allowed at
> all.
> > > Such transacrtons would be able to bypass some Ignite internals to
> > achieve
> > > greater performance, what could be valuable for mostly-read use cases
> > (e.g.
> > > OLAP).
> > >
> >
> > Love the idea. We have already seen many use cases that could benefit
> from
> > it.
> > How hard is it to implement?
> >
> >
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> >
>

[Vladimir Ozerov]

Dima,

Becuase if you take any pair of our concurrency/isolation modes, you will
not find anything similar in SQL. The main problem is that there is no
"OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are "optimistic", but
can be converted to "perssimistic" optionally, and all writes are
"pessimistic". Another problem is that some of our pairs cannot be used in
practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and some
duplicates each other (PESSIMISTIC/REPEATABLE_READ -
PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
reason about. If you cannot explain API in 5 mins, it is broken :-)

If we were many years in the past allowed to choose between Ignite and SQL
approaches I would prefer SQL because it is much simpler.

On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Vladimir,
>
> This looks like a complete overhaul of our transactional behavior and I do
> not think we are at a liberty to make such drastic changes. Can you please
> explain why the current behavior would not map to the SQL transactions?
>
> D.
>
> On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Folks,
> >
> > Sorry for late reply. I had a chat with several Ignite veterans today. We
> > tried to design transactional SQL for Ignite. One of our questions was
> how
> > to align SQL transactions with current Ignite transactions. And we
> failed.
> > And then we came to conclusion that current transaction API is unusable.
> We
> > have 6 pairs of modes from API standpoint and 4 real modes. This is very
> > counterintuitive and cannot be mapped to any transactional framework our
> > users are familiar with.
> >
> > So we thought how new tx API might looks like, and here is the draft.
> >
> > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > current enum) with three standard modes - READ_COMMITTED,
> REPEATABLE_READ,
> > SERIALIZABLE.
> > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > 4) Reads never acuire locks
> > 5) Writes always acquire locks
> > 6) *IgniteCache.withReadForUpdate()* will return special facade which
> will
> > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
> > on any update
> > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> > SERIALIZABLE isolation level.
> > 9) Define new API methods:
> > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > TransactionHint... hints)
> > 10) Deprecate old TX start methods
> >
> > As a result we will have simple, clean and extensible API. Which can be
> > explained to users in 5 minutes, instead of current half an hour. And
> which
> > is perfectly aligned with upcoming transactional SQL.
> >
> > Thoughts?
> >
> >
> > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> > > Vova,
> > >
> > > Thanks for doing the research. The changes you are suggesting are a bit
> > too
> > > bold, so let's discuss them in some more detail...
> > >
> > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <[hidden email]>
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > We are moving towards DBMS system. None of them has a notion of
> > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > > > UPDATE)
> > > > 3) Writes do acuire explicit row locks
> > > > 4) Locks are always acquired immediately once statement is executed
> > > > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > > > so-called *range locks* (or *predicate locks*) to track dependencies
> > > > between transactions. Some vendors throw an exception in case of
> > > conflict -
> > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> Oracle.
> > > > Others do aggressive locking - ones where two-phase locking algorithm
> > is
> > > > used - SQL Server, MySQL.
> > > >
> > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> > Instead,
> > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> become
> > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors
> > (we
> > > > are going in this direction) read-write conflicts are resolved in
> > manner
> > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > >
> > > > That said, I would propose to think on how transactions could look
> like
> > > in
> > > > future Ignite versions (say, 3.0). My rough vision:
> > > >
> > > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> > > only
> > > > advantage is deadlock-freedom when combined with SERIALIZABLE. If we
> > have
> > > > good deadlock detector and nice administrative capabilities, this
> would
> > > not
> > > > be a problem for us.
> > >
> > >
> > > Hm... The advantage of Optimistic Serialiazable mode is actually
> > lock-free
> > > transactions. The deadlock is impossible in this case. I doubt any
> > deadlock
> > > detector would match the performance advantage we get from lock-free
> > > transactions.
> > >
> > >
> > > >
> > > > 2) Behavior of reads could be controlled through "with" facade:
> > > > V val1 = cache.get(key1);                 // Shared lock or no lock
> > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > >
> > >
> > > Don't like the API. We are not trying to abandon the data grid use-case
> > or
> > > API, we are trying to add the database use case.
> > >
> > >
> > > > 3) REPEATABLE_READ - throw exception in case of write-write conflict
> > > >
> > >
> > > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > > convenient than the "withForUpdate" API. It almost seems like you are
> > > trying to force the pendulum too far in the opposite direction.
> > >
> > >
> > > > 4) SERIALIZABLE - throw exception in case of write-write and
> write-read
> > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it
> > > doesn't
> > > > support predicates)
> > > >
> > >
> > > So, no change here? Good :)
> > >
> > >
> > > > 5) Add READ_ONLY isolation mode where updates will not be allowed at
> > all.
> > > > Such transacrtons would be able to bypass some Ignite internals to
> > > achieve
> > > > greater performance, what could be valuable for mostly-read use cases
> > > (e.g.
> > > > OLAP).
> > > >
> > >
> > > Love the idea. We have already seen many use cases that could benefit
> > from
> > > it.
> > > How hard is it to implement?
> > >
> > >
> > > >
> > > > Thoughts?
> > > >
> > > > Vladimir.
> > > >
> > >
> >
>

[dsetrakyan]

Vladimir,

I do not think we have a luxury of changing Ignite transaction APIs. It is
almost the same as changing Ignite put and get APIs. Too much code has
already been written to these APIs. The upgrade path will be so onerous
that no one will ever take it.

As far as the current transaction flags, we will have to continue to
support them. Ideally we should extend them to Ignite SQL as well. If there
is a new transaction flag that you wish to add to Ignite SQL, please
suggest it here.

D.

On Mon, Sep 25, 2017 at 10:48 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Dima,
>
> Becuase if you take any pair of our concurrency/isolation modes, you will
> not find anything similar in SQL. The main problem is that there is no
> "OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are "optimistic", but
> can be converted to "perssimistic" optionally, and all writes are
> "pessimistic". Another problem is that some of our pairs cannot be used in
> practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and some
> duplicates each other (PESSIMISTIC/REPEATABLE_READ -
> PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
> reason about. If you cannot explain API in 5 mins, it is broken :-)
>
> If we were many years in the past allowed to choose between Ignite and SQL
> approaches I would prefer SQL because it is much simpler.
>
> On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Vladimir,
> >
> > This looks like a complete overhaul of our transactional behavior and I
> do
> > not think we are at a liberty to make such drastic changes. Can you
> please
> > explain why the current behavior would not map to the SQL transactions?
> >
> > D.
> >
> > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Folks,
> > >
> > > Sorry for late reply. I had a chat with several Ignite veterans today.
> We
> > > tried to design transactional SQL for Ignite. One of our questions was
> > how
> > > to align SQL transactions with current Ignite transactions. And we
> > failed.
> > > And then we came to conclusion that current transaction API is
> unusable.
> > We
> > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> very
> > > counterintuitive and cannot be mapped to any transactional framework
> our
> > > users are familiar with.
> > >
> > > So we thought how new tx API might looks like, and here is the draft.
> > >
> > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > > current enum) with three standard modes - READ_COMMITTED,
> > REPEATABLE_READ,
> > > SERIALIZABLE.
> > > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > 4) Reads never acuire locks
> > > 5) Writes always acquire locks
> > > 6) *IgniteCache.withReadForUpdate()* will return special facade which
> > will
> > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> exception
> > > on any update
> > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> > > SERIALIZABLE isolation level.
> > > 9) Define new API methods:
> > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > TransactionHint... hints)
> > > 10) Deprecate old TX start methods
> > >
> > > As a result we will have simple, clean and extensible API. Which can be
> > > explained to users in 5 minutes, instead of current half an hour. And
> > which
> > > is perfectly aligned with upcoming transactional SQL.
> > >
> > > Thoughts?
> > >
> > >
> > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Vova,
> > > >
> > > > Thanks for doing the research. The changes you are suggesting are a
> bit
> > > too
> > > > bold, so let's discuss them in some more detail...
> > > >
> > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > We are moving towards DBMS system. None of them has a notion of
> > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > > 2) Exclusive lock on read could be forced explicitly (SELECT ...
> FOR
> > > > > UPDATE)
> > > > > 3) Writes do acuire explicit row locks
> > > > > 4) Locks are always acquired immediately once statement is executed
> > > > > 5) The strictest concurrency level - typically SERIALIZABLE - rely
> on
> > > > > so-called *range locks* (or *predicate locks*) to track
> dependencies
> > > > > between transactions. Some vendors throw an exception in case of
> > > > conflict -
> > > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> > Oracle.
> > > > > Others do aggressive locking - ones where two-phase locking
> algorithm
> > > is
> > > > > used - SQL Server, MySQL.
> > > > >
> > > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> > > Instead,
> > > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> > become
> > > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based
> vendors
> > > (we
> > > > > are going in this direction) read-write conflicts are resolved in
> > > manner
> > > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > > >
> > > > > That said, I would propose to think on how transactions could look
> > like
> > > > in
> > > > > future Ignite versions (say, 3.0). My rough vision:
> > > > >
> > > > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex.
> It's
> > > > only
> > > > > advantage is deadlock-freedom when combined with SERIALIZABLE. If
> we
> > > have
> > > > > good deadlock detector and nice administrative capabilities, this
> > would
> > > > not
> > > > > be a problem for us.
> > > >
> > > >
> > > > Hm... The advantage of Optimistic Serialiazable mode is actually
> > > lock-free
> > > > transactions. The deadlock is impossible in this case. I doubt any
> > > deadlock
> > > > detector would match the performance advantage we get from lock-free
> > > > transactions.
> > > >
> > > >
> > > > >
> > > > > 2) Behavior of reads could be controlled through "with" facade:
> > > > > V val1 = cache.get(key1);                 // Shared lock or no lock
> > > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > > >
> > > >
> > > > Don't like the API. We are not trying to abandon the data grid
> use-case
> > > or
> > > > API, we are trying to add the database use case.
> > > >
> > > >
> > > > > 3) REPEATABLE_READ - throw exception in case of write-write
> conflict
> > > > >
> > > >
> > > > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > > > convenient than the "withForUpdate" API. It almost seems like you are
> > > > trying to force the pendulum too far in the opposite direction.
> > > >
> > > >
> > > > > 4) SERIALIZABLE - throw exception in case of write-write and
> > write-read
> > > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it
> > > > doesn't
> > > > > support predicates)
> > > > >
> > > >
> > > > So, no change here? Good :)
> > > >
> > > >
> > > > > 5) Add READ_ONLY isolation mode where updates will not be allowed
> at
> > > all.
> > > > > Such transacrtons would be able to bypass some Ignite internals to
> > > > achieve
> > > > > greater performance, what could be valuable for mostly-read use
> cases
> > > > (e.g.
> > > > > OLAP).
> > > > >
> > > >
> > > > Love the idea. We have already seen many use cases that could benefit
> > > from
> > > > it.
> > > > How hard is it to implement?
> > > >
> > > >
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vladimir.
> > > > >
> > > >
> > >
> >
>

[Vladimir Ozerov]

Dima,

My proposal would require two changes:
1) Public - for sure we will continue support old API
2) Internal TX engine - these changes are inevitable for transactional SQL
support.

So we have no freedom to choose whether to spend time on internals or not.
We will have to do that anyway.

On Tue, Sep 26, 2017 at 9:02 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Vladimir,
>
> I do not think we have a luxury of changing Ignite transaction APIs. It is
> almost the same as changing Ignite put and get APIs. Too much code has
> already been written to these APIs. The upgrade path will be so onerous
> that no one will ever take it.
>
> As far as the current transaction flags, we will have to continue to
> support them. Ideally we should extend them to Ignite SQL as well. If there
> is a new transaction flag that you wish to add to Ignite SQL, please
> suggest it here.
>
> D.
>
> On Mon, Sep 25, 2017 at 10:48 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Dima,
> >
> > Becuase if you take any pair of our concurrency/isolation modes, you will
> > not find anything similar in SQL. The main problem is that there is no
> > "OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are "optimistic",
> but
> > can be converted to "perssimistic" optionally, and all writes are
> > "pessimistic". Another problem is that some of our pairs cannot be used
> in
> > practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and
> some
> > duplicates each other (PESSIMISTIC/REPEATABLE_READ -
> > PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
> > reason about. If you cannot explain API in 5 mins, it is broken :-)
> >
> > If we were many years in the past allowed to choose between Ignite and
> SQL
> > approaches I would prefer SQL because it is much simpler.
> >
> > On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan <
> [hidden email]>
> > wrote:
> >
> > > Vladimir,
> > >
> > > This looks like a complete overhaul of our transactional behavior and I
> > do
> > > not think we are at a liberty to make such drastic changes. Can you
> > please
> > > explain why the current behavior would not map to the SQL transactions?
> > >
> > > D.
> > >
> > > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <
> [hidden email]>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > > Sorry for late reply. I had a chat with several Ignite veterans
> today.
> > We
> > > > tried to design transactional SQL for Ignite. One of our questions
> was
> > > how
> > > > to align SQL transactions with current Ignite transactions. And we
> > > failed.
> > > > And then we came to conclusion that current transaction API is
> > unusable.
> > > We
> > > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> > very
> > > > counterintuitive and cannot be mapped to any transactional framework
> > our
> > > > users are familiar with.
> > > >
> > > > So we thought how new tx API might looks like, and here is the draft.
> > > >
> > > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > > > current enum) with three standard modes - READ_COMMITTED,
> > > REPEATABLE_READ,
> > > > SERIALIZABLE.
> > > > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> > > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > > 4) Reads never acuire locks
> > > > 5) Writes always acquire locks
> > > > 6) *IgniteCache.withReadForUpdate()* will return special facade
> which
> > > will
> > > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> > > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> > exception
> > > > on any update
> > > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only
> to
> > > > SERIALIZABLE isolation level.
> > > > 9) Define new API methods:
> > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > > TransactionHint... hints)
> > > > 10) Deprecate old TX start methods
> > > >
> > > > As a result we will have simple, clean and extensible API. Which can
> be
> > > > explained to users in 5 minutes, instead of current half an hour. And
> > > which
> > > > is perfectly aligned with upcoming transactional SQL.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Vova,
> > > > >
> > > > > Thanks for doing the research. The changes you are suggesting are a
> > bit
> > > > too
> > > > > bold, so let's discuss them in some more detail...
> > > > >
> > > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > We are moving towards DBMS system. None of them has a notion of
> > > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as
> follows:
> > > > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > > > 2) Exclusive lock on read could be forced explicitly (SELECT ...
> > FOR
> > > > > > UPDATE)
> > > > > > 3) Writes do acuire explicit row locks
> > > > > > 4) Locks are always acquired immediately once statement is
> executed
> > > > > > 5) The strictest concurrency level - typically SERIALIZABLE -
> rely
> > on
> > > > > > so-called *range locks* (or *predicate locks*) to track
> > dependencies
> > > > > > between transactions. Some vendors throw an exception in case of
> > > > > conflict -
> > > > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> > > Oracle.
> > > > > > Others do aggressive locking - ones where two-phase locking
> > algorithm
> > > > is
> > > > > > used - SQL Server, MySQL.
> > > > > >
> > > > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> > > > Instead,
> > > > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> > > become
> > > > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based
> > vendors
> > > > (we
> > > > > > are going in this direction) read-write conflicts are resolved in
> > > > manner
> > > > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > > > >
> > > > > > That said, I would propose to think on how transactions could
> look
> > > like
> > > > > in
> > > > > > future Ignite versions (say, 3.0). My rough vision:
> > > > > >
> > > > > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex.
> > It's
> > > > > only
> > > > > > advantage is deadlock-freedom when combined with SERIALIZABLE. If
> > we
> > > > have
> > > > > > good deadlock detector and nice administrative capabilities, this
> > > would
> > > > > not
> > > > > > be a problem for us.
> > > > >
> > > > >
> > > > > Hm... The advantage of Optimistic Serialiazable mode is actually
> > > > lock-free
> > > > > transactions. The deadlock is impossible in this case. I doubt any
> > > > deadlock
> > > > > detector would match the performance advantage we get from
> lock-free
> > > > > transactions.
> > > > >
> > > > >
> > > > > >
> > > > > > 2) Behavior of reads could be controlled through "with" facade:
> > > > > > V val1 = cache.get(key1);                 // Shared lock or no
> lock
> > > > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > > > >
> > > > >
> > > > > Don't like the API. We are not trying to abandon the data grid
> > use-case
> > > > or
> > > > > API, we are trying to add the database use case.
> > > > >
> > > > >
> > > > > > 3) REPEATABLE_READ - throw exception in case of write-write
> > conflict
> > > > > >
> > > > >
> > > > > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > > > > convenient than the "withForUpdate" API. It almost seems like you
> are
> > > > > trying to force the pendulum too far in the opposite direction.
> > > > >
> > > > >
> > > > > > 4) SERIALIZABLE - throw exception in case of write-write and
> > > write-read
> > > > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but
> it
> > > > > doesn't
> > > > > > support predicates)
> > > > > >
> > > > >
> > > > > So, no change here? Good :)
> > > > >
> > > > >
> > > > > > 5) Add READ_ONLY isolation mode where updates will not be allowed
> > at
> > > > all.
> > > > > > Such transacrtons would be able to bypass some Ignite internals
> to
> > > > > achieve
> > > > > > greater performance, what could be valuable for mostly-read use
> > cases
> > > > > (e.g.
> > > > > > OLAP).
> > > > > >
> > > > >
> > > > > Love the idea. We have already seen many use cases that could
> benefit
> > > > from
> > > > > it.
> > > > > How hard is it to implement?
> > > > >
> > > > >
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Vladimir.
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

On Tue, Sep 26, 2017 at 12:08 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Dima,
>
> My proposal would require two changes:
> 1) Public - for sure we will continue support old API
>

Of course. But my assumption is that you would deprecate the old API,
right? If yes, then you will never be able to remove the deprecated API.

2) Internal TX engine - these changes are inevitable for transactional SQL
> support.
>

Internally we can do any changes we like, as long as they don't change the
expected user behavior.

>
> So we have no freedom to choose whether to spend time on internals or not.
> We will have to do that anyway.
>
> On Tue, Sep 26, 2017 at 9:02 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Vladimir,
> >
> > I do not think we have a luxury of changing Ignite transaction APIs. It
> is
> > almost the same as changing Ignite put and get APIs. Too much code has
> > already been written to these APIs. The upgrade path will be so onerous
> > that no one will ever take it.
> >
> > As far as the current transaction flags, we will have to continue to
> > support them. Ideally we should extend them to Ignite SQL as well. If
> there
> > is a new transaction flag that you wish to add to Ignite SQL, please
> > suggest it here.
> >
> > D.
> >
> > On Mon, Sep 25, 2017 at 10:48 PM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Dima,
> > >
> > > Becuase if you take any pair of our concurrency/isolation modes, you
> will
> > > not find anything similar in SQL. The main problem is that there is no
> > > "OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are "optimistic",
> > but
> > > can be converted to "perssimistic" optionally, and all writes are
> > > "pessimistic". Another problem is that some of our pairs cannot be used
> > in
> > > practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and
> > some
> > > duplicates each other (PESSIMISTIC/REPEATABLE_READ -
> > > PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
> > > reason about. If you cannot explain API in 5 mins, it is broken :-)
> > >
> > > If we were many years in the past allowed to choose between Ignite and
> > SQL
> > > approaches I would prefer SQL because it is much simpler.
> > >
> > > On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan <
> > [hidden email]>
> > > wrote:
> > >
> > > > Vladimir,
> > > >
> > > > This looks like a complete overhaul of our transactional behavior
> and I
> > > do
> > > > not think we are at a liberty to make such drastic changes. Can you
> > > please
> > > > explain why the current behavior would not map to the SQL
> transactions?
> > > >
> > > > D.
> > > >
> > > > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > Sorry for late reply. I had a chat with several Ignite veterans
> > today.
> > > We
> > > > > tried to design transactional SQL for Ignite. One of our questions
> > was
> > > > how
> > > > > to align SQL transactions with current Ignite transactions. And we
> > > > failed.
> > > > > And then we came to conclusion that current transaction API is
> > > unusable.
> > > > We
> > > > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> > > very
> > > > > counterintuitive and cannot be mapped to any transactional
> framework
> > > our
> > > > > users are familiar with.
> > > > >
> > > > > So we thought how new tx API might looks like, and here is the
> draft.
> > > > >
> > > > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes
> with
> > > > > current enum) with three standard modes - READ_COMMITTED,
> > > > REPEATABLE_READ,
> > > > > SERIALIZABLE.
> > > > > 2) Define new enum *TransactionHint* - READ_ONLY,
> OPTIMISTIC_LOCKING
> > > > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > > > 4) Reads never acuire locks
> > > > > 5) Writes always acquire locks
> > > > > 6) *IgniteCache.withReadForUpdate()* will return special facade
> > which
> > > > will
> > > > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in
> SQL.
> > > > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> > > exception
> > > > > on any update
> > > > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into
> our
> > > > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only
> > to
> > > > > SERIALIZABLE isolation level.
> > > > > 9) Define new API methods:
> > > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > > > TransactionHint... hints)
> > > > > 10) Deprecate old TX start methods
> > > > >
> > > > > As a result we will have simple, clean and extensible API. Which
> can
> > be
> > > > > explained to users in 5 minutes, instead of current half an hour.
> And
> > > > which
> > > > > is perfectly aligned with upcoming transactional SQL.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Vova,
> > > > > >
> > > > > > Thanks for doing the research. The changes you are suggesting
> are a
> > > bit
> > > > > too
> > > > > > bold, so let's discuss them in some more detail...
> > > > > >
> > > > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > We are moving towards DBMS system. None of them has a notion of
> > > > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as
> > follows:
> > > > > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > > > > 2) Exclusive lock on read could be forced explicitly (SELECT
> ...
> > > FOR
> > > > > > > UPDATE)
> > > > > > > 3) Writes do acuire explicit row locks
> > > > > > > 4) Locks are always acquired immediately once statement is
> > executed
> > > > > > > 5) The strictest concurrency level - typically SERIALIZABLE -
> > rely
> > > on
> > > > > > > so-called *range locks* (or *predicate locks*) to track
> > > dependencies
> > > > > > > between transactions. Some vendors throw an exception in case
> of
> > > > > > conflict -
> > > > > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> > > > Oracle.
> > > > > > > Others do aggressive locking - ones where two-phase locking
> > > algorithm
> > > > > is
> > > > > > > used - SQL Server, MySQL.
> > > > > > >
> > > > > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC
> modes.
> > > > > Instead,
> > > > > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> > > > become
> > > > > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based
> > > vendors
> > > > > (we
> > > > > > > are going in this direction) read-write conflicts are resolved
> in
> > > > > manner
> > > > > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > > > > >
> > > > > > > That said, I would propose to think on how transactions could
> > look
> > > > like
> > > > > > in
> > > > > > > future Ignite versions (say, 3.0). My rough vision:
> > > > > > >
> > > > > > > 1) No OPTIMISTIC mode at all - too counterintuitive and
> complex.
> > > It's
> > > > > > only
> > > > > > > advantage is deadlock-freedom when combined with SERIALIZABLE.
> If
> > > we
> > > > > have
> > > > > > > good deadlock detector and nice administrative capabilities,
> this
> > > > would
> > > > > > not
> > > > > > > be a problem for us.
> > > > > >
> > > > > >
> > > > > > Hm... The advantage of Optimistic Serialiazable mode is actually
> > > > > lock-free
> > > > > > transactions. The deadlock is impossible in this case. I doubt
> any
> > > > > deadlock
> > > > > > detector would match the performance advantage we get from
> > lock-free
> > > > > > transactions.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > 2) Behavior of reads could be controlled through "with" facade:
> > > > > > > V val1 = cache.get(key1);                 // Shared lock or no
> > lock
> > > > > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > > > > >
> > > > > >
> > > > > > Don't like the API. We are not trying to abandon the data grid
> > > use-case
> > > > > or
> > > > > > API, we are trying to add the database use case.
> > > > > >
> > > > > >
> > > > > > > 3) REPEATABLE_READ - throw exception in case of write-write
> > > conflict
> > > > > > >
> > > > > >
> > > > > > Well, I would like to preserve the PESSIMISTIC mode. I find it
> more
> > > > > > convenient than the "withForUpdate" API. It almost seems like you
> > are
> > > > > > trying to force the pendulum too far in the opposite direction.
> > > > > >
> > > > > >
> > > > > > > 4) SERIALIZABLE - throw exception in case of write-write and
> > > > write-read
> > > > > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but
> > it
> > > > > > doesn't
> > > > > > > support predicates)
> > > > > > >
> > > > > >
> > > > > > So, no change here? Good :)
> > > > > >
> > > > > >
> > > > > > > 5) Add READ_ONLY isolation mode where updates will not be
> allowed
> > > at
> > > > > all.
> > > > > > > Such transacrtons would be able to bypass some Ignite internals
> > to
> > > > > > achieve
> > > > > > > greater performance, what could be valuable for mostly-read use
> > > cases
> > > > > > (e.g.
> > > > > > > OLAP).
> > > > > > >
> > > > > >
> > > > > > Love the idea. We have already seen many use cases that could
> > benefit
> > > > > from
> > > > > > it.
> > > > > > How hard is it to implement?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Alexey Goncharuk]

Dmitriy,

Agree with Vladimir here. OPTIMISTIC + (READ_COMMITTED | REPEATABLE_READ)
modes are completely unusable in real-life use-cases because they do not
allow any read-write conflict detection and thus the explicit transaction
statement can be omitted at all. The remaining combinations of tx
concurrency and isolation can be reduced to a new enum that will match SQL
isolation levels with crystal clear isolation guarantees.

I see no problem with deprecating the old API and completely removing it in
3.0.

2017-09-26 11:52 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> On Tue, Sep 26, 2017 at 12:08 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Dima,
> >
> > My proposal would require two changes:
> > 1) Public - for sure we will continue support old API
> >
>
> Of course. But my assumption is that you would deprecate the old API,
> right? If yes, then you will never be able to remove the deprecated API.
>
> 2) Internal TX engine - these changes are inevitable for transactional SQL
> > support.
> >
>
> Internally we can do any changes we like, as long as they don't change the
> expected user behavior.
>
>
> >
> > So we have no freedom to choose whether to spend time on internals or
> not.
> > We will have to do that anyway.
> >
> > On Tue, Sep 26, 2017 at 9:02 AM, Dmitriy Setrakyan <
> [hidden email]>
> > wrote:
> >
> > > Vladimir,
> > >
> > > I do not think we have a luxury of changing Ignite transaction APIs. It
> > is
> > > almost the same as changing Ignite put and get APIs. Too much code has
> > > already been written to these APIs. The upgrade path will be so onerous
> > > that no one will ever take it.
> > >
> > > As far as the current transaction flags, we will have to continue to
> > > support them. Ideally we should extend them to Ignite SQL as well. If
> > there
> > > is a new transaction flag that you wish to add to Ignite SQL, please
> > > suggest it here.
> > >
> > > D.
> > >
> > > On Mon, Sep 25, 2017 at 10:48 PM, Vladimir Ozerov <
> [hidden email]>
> > > wrote:
> > >
> > > > Dima,
> > > >
> > > > Becuase if you take any pair of our concurrency/isolation modes, you
> > will
> > > > not find anything similar in SQL. The main problem is that there is
> no
> > > > "OPTIMISTIC" or "PESSIMISTIC" in SQL world. All reads are
> "optimistic",
> > > but
> > > > can be converted to "perssimistic" optionally, and all writes are
> > > > "pessimistic". Another problem is that some of our pairs cannot be
> used
> > > in
> > > > practice (OPTIMISTIC/READ_COMMITTED, OPTIMISTIC/REPEATABLE_READ), and
> > > some
> > > > duplicates each other (PESSIMISTIC/REPEATABLE_READ -
> > > > PESSIMISTIC/SERIALIZABLE). This make our API very hard to explain and
> > > > reason about. If you cannot explain API in 5 mins, it is broken :-)
> > > >
> > > > If we were many years in the past allowed to choose between Ignite
> and
> > > SQL
> > > > approaches I would prefer SQL because it is much simpler.
> > > >
> > > > On Tue, Sep 26, 2017 at 1:29 AM, Dmitriy Setrakyan <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Vladimir,
> > > > >
> > > > > This looks like a complete overhaul of our transactional behavior
> > and I
> > > > do
> > > > > not think we are at a liberty to make such drastic changes. Can you
> > > > please
> > > > > explain why the current behavior would not map to the SQL
> > transactions?
> > > > >
> > > > > D.
> > > > >
> > > > > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <
> > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > Sorry for late reply. I had a chat with several Ignite veterans
> > > today.
> > > > We
> > > > > > tried to design transactional SQL for Ignite. One of our
> questions
> > > was
> > > > > how
> > > > > > to align SQL transactions with current Ignite transactions. And
> we
> > > > > failed.
> > > > > > And then we came to conclusion that current transaction API is
> > > > unusable.
> > > > > We
> > > > > > have 6 pairs of modes from API standpoint and 4 real modes. This
> is
> > > > very
> > > > > > counterintuitive and cannot be mapped to any transactional
> > framework
> > > > our
> > > > > > users are familiar with.
> > > > > >
> > > > > > So we thought how new tx API might looks like, and here is the
> > draft.
> > > > > >
> > > > > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes
> > with
> > > > > > current enum) with three standard modes - READ_COMMITTED,
> > > > > REPEATABLE_READ,
> > > > > > SERIALIZABLE.
> > > > > > 2) Define new enum *TransactionHint* - READ_ONLY,
> > OPTIMISTIC_LOCKING
> > > > > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > > > > 4) Reads never acuire locks
> > > > > > 5) Writes always acquire locks
> > > > > > 6) *IgniteCache.withReadForUpdate()* will return special facade
> > > which
> > > > > will
> > > > > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in
> > SQL.
> > > > > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> > > > exception
> > > > > > on any update
> > > > > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into
> > our
> > > > > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable
> only
> > > to
> > > > > > SERIALIZABLE isolation level.
> > > > > > 9) Define new API methods:
> > > > > > - IgniteTransactions.txStart(TransactionIsolationLevel
> isolation)
> > > > > > - IgniteTransactions.txStart(TransactionIsolationLevel
> isolation,
> > > > > > TransactionHint... hints)
> > > > > > 10) Deprecate old TX start methods
> > > > > >
> > > > > > As a result we will have simple, clean and extensible API. Which
> > can
> > > be
> > > > > > explained to users in 5 minutes, instead of current half an hour.
> > And
> > > > > which
> > > > > > is perfectly aligned with upcoming transactional SQL.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Vova,
> > > > > > >
> > > > > > > Thanks for doing the research. The changes you are suggesting
> > are a
> > > > bit
> > > > > > too
> > > > > > > bold, so let's discuss them in some more detail...
> > > > > > >
> > > > > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > We are moving towards DBMS system. None of them has a notion
> of
> > > > > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as
> > > follows:
> > > > > > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > > > > > 2) Exclusive lock on read could be forced explicitly (SELECT
> > ...
> > > > FOR
> > > > > > > > UPDATE)
> > > > > > > > 3) Writes do acuire explicit row locks
> > > > > > > > 4) Locks are always acquired immediately once statement is
> > > executed
> > > > > > > > 5) The strictest concurrency level - typically SERIALIZABLE -
> > > rely
> > > > on
> > > > > > > > so-called *range locks* (or *predicate locks*) to track
> > > > dependencies
> > > > > > > > between transactions. Some vendors throw an exception in case
> > of
> > > > > > > conflict -
> > > > > > > > these are ones where snapshot-based MVCC is used -
> PostgreSQL,
> > > > > Oracle.
> > > > > > > > Others do aggressive locking - ones where two-phase locking
> > > > algorithm
> > > > > > is
> > > > > > > > used - SQL Server, MySQL.
> > > > > > > >
> > > > > > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC
> > modes.
> > > > > > Instead,
> > > > > > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but
> could
> > > > > become
> > > > > > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based
> > > > vendors
> > > > > > (we
> > > > > > > > are going in this direction) read-write conflicts are
> resolved
> > in
> > > > > > manner
> > > > > > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > > > > > >
> > > > > > > > That said, I would propose to think on how transactions could
> > > look
> > > > > like
> > > > > > > in
> > > > > > > > future Ignite versions (say, 3.0). My rough vision:
> > > > > > > >
> > > > > > > > 1) No OPTIMISTIC mode at all - too counterintuitive and
> > complex.
> > > > It's
> > > > > > > only
> > > > > > > > advantage is deadlock-freedom when combined with
> SERIALIZABLE.
> > If
> > > > we
> > > > > > have
> > > > > > > > good deadlock detector and nice administrative capabilities,
> > this
> > > > > would
> > > > > > > not
> > > > > > > > be a problem for us.
> > > > > > >
> > > > > > >
> > > > > > > Hm... The advantage of Optimistic Serialiazable mode is
> actually
> > > > > > lock-free
> > > > > > > transactions. The deadlock is impossible in this case. I doubt
> > any
> > > > > > deadlock
> > > > > > > detector would match the performance advantage we get from
> > > lock-free
> > > > > > > transactions.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > 2) Behavior of reads could be controlled through "with"
> facade:
> > > > > > > > V val1 = cache.get(key1);                 // Shared lock or
> no
> > > lock
> > > > > > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > > > > > >
> > > > > > >
> > > > > > > Don't like the API. We are not trying to abandon the data grid
> > > > use-case
> > > > > > or
> > > > > > > API, we are trying to add the database use case.
> > > > > > >
> > > > > > >
> > > > > > > > 3) REPEATABLE_READ - throw exception in case of write-write
> > > > conflict
> > > > > > > >
> > > > > > >
> > > > > > > Well, I would like to preserve the PESSIMISTIC mode. I find it
> > more
> > > > > > > convenient than the "withForUpdate" API. It almost seems like
> you
> > > are
> > > > > > > trying to force the pendulum too far in the opposite direction.
> > > > > > >
> > > > > > >
> > > > > > > > 4) SERIALIZABLE - throw exception in case of write-write and
> > > > > write-read
> > > > > > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now,
> but
> > > it
> > > > > > > doesn't
> > > > > > > > support predicates)
> > > > > > > >
> > > > > > >
> > > > > > > So, no change here? Good :)
> > > > > > >
> > > > > > >
> > > > > > > > 5) Add READ_ONLY isolation mode where updates will not be
> > allowed
> > > > at
> > > > > > all.
> > > > > > > > Such transacrtons would be able to bypass some Ignite
> internals
> > > to
> > > > > > > achieve
> > > > > > > > greater performance, what could be valuable for mostly-read
> use
> > > > cases
> > > > > > > (e.g.
> > > > > > > > OLAP).
> > > > > > > >
> > > > > > >
> > > > > > > Love the idea. We have already seen many use cases that could
> > > benefit
> > > > > > from
> > > > > > > it.
> > > > > > > How hard is it to implement?
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Vladimir.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

OK, if we must change the current behavior, let's discuss the new design.
My comments/questions are below...

On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Folks,
>
> Sorry for late reply. I had a chat with several Ignite veterans today. We
> tried to design transactional SQL for Ignite. One of our questions was how
> to align SQL transactions with current Ignite transactions. And we failed.
> And then we came to conclusion that current transaction API is unusable. We
> have 6 pairs of modes from API standpoint and 4 real modes. This is very
> counterintuitive and cannot be mapped to any transactional framework our
> users are familiar with.
>
> So we thought how new tx API might looks like, and here is the draft.
>
> 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> current enum) with three standard modes - READ_COMMITTED, REPEATABLE_READ,
> SERIALIZABLE.
>

If it is the same values as we have today, why create a new enum?


> 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
>

The word *hint* means no guarantee. When it comes to transactions, we must
have guarantees. Also, what happens to PESSIMISTIC locking?


> 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
>

But you are still proposing OPTIMISTIC_LOCKING above. So, we have
OPTIMISTIC without PESSIMISTIC?


> 4) Reads never acuire locks
> 5) Writes always acquire locks
>

Hm... what locks? I think we are getting to deep into the weeds here. Isn't
it our internal implementation detail to acquire locks or not? For example,
MVCC approach we are working on will be virtually lock-free for all
scenarios.

I would focus on transaction behavior, not locks.



> 6) *IgniteCache.withReadForUpdate()* will return special facade which will
> obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
>

This syntax is very ugly in databases and we do not need to carry this over
to Ignite. All it means is that a pessimistic lock is acquired. To be
honest, I would rather start a pessimistic transaction to get the same
behavior.


> 7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
> on any update
>

I really like the READ_ONLY flag. Allows us to put many optimizations in
place.


> 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> SERIALIZABLE isolation level.
>

Is there a PESSIMISTIC/SERIALIZABLE mode?


> 9) Define new API methods:
> - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> TransactionHint... hints)

10) Deprecate old TX start methods
>
> As a result we will have simple, clean and extensible API. Which can be
> explained to users in 5 minutes, instead of current half an hour. And which
> is perfectly aligned with upcoming transactional SQL.
>

The API does not look clean yet. Still requires lots of work.

>
> Thoughts?
>
>
> On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Vova,
> >
> > Thanks for doing the research. The changes you are suggesting are a bit
> too
> > bold, so let's discuss them in some more detail...
> >
> > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Igniters,
> > >
> > > We are moving towards DBMS system. None of them has a notion of
> > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > > UPDATE)
> > > 3) Writes do acuire explicit row locks
> > > 4) Locks are always acquired immediately once statement is executed
> > > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > > so-called *range locks* (or *predicate locks*) to track dependencies
> > > between transactions. Some vendors throw an exception in case of
> > conflict -
> > > these are ones where snapshot-based MVCC is used - PostgreSQL, Oracle.
> > > Others do aggressive locking - ones where two-phase locking algorithm
> is
> > > used - SQL Server, MySQL.
> > >
> > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> Instead,
> > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could become
> > > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors
> (we
> > > are going in this direction) read-write conflicts are resolved in
> manner
> > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > >
> > > That said, I would propose to think on how transactions could look like
> > in
> > > future Ignite versions (say, 3.0). My rough vision:
> > >
> > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> > only
> > > advantage is deadlock-freedom when combined with SERIALIZABLE. If we
> have
> > > good deadlock detector and nice administrative capabilities, this would
> > not
> > > be a problem for us.
> >
> >
> > Hm... The advantage of Optimistic Serialiazable mode is actually
> lock-free
> > transactions. The deadlock is impossible in this case. I doubt any
> deadlock
> > detector would match the performance advantage we get from lock-free
> > transactions.
> >
> >
> > >
> > > 2) Behavior of reads could be controlled through "with" facade:
> > > V val1 = cache.get(key1);                 // Shared lock or no lock
> > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > >
> >
> > Don't like the API. We are not trying to abandon the data grid use-case
> or
> > API, we are trying to add the database use case.
> >
> >
> > > 3) REPEATABLE_READ - throw exception in case of write-write conflict
> > >
> >
> > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > convenient than the "withForUpdate" API. It almost seems like you are
> > trying to force the pendulum too far in the opposite direction.
> >
> >
> > > 4) SERIALIZABLE - throw exception in case of write-write and write-read
> > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it
> > doesn't
> > > support predicates)
> > >
> >
> > So, no change here? Good :)
> >
> >
> > > 5) Add READ_ONLY isolation mode where updates will not be allowed at
> all.
> > > Such transacrtons would be able to bypass some Ignite internals to
> > achieve
> > > greater performance, what could be valuable for mostly-read use cases
> > (e.g.
> > > OLAP).
> > >
> >
> > Love the idea. We have already seen many use cases that could benefit
> from
> > it.
> > How hard is it to implement?
> >
> >
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> >
>

[Alexei Scherbakov]

Do we have a plan to support multi-version concurrency control ?

Proper implementation allows to achieve better isolation without blocking
for the cost of additional resource consumption.

The feature can be configurable per transaction.

6) What if I want to acquire write locks only for some specific reads
within same transaction ?


2017-09-27 5:52 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> OK, if we must change the current behavior, let's discuss the new design.
> My comments/questions are below...
>
> On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Folks,
> >
> > Sorry for late reply. I had a chat with several Ignite veterans today. We
> > tried to design transactional SQL for Ignite. One of our questions was
> how
> > to align SQL transactions with current Ignite transactions. And we
> failed.
> > And then we came to conclusion that current transaction API is unusable.
> We
> > have 6 pairs of modes from API standpoint and 4 real modes. This is very
> > counterintuitive and cannot be mapped to any transactional framework our
> > users are familiar with.
> >
> > So we thought how new tx API might looks like, and here is the draft.
> >
> > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > current enum) with three standard modes - READ_COMMITTED,
> REPEATABLE_READ,
> > SERIALIZABLE.
> >
>
> If it is the same values as we have today, why create a new enum?
>
>
> > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> >
>
> The word *hint* means no guarantee. When it comes to transactions, we must
> have guarantees. Also, what happens to PESSIMISTIC locking?
>
>
> > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> >
>
> But you are still proposing OPTIMISTIC_LOCKING above. So, we have
> OPTIMISTIC without PESSIMISTIC?
>
>
> > 4) Reads never acuire locks
> > 5) Writes always acquire locks
> >
>
> Hm... what locks? I think we are getting to deep into the weeds here. Isn't
> it our internal implementation detail to acquire locks or not? For example,
> MVCC approach we are working on will be virtually lock-free for all
> scenarios.
>
> I would focus on transaction behavior, not locks.
>
>
>
> > 6) *IgniteCache.withReadForUpdate()* will return special facade which
> will
> > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> >
>
> This syntax is very ugly in databases and we do not need to carry this over
> to Ignite. All it means is that a pessimistic lock is acquired. To be
> honest, I would rather start a pessimistic transaction to get the same
> behavior.
>
>
> > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an exception
> > on any update
> >
>
> I really like the READ_ONLY flag. Allows us to put many optimizations in
> place.
>
>
> > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> > SERIALIZABLE isolation level.
> >
>
> Is there a PESSIMISTIC/SERIALIZABLE mode?
>
>
> > 9) Define new API methods:
> > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > TransactionHint... hints)
>
> 10) Deprecate old TX start methods
> >
> > As a result we will have simple, clean and extensible API. Which can be
> > explained to users in 5 minutes, instead of current half an hour. And
> which
> > is perfectly aligned with upcoming transactional SQL.
> >
>
> The API does not look clean yet. Still requires lots of work.
>
>
> >
> > Thoughts?
> >
> >
> > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> > > Vova,
> > >
> > > Thanks for doing the research. The changes you are suggesting are a bit
> > too
> > > bold, so let's discuss them in some more detail...
> > >
> > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <[hidden email]>
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > We are moving towards DBMS system. None of them has a notion of
> > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > 2) Exclusive lock on read could be forced explicitly (SELECT ... FOR
> > > > UPDATE)
> > > > 3) Writes do acuire explicit row locks
> > > > 4) Locks are always acquired immediately once statement is executed
> > > > 5) The strictest concurrency level - typically SERIALIZABLE - rely on
> > > > so-called *range locks* (or *predicate locks*) to track dependencies
> > > > between transactions. Some vendors throw an exception in case of
> > > conflict -
> > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> Oracle.
> > > > Others do aggressive locking - ones where two-phase locking algorithm
> > is
> > > > used - SQL Server, MySQL.
> > > >
> > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> > Instead,
> > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> become
> > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based vendors
> > (we
> > > > are going in this direction) read-write conflicts are resolved in
> > manner
> > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > >
> > > > That said, I would propose to think on how transactions could look
> like
> > > in
> > > > future Ignite versions (say, 3.0). My rough vision:
> > > >
> > > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex. It's
> > > only
> > > > advantage is deadlock-freedom when combined with SERIALIZABLE. If we
> > have
> > > > good deadlock detector and nice administrative capabilities, this
> would
> > > not
> > > > be a problem for us.
> > >
> > >
> > > Hm... The advantage of Optimistic Serialiazable mode is actually
> > lock-free
> > > transactions. The deadlock is impossible in this case. I doubt any
> > deadlock
> > > detector would match the performance advantage we get from lock-free
> > > transactions.
> > >
> > >
> > > >
> > > > 2) Behavior of reads could be controlled through "with" facade:
> > > > V val1 = cache.get(key1);                 // Shared lock or no lock
> > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > >
> > >
> > > Don't like the API. We are not trying to abandon the data grid use-case
> > or
> > > API, we are trying to add the database use case.
> > >
> > >
> > > > 3) REPEATABLE_READ - throw exception in case of write-write conflict
> > > >
> > >
> > > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > > convenient than the "withForUpdate" API. It almost seems like you are
> > > trying to force the pendulum too far in the opposite direction.
> > >
> > >
> > > > 4) SERIALIZABLE - throw exception in case of write-write and
> write-read
> > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it
> > > doesn't
> > > > support predicates)
> > > >
> > >
> > > So, no change here? Good :)
> > >
> > >
> > > > 5) Add READ_ONLY isolation mode where updates will not be allowed at
> > all.
> > > > Such transacrtons would be able to bypass some Ignite internals to
> > > achieve
> > > > greater performance, what could be valuable for mostly-read use cases
> > > (e.g.
> > > > OLAP).
> > > >
> > >
> > > Love the idea. We have already seen many use cases that could benefit
> > from
> > > it.
> > > How hard is it to implement?
> > >
> > >
> > > >
> > > > Thoughts?
> > > >
> > > > Vladimir.
> > > >
> > >
> >
>

--

Best regards,
Alexei Scherbakov

[Vladimir Ozerov]

Alex,

MVCC concept and blocking/non-blocking behavior are not inter-related. You
can have both blocking and non-blocking writes with MVCC and without it.
The thing is that users *want* blocking behavior as it is easy to reason
about. This is why even MVCC-based RDBMS vendors (Oracle, PostgreSQL) do
block on writes.

OPTIMSITIC mode and non-blocking approaches are for rare enthusiasts. It is
not in priority for us.

On Thu, Sep 28, 2017 at 1:59 PM, Alexei Scherbakov <
[hidden email]> wrote:

> Do we have a plan to support multi-version concurrency control ?
>
> Proper implementation allows to achieve better isolation without blocking
> for the cost of additional resource consumption.
>
> The feature can be configurable per transaction.
>
> 6) What if I want to acquire write locks only for some specific reads
> within same transaction ?
>
>
> 2017-09-27 5:52 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > OK, if we must change the current behavior, let's discuss the new design.
> > My comments/questions are below...
> >
> > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Folks,
> > >
> > > Sorry for late reply. I had a chat with several Ignite veterans today.
> We
> > > tried to design transactional SQL for Ignite. One of our questions was
> > how
> > > to align SQL transactions with current Ignite transactions. And we
> > failed.
> > > And then we came to conclusion that current transaction API is
> unusable.
> > We
> > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> very
> > > counterintuitive and cannot be mapped to any transactional framework
> our
> > > users are familiar with.
> > >
> > > So we thought how new tx API might looks like, and here is the draft.
> > >
> > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > > current enum) with three standard modes - READ_COMMITTED,
> > REPEATABLE_READ,
> > > SERIALIZABLE.
> > >
> >
> > If it is the same values as we have today, why create a new enum?
> >
> >
> > > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> > >
> >
> > The word *hint* means no guarantee. When it comes to transactions, we
> must
> > have guarantees. Also, what happens to PESSIMISTIC locking?
> >
> >
> > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > >
> >
> > But you are still proposing OPTIMISTIC_LOCKING above. So, we have
> > OPTIMISTIC without PESSIMISTIC?
> >
> >
> > > 4) Reads never acuire locks
> > > 5) Writes always acquire locks
> > >
> >
> > Hm... what locks? I think we are getting to deep into the weeds here.
> Isn't
> > it our internal implementation detail to acquire locks or not? For
> example,
> > MVCC approach we are working on will be virtually lock-free for all
> > scenarios.
> >
> > I would focus on transaction behavior, not locks.
> >
> >
> >
> > > 6) *IgniteCache.withReadForUpdate()* will return special facade which
> > will
> > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> > >
> >
> > This syntax is very ugly in databases and we do not need to carry this
> over
> > to Ignite. All it means is that a pessimistic lock is acquired. To be
> > honest, I would rather start a pessimistic transaction to get the same
> > behavior.
> >
> >
> > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> exception
> > > on any update
> > >
> >
> > I really like the READ_ONLY flag. Allows us to put many optimizations in
> > place.
> >
> >
> > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only to
> > > SERIALIZABLE isolation level.
> > >
> >
> > Is there a PESSIMISTIC/SERIALIZABLE mode?
> >
> >
> > > 9) Define new API methods:
> > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > TransactionHint... hints)
> >
> > 10) Deprecate old TX start methods
> > >
> > > As a result we will have simple, clean and extensible API. Which can be
> > > explained to users in 5 minutes, instead of current half an hour. And
> > which
> > > is perfectly aligned with upcoming transactional SQL.
> > >
> >
> > The API does not look clean yet. Still requires lots of work.
> >
> >
> > >
> > > Thoughts?
> > >
> > >
> > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Vova,
> > > >
> > > > Thanks for doing the research. The changes you are suggesting are a
> bit
> > > too
> > > > bold, so let's discuss them in some more detail...
> > > >
> > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > We are moving towards DBMS system. None of them has a notion of
> > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as follows:
> > > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > > 2) Exclusive lock on read could be forced explicitly (SELECT ...
> FOR
> > > > > UPDATE)
> > > > > 3) Writes do acuire explicit row locks
> > > > > 4) Locks are always acquired immediately once statement is executed
> > > > > 5) The strictest concurrency level - typically SERIALIZABLE - rely
> on
> > > > > so-called *range locks* (or *predicate locks*) to track
> dependencies
> > > > > between transactions. Some vendors throw an exception in case of
> > > > conflict -
> > > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> > Oracle.
> > > > > Others do aggressive locking - ones where two-phase locking
> algorithm
> > > is
> > > > > used - SQL Server, MySQL.
> > > > >
> > > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> > > Instead,
> > > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> > become
> > > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based
> vendors
> > > (we
> > > > > are going in this direction) read-write conflicts are resolved in
> > > manner
> > > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > > >
> > > > > That said, I would propose to think on how transactions could look
> > like
> > > > in
> > > > > future Ignite versions (say, 3.0). My rough vision:
> > > > >
> > > > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex.
> It's
> > > > only
> > > > > advantage is deadlock-freedom when combined with SERIALIZABLE. If
> we
> > > have
> > > > > good deadlock detector and nice administrative capabilities, this
> > would
> > > > not
> > > > > be a problem for us.
> > > >
> > > >
> > > > Hm... The advantage of Optimistic Serialiazable mode is actually
> > > lock-free
> > > > transactions. The deadlock is impossible in this case. I doubt any
> > > deadlock
> > > > detector would match the performance advantage we get from lock-free
> > > > transactions.
> > > >
> > > >
> > > > >
> > > > > 2) Behavior of reads could be controlled through "with" facade:
> > > > > V val1 = cache.get(key1);                 // Shared lock or no lock
> > > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > > >
> > > >
> > > > Don't like the API. We are not trying to abandon the data grid
> use-case
> > > or
> > > > API, we are trying to add the database use case.
> > > >
> > > >
> > > > > 3) REPEATABLE_READ - throw exception in case of write-write
> conflict
> > > > >
> > > >
> > > > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > > > convenient than the "withForUpdate" API. It almost seems like you are
> > > > trying to force the pendulum too far in the opposite direction.
> > > >
> > > >
> > > > > 4) SERIALIZABLE - throw exception in case of write-write and
> > write-read
> > > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but it
> > > > doesn't
> > > > > support predicates)
> > > > >
> > > >
> > > > So, no change here? Good :)
> > > >
> > > >
> > > > > 5) Add READ_ONLY isolation mode where updates will not be allowed
> at
> > > all.
> > > > > Such transacrtons would be able to bypass some Ignite internals to
> > > > achieve
> > > > > greater performance, what could be valuable for mostly-read use
> cases
> > > > (e.g.
> > > > > OLAP).
> > > > >
> > > >
> > > > Love the idea. We have already seen many use cases that could benefit
> > > from
> > > > it.
> > > > How hard is it to implement?
> > > >
> > > >
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vladimir.
> > > > >
> > > >
> > >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

[dsetrakyan]

But what if blocking on reads is needed? Also, how about pessimistic
read-only transactions? Do we plan to support them?

D.

On Thu, Sep 28, 2017 at 6:44 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Alex,
>
> MVCC concept and blocking/non-blocking behavior are not inter-related. You
> can have both blocking and non-blocking writes with MVCC and without it.
> The thing is that users *want* blocking behavior as it is easy to reason
> about. This is why even MVCC-based RDBMS vendors (Oracle, PostgreSQL) do
> block on writes.
>
> OPTIMSITIC mode and non-blocking approaches are for rare enthusiasts. It is
> not in priority for us.
>
> On Thu, Sep 28, 2017 at 1:59 PM, Alexei Scherbakov <
> [hidden email]> wrote:
>
> > Do we have a plan to support multi-version concurrency control ?
> >
> > Proper implementation allows to achieve better isolation without blocking
> > for the cost of additional resource consumption.
> >
> > The feature can be configurable per transaction.
> >
> > 6) What if I want to acquire write locks only for some specific reads
> > within same transaction ?
> >
> >
> > 2017-09-27 5:52 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > OK, if we must change the current behavior, let's discuss the new
> design.
> > > My comments/questions are below...
> > >
> > > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <
> [hidden email]>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > > Sorry for late reply. I had a chat with several Ignite veterans
> today.
> > We
> > > > tried to design transactional SQL for Ignite. One of our questions
> was
> > > how
> > > > to align SQL transactions with current Ignite transactions. And we
> > > failed.
> > > > And then we came to conclusion that current transaction API is
> > unusable.
> > > We
> > > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> > very
> > > > counterintuitive and cannot be mapped to any transactional framework
> > our
> > > > users are familiar with.
> > > >
> > > > So we thought how new tx API might looks like, and here is the draft.
> > > >
> > > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes with
> > > > current enum) with three standard modes - READ_COMMITTED,
> > > REPEATABLE_READ,
> > > > SERIALIZABLE.
> > > >
> > >
> > > If it is the same values as we have today, why create a new enum?
> > >
> > >
> > > > 2) Define new enum *TransactionHint* - READ_ONLY, OPTIMISTIC_LOCKING
> > > >
> > >
> > > The word *hint* means no guarantee. When it comes to transactions, we
> > must
> > > have guarantees. Also, what happens to PESSIMISTIC locking?
> > >
> > >
> > > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > >
> > >
> > > But you are still proposing OPTIMISTIC_LOCKING above. So, we have
> > > OPTIMISTIC without PESSIMISTIC?
> > >
> > >
> > > > 4) Reads never acuire locks
> > > > 5) Writes always acquire locks
> > > >
> > >
> > > Hm... what locks? I think we are getting to deep into the weeds here.
> > Isn't
> > > it our internal implementation detail to acquire locks or not? For
> > example,
> > > MVCC approach we are working on will be virtually lock-free for all
> > > scenarios.
> > >
> > > I would focus on transaction behavior, not locks.
> > >
> > >
> > >
> > > > 6) *IgniteCache.withReadForUpdate()* will return special facade
> which
> > > will
> > > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in SQL.
> > > >
> > >
> > > This syntax is very ugly in databases and we do not need to carry this
> > over
> > > to Ignite. All it means is that a pessimistic lock is acquired. To be
> > > honest, I would rather start a pessimistic transaction to get the same
> > > behavior.
> > >
> > >
> > > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> > exception
> > > > on any update
> > > >
> > >
> > > I really like the READ_ONLY flag. Allows us to put many optimizations
> in
> > > place.
> > >
> > >
> > > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into our
> > > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only
> to
> > > > SERIALIZABLE isolation level.
> > > >
> > >
> > > Is there a PESSIMISTIC/SERIALIZABLE mode?
> > >
> > >
> > > > 9) Define new API methods:
> > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > > TransactionHint... hints)
> > >
> > > 10) Deprecate old TX start methods
> > > >
> > > > As a result we will have simple, clean and extensible API. Which can
> be
> > > > explained to users in 5 minutes, instead of current half an hour. And
> > > which
> > > > is perfectly aligned with upcoming transactional SQL.
> > > >
> > >
> > > The API does not look clean yet. Still requires lots of work.
> > >
> > >
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Vova,
> > > > >
> > > > > Thanks for doing the research. The changes you are suggesting are a
> > bit
> > > > too
> > > > > bold, so let's discuss them in some more detail...
> > > > >
> > > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > We are moving towards DBMS system. None of them has a notion of
> > > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as
> follows:
> > > > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > > > 2) Exclusive lock on read could be forced explicitly (SELECT ...
> > FOR
> > > > > > UPDATE)
> > > > > > 3) Writes do acuire explicit row locks
> > > > > > 4) Locks are always acquired immediately once statement is
> executed
> > > > > > 5) The strictest concurrency level - typically SERIALIZABLE -
> rely
> > on
> > > > > > so-called *range locks* (or *predicate locks*) to track
> > dependencies
> > > > > > between transactions. Some vendors throw an exception in case of
> > > > > conflict -
> > > > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> > > Oracle.
> > > > > > Others do aggressive locking - ones where two-phase locking
> > algorithm
> > > > is
> > > > > > used - SQL Server, MySQL.
> > > > > >
> > > > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC modes.
> > > > Instead,
> > > > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> > > become
> > > > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based
> > vendors
> > > > (we
> > > > > > are going in this direction) read-write conflicts are resolved in
> > > > manner
> > > > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > > > >
> > > > > > That said, I would propose to think on how transactions could
> look
> > > like
> > > > > in
> > > > > > future Ignite versions (say, 3.0). My rough vision:
> > > > > >
> > > > > > 1) No OPTIMISTIC mode at all - too counterintuitive and complex.
> > It's
> > > > > only
> > > > > > advantage is deadlock-freedom when combined with SERIALIZABLE. If
> > we
> > > > have
> > > > > > good deadlock detector and nice administrative capabilities, this
> > > would
> > > > > not
> > > > > > be a problem for us.
> > > > >
> > > > >
> > > > > Hm... The advantage of Optimistic Serialiazable mode is actually
> > > > lock-free
> > > > > transactions. The deadlock is impossible in this case. I doubt any
> > > > deadlock
> > > > > detector would match the performance advantage we get from
> lock-free
> > > > > transactions.
> > > > >
> > > > >
> > > > > >
> > > > > > 2) Behavior of reads could be controlled through "with" facade:
> > > > > > V val1 = cache.get(key1);                 // Shared lock or no
> lock
> > > > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > > > >
> > > > >
> > > > > Don't like the API. We are not trying to abandon the data grid
> > use-case
> > > > or
> > > > > API, we are trying to add the database use case.
> > > > >
> > > > >
> > > > > > 3) REPEATABLE_READ - throw exception in case of write-write
> > conflict
> > > > > >
> > > > >
> > > > > Well, I would like to preserve the PESSIMISTIC mode. I find it more
> > > > > convenient than the "withForUpdate" API. It almost seems like you
> are
> > > > > trying to force the pendulum too far in the opposite direction.
> > > > >
> > > > >
> > > > > > 4) SERIALIZABLE - throw exception in case of write-write and
> > > write-read
> > > > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but
> it
> > > > > doesn't
> > > > > > support predicates)
> > > > > >
> > > > >
> > > > > So, no change here? Good :)
> > > > >
> > > > >
> > > > > > 5) Add READ_ONLY isolation mode where updates will not be allowed
> > at
> > > > all.
> > > > > > Such transacrtons would be able to bypass some Ignite internals
> to
> > > > > achieve
> > > > > > greater performance, what could be valuable for mostly-read use
> > cases
> > > > > (e.g.
> > > > > > OLAP).
> > > > > >
> > > > >
> > > > > Love the idea. We have already seen many use cases that could
> benefit
> > > > from
> > > > > it.
> > > > > How hard is it to implement?
> > > > >
> > > > >
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Vladimir.
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>

[Vladimir Ozerov]

Dima,

IgniteCache.withReadForUpdate() :-)

On Fri, Sep 29, 2017 at 3:29 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> But what if blocking on reads is needed? Also, how about pessimistic
> read-only transactions? Do we plan to support them?
>
> D.
>
> On Thu, Sep 28, 2017 at 6:44 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Alex,
> >
> > MVCC concept and blocking/non-blocking behavior are not inter-related.
> You
> > can have both blocking and non-blocking writes with MVCC and without it.
> > The thing is that users *want* blocking behavior as it is easy to reason
> > about. This is why even MVCC-based RDBMS vendors (Oracle, PostgreSQL) do
> > block on writes.
> >
> > OPTIMSITIC mode and non-blocking approaches are for rare enthusiasts. It
> is
> > not in priority for us.
> >
> > On Thu, Sep 28, 2017 at 1:59 PM, Alexei Scherbakov <
> > [hidden email]> wrote:
> >
> > > Do we have a plan to support multi-version concurrency control ?
> > >
> > > Proper implementation allows to achieve better isolation without
> blocking
> > > for the cost of additional resource consumption.
> > >
> > > The feature can be configurable per transaction.
> > >
> > > 6) What if I want to acquire write locks only for some specific reads
> > > within same transaction ?
> > >
> > >
> > > 2017-09-27 5:52 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > >
> > > > OK, if we must change the current behavior, let's discuss the new
> > design.
> > > > My comments/questions are below...
> > > >
> > > > On Mon, Sep 25, 2017 at 12:09 PM, Vladimir Ozerov <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > Sorry for late reply. I had a chat with several Ignite veterans
> > today.
> > > We
> > > > > tried to design transactional SQL for Ignite. One of our questions
> > was
> > > > how
> > > > > to align SQL transactions with current Ignite transactions. And we
> > > > failed.
> > > > > And then we came to conclusion that current transaction API is
> > > unusable.
> > > > We
> > > > > have 6 pairs of modes from API standpoint and 4 real modes. This is
> > > very
> > > > > counterintuitive and cannot be mapped to any transactional
> framework
> > > our
> > > > > users are familiar with.
> > > > >
> > > > > So we thought how new tx API might looks like, and here is the
> draft.
> > > > >
> > > > > 1) Define new enum *TransactionIsolationLevel *(to avoid clashes
> with
> > > > > current enum) with three standard modes - READ_COMMITTED,
> > > > REPEATABLE_READ,
> > > > > SERIALIZABLE.
> > > > >
> > > >
> > > > If it is the same values as we have today, why create a new enum?
> > > >
> > > >
> > > > > 2) Define new enum *TransactionHint* - READ_ONLY,
> OPTIMISTIC_LOCKING
> > > > >
> > > >
> > > > The word *hint* means no guarantee. When it comes to transactions, we
> > > must
> > > > have guarantees. Also, what happens to PESSIMISTIC locking?
> > > >
> > > >
> > > > > 3) No more OPTIMISTIC and PESSIMISTIC. Seriously.
> > > > >
> > > >
> > > > But you are still proposing OPTIMISTIC_LOCKING above. So, we have
> > > > OPTIMISTIC without PESSIMISTIC?
> > > >
> > > >
> > > > > 4) Reads never acuire locks
> > > > > 5) Writes always acquire locks
> > > > >
> > > >
> > > > Hm... what locks? I think we are getting to deep into the weeds here.
> > > Isn't
> > > > it our internal implementation detail to acquire locks or not? For
> > > example,
> > > > MVCC approach we are working on will be virtually lock-free for all
> > > > scenarios.
> > > >
> > > > I would focus on transaction behavior, not locks.
> > > >
> > > >
> > > >
> > > > > 6) *IgniteCache.withReadForUpdate()* will return special facade
> > which
> > > > will
> > > > > obtain locks on reads. This is analogue of SELECT FOR UPDATE in
> SQL.
> > > > >
> > > >
> > > > This syntax is very ugly in databases and we do not need to carry
> this
> > > over
> > > > to Ignite. All it means is that a pessimistic lock is acquired. To be
> > > > honest, I would rather start a pessimistic transaction to get the
> same
> > > > behavior.
> > > >
> > > >
> > > > > 7) *TransactionHint.READ_ONLY* - forces transaction to throw an
> > > exception
> > > > > on any update
> > > > >
> > > >
> > > > I really like the READ_ONLY flag. Allows us to put many optimizations
> > in
> > > > place.
> > > >
> > > >
> > > > > 8) *TransactionHint.OPTIMISTIC_LOCKING* - turns transaction into
> our
> > > > > current deadlock-free OPTIMISTIC/SERIALIZABLE mode. Applicable only
> > to
> > > > > SERIALIZABLE isolation level.
> > > > >
> > > >
> > > > Is there a PESSIMISTIC/SERIALIZABLE mode?
> > > >
> > > >
> > > > > 9) Define new API methods:
> > > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation)
> > > > > - IgniteTransactions.txStart(TransactionIsolationLevel isolation,
> > > > > TransactionHint... hints)
> > > >
> > > > 10) Deprecate old TX start methods
> > > > >
> > > > > As a result we will have simple, clean and extensible API. Which
> can
> > be
> > > > > explained to users in 5 minutes, instead of current half an hour.
> And
> > > > which
> > > > > is perfectly aligned with upcoming transactional SQL.
> > > > >
> > > >
> > > > The API does not look clean yet. Still requires lots of work.
> > > >
> > > >
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Thu, Sep 7, 2017 at 6:48 AM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Vova,
> > > > > >
> > > > > > Thanks for doing the research. The changes you are suggesting
> are a
> > > bit
> > > > > too
> > > > > > bold, so let's discuss them in some more detail...
> > > > > >
> > > > > > On Wed, Sep 6, 2017 at 4:51 AM, Vladimir Ozerov <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > We are moving towards DBMS system. None of them has a notion of
> > > > > > > OPTIMISTIC/PESSIMISTIC transactions. Instead, they work as
> > follows:
> > > > > > > 1) Reads (SELECT) do not acquire exclusive row locks
> > > > > > > 2) Exclusive lock on read could be forced explicitly (SELECT
> ...
> > > FOR
> > > > > > > UPDATE)
> > > > > > > 3) Writes do acuire explicit row locks
> > > > > > > 4) Locks are always acquired immediately once statement is
> > executed
> > > > > > > 5) The strictest concurrency level - typically SERIALIZABLE -
> > rely
> > > on
> > > > > > > so-called *range locks* (or *predicate locks*) to track
> > > dependencies
> > > > > > > between transactions. Some vendors throw an exception in case
> of
> > > > > > conflict -
> > > > > > > these are ones where snapshot-based MVCC is used - PostgreSQL,
> > > > Oracle.
> > > > > > > Others do aggressive locking - ones where two-phase locking
> > > algorithm
> > > > > is
> > > > > > > used - SQL Server, MySQL.
> > > > > > >
> > > > > > > As you see, there is no concept of PESSIMISTIC/OPTIMISTIC
> modes.
> > > > > Instead,
> > > > > > > all updates are "PESSIMISTIC", reads are "OPTIMISTIC" but could
> > > > become
> > > > > > > "PESSIMISTIC" if requested explicitly, and for snapshot-based
> > > vendors
> > > > > (we
> > > > > > > are going in this direction) read-write conflicts are resolved
> in
> > > > > manner
> > > > > > > somewhat similar to our OPTIMISTIC/SERIALIZABLE.
> > > > > > >
> > > > > > > That said, I would propose to think on how transactions could
> > look
> > > > like
> > > > > > in
> > > > > > > future Ignite versions (say, 3.0). My rough vision:
> > > > > > >
> > > > > > > 1) No OPTIMISTIC mode at all - too counterintuitive and
> complex.
> > > It's
> > > > > > only
> > > > > > > advantage is deadlock-freedom when combined with SERIALIZABLE.
> If
> > > we
> > > > > have
> > > > > > > good deadlock detector and nice administrative capabilities,
> this
> > > > would
> > > > > > not
> > > > > > > be a problem for us.
> > > > > >
> > > > > >
> > > > > > Hm... The advantage of Optimistic Serialiazable mode is actually
> > > > > lock-free
> > > > > > transactions. The deadlock is impossible in this case. I doubt
> any
> > > > > deadlock
> > > > > > detector would match the performance advantage we get from
> > lock-free
> > > > > > transactions.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > 2) Behavior of reads could be controlled through "with" facade:
> > > > > > > V val1 = cache.get(key1);                 // Shared lock or no
> > lock
> > > > > > > V val2 = cache.withForUpdate().get(key2); // Exclusive lock
> > > > > > >
> > > > > >
> > > > > > Don't like the API. We are not trying to abandon the data grid
> > > use-case
> > > > > or
> > > > > > API, we are trying to add the database use case.
> > > > > >
> > > > > >
> > > > > > > 3) REPEATABLE_READ - throw exception in case of write-write
> > > conflict
> > > > > > >
> > > > > >
> > > > > > Well, I would like to preserve the PESSIMISTIC mode. I find it
> more
> > > > > > convenient than the "withForUpdate" API. It almost seems like you
> > are
> > > > > > trying to force the pendulum too far in the opposite direction.
> > > > > >
> > > > > >
> > > > > > > 4) SERIALIZABLE - throw exception in case of write-write and
> > > > write-read
> > > > > > > confilct (this is how our OPTIMISTIC/SERIALZABLE works now, but
> > it
> > > > > > doesn't
> > > > > > > support predicates)
> > > > > > >
> > > > > >
> > > > > > So, no change here? Good :)
> > > > > >
> > > > > >
> > > > > > > 5) Add READ_ONLY isolation mode where updates will not be
> allowed
> > > at
> > > > > all.
> > > > > > > Such transacrtons would be able to bypass some Ignite internals
> > to
> > > > > > achieve
> > > > > > > greater performance, what could be valuable for mostly-read use
> > > cases
> > > > > > (e.g.
> > > > > > > OLAP).
> > > > > > >
> > > > > >
> > > > > > Love the idea. We have already seen many use cases that could
> > benefit
> > > > > from
> > > > > > it.
> > > > > > How hard is it to implement?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>

[dsetrakyan]

On Thu, Sep 28, 2017 at 10:30 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Dima,
>
> IgniteCache.withReadForUpdate() :-)
>

And how is it better than a pessimistic transaction with read?

[Vladimir Ozerov]

You can mix both "optimistic" and "pessimistic" reads in a single
transaction. This is one of the main points of proposed API. Normally users
do not define blocking behavior on TX level. They do that on per-operation
level.

On Fri, Sep 29, 2017 at 3:30 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Thu, Sep 28, 2017 at 10:30 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Dima,
> >
> > IgniteCache.withReadForUpdate() :-)
> >
>
> And how is it better than a pessimistic transaction with read?
>

[dsetrakyan]

On Fri, Sep 29, 2017 at 5:45 AM, Vladimir Ozerov <[hidden email]>
wrote:

> You can mix both "optimistic" and "pessimistic" reads in a single
> transaction. This is one of the main points of proposed API. Normally users
> do not define blocking behavior on TX level. They do that on per-operation
> level.
>

In that case, why do you suggest the "with" API which will set this flag
for all operations. Why not just add "getWithLock()" method? Also, will
this method work on non-transactional caches?

>
> On Fri, Sep 29, 2017 at 3:30 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > On Thu, Sep 28, 2017 at 10:30 PM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Dima,
> > >
> > > IgniteCache.withReadForUpdate() :-)
> > >
> >
> > And how is it better than a pessimistic transaction with read?
> >
>

[Vladimir Ozerov]

Dima,

My point was that we have a number of read-only methods and I do not want
to pollute base cache API with their counterparts (get, getAll, getEntry,
getEntries). Another point is that "pessimistic" reads is relatively rare
use case comparing to "optimistic". This is why "with" approach looks
better to me.

Blocking reads doesn't make sense outside of explicit transaction.

On Fri, Sep 29, 2017 at 3:47 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Fri, Sep 29, 2017 at 5:45 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > You can mix both "optimistic" and "pessimistic" reads in a single
> > transaction. This is one of the main points of proposed API. Normally
> users
> > do not define blocking behavior on TX level. They do that on
> per-operation
> > level.
> >
>
> In that case, why do you suggest the "with" API which will set this flag
> for all operations. Why not just add "getWithLock()" method? Also, will
> this method work on non-transactional caches?
>
>
> >
> > On Fri, Sep 29, 2017 at 3:30 PM, Dmitriy Setrakyan <
> [hidden email]>
> > wrote:
> >
> > > On Thu, Sep 28, 2017 at 10:30 PM, Vladimir Ozerov <
> [hidden email]>
> > > wrote:
> > >
> > > > Dima,
> > > >
> > > > IgniteCache.withReadForUpdate() :-)
> > > >
> > >
> > > And how is it better than a pessimistic transaction with read?
> > >
> >
>

[dsetrakyan]

I still have a feeling that we are fixing something that was not broken. I
have never heard from any user that they need to do both, blocking and
non-blocking reads in the same transaction.

The only requests I heard so far are:
- snapshot isolation
- read-only transactions

D.

On Fri, Sep 29, 2017 at 6:00 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Dima,
>
> My point was that we have a number of read-only methods and I do not want
> to pollute base cache API with their counterparts (get, getAll, getEntry,
> getEntries). Another point is that "pessimistic" reads is relatively rare
> use case comparing to "optimistic". This is why "with" approach looks
> better to me.
>
> Blocking reads doesn't make sense outside of explicit transaction.
>
> On Fri, Sep 29, 2017 at 3:47 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > On Fri, Sep 29, 2017 at 5:45 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > You can mix both "optimistic" and "pessimistic" reads in a single
> > > transaction. This is one of the main points of proposed API. Normally
> > users
> > > do not define blocking behavior on TX level. They do that on
> > per-operation
> > > level.
> > >
> >
> > In that case, why do you suggest the "with" API which will set this flag
> > for all operations. Why not just add "getWithLock()" method? Also, will
> > this method work on non-transactional caches?
> >
> >
> > >
> > > On Fri, Sep 29, 2017 at 3:30 PM, Dmitriy Setrakyan <
> > [hidden email]>
> > > wrote:
> > >
> > > > On Thu, Sep 28, 2017 at 10:30 PM, Vladimir Ozerov <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Dima,
> > > > >
> > > > > IgniteCache.withReadForUpdate() :-)
> > > > >
> > > >
> > > > And how is it better than a pessimistic transaction with read?
> > > >
> > >
> >
>

[Vladimir Ozerov]

Dima,

I doubt you ever heard from users "we need snapshot isolation" because this
is implementation detail, rather than public behavior :-)
I already explained what we are fixing - broken TX API. 6 possible modes, 5
real modes, 2 broken modes (OPT+RC, OPT+RR), and only *two (!!!)* modes
which are really used in practice (PES+RR, OPT+SER). Do you still think it
is not broken?

On Fri, Sep 29, 2017 at 4:05 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> I still have a feeling that we are fixing something that was not broken. I
> have never heard from any user that they need to do both, blocking and
> non-blocking reads in the same transaction.
>
> The only requests I heard so far are:
> - snapshot isolation
> - read-only transactions
>
> D.
>
> On Fri, Sep 29, 2017 at 6:00 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Dima,
> >
> > My point was that we have a number of read-only methods and I do not want
> > to pollute base cache API with their counterparts (get, getAll, getEntry,
> > getEntries). Another point is that "pessimistic" reads is relatively rare
> > use case comparing to "optimistic". This is why "with" approach looks
> > better to me.
> >
> > Blocking reads doesn't make sense outside of explicit transaction.
> >
> > On Fri, Sep 29, 2017 at 3:47 PM, Dmitriy Setrakyan <
> [hidden email]>
> > wrote:
> >
> > > On Fri, Sep 29, 2017 at 5:45 AM, Vladimir Ozerov <[hidden email]
> >
> > > wrote:
> > >
> > > > You can mix both "optimistic" and "pessimistic" reads in a single
> > > > transaction. This is one of the main points of proposed API. Normally
> > > users
> > > > do not define blocking behavior on TX level. They do that on
> > > per-operation
> > > > level.
> > > >
> > >
> > > In that case, why do you suggest the "with" API which will set this
> flag
> > > for all operations. Why not just add "getWithLock()" method? Also, will
> > > this method work on non-transactional caches?
> > >
> > >
> > > >
> > > > On Fri, Sep 29, 2017 at 3:30 PM, Dmitriy Setrakyan <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > On Thu, Sep 28, 2017 at 10:30 PM, Vladimir Ozerov <
> > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Dima,
> > > > > >
> > > > > > IgniteCache.withReadForUpdate() :-)
> > > > > >
> > > > >
> > > > > And how is it better than a pessimistic transaction with read?
> > > > >
> > > >
> > >
> >
>