Apache Ignite Developers - Legacy Mail Archive

async operation is not fair async

Classic

List

Threaded

33 messages Options

agura

Re: async operation is not fair async

+1 for fair async operations.

But I don't like idea use withFairSync() method. We added xxxAsync()
methods recently and withAsync() is deprecated.

I think we should just make methods are async in nature and provide
ability of switching to the old behaviour using flag or property.

On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
<[hidden email]> wrote:

> Vladimir,
>
> In general I agree, but I do get greatly *close-minded* (pun intended)
> whenever users' code that worked for the past several years all of a sudden
> gets deadlocked after an upgrade. Making this feature optional is even
> worse and more confusing. In this case the best action is no action at all.
>
> BTW, would be interesting to find out how Oracle async driver behaves in
> this case.
>
> D.
>
>
>
>
>
> On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
>> Guys,
>>
>> To build a great product we should be open minded and look to the future,
>> not to the past.
>>
>> Dima raised very valid point - why async is not async? Current programming
>> culture and demanding performance requirements pushes users towards
>> reactive-style programming. I do not want my thread to ever be blocked.
>> Instead, I want to send a number of concurrent commands and optionally
>> subscribe to final result. So trully async API makes total sense to me.
>>
>> But personally, my primary interest in this area is SQL. Oracle is
>> preparing new async driver. ADBA - async database access. It was presented
>> on recent JavaOne [1]. It is under active development right now - juse
>> weave through the mailing list [2]. Some prototypes are already there [3].
>> PostgreSQL community even started adopted it [4]!
>>
>> I am not pushing for immediate actions, but at least we should understand
>> which way the wind is blowing. As a mid-term goals I would propose to
>> finally remove thread ID from our PESSIMISTIC transactions to allow for
>> suspend/resume in different threads. And as a next step I would think on
>> adopting async cache and SQL APIs.
>>
>> Vladimir.
>>
>> [1]
>> http://www.oracle.com/technetwork/database/application-development/jdbc/
>> con1491-3961036.pdf
>> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
>> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
>> [4] https://github.com/pgjdbc/pgjdbc/issues/978
>>
>> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <[hidden email]>
>> wrote:
>>
>> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
>> > [hidden email]> wrote:
>> >
>> > > I will edit IGNITE-8475, and remove all part that belong to the public
>> > api.
>> > > Is it acceptable for you?
>> > >
>> >
>> > Everything is acceptable, as long as the public API is safe :)
>> >
>>

Dmitriy Govorukhin

Re: async operation is not fair async

Andrey,

Do you prefer change behavior at runtime?
I guess will be better have different methods for getting cache instance
with fair and not fair sync.

On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <[hidden email]> wrote:

> +1 for fair async operations.
>
> But I don't like idea use withFairSync() method. We added xxxAsync()
> methods recently and withAsync() is deprecated.
>
> I think we should just make methods are async in nature and provide
> ability of switching to the old behaviour using flag or property.
>
> On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> <[hidden email]> wrote:
> > Vladimir,
> >
> > In general I agree, but I do get greatly *close-minded* (pun intended)
> > whenever users' code that worked for the past several years all of a
> sudden
> > gets deadlocked after an upgrade. Making this feature optional is even
> > worse and more confusing. In this case the best action is no action at
> all.
> >
> > BTW, would be interesting to find out how Oracle async driver behaves in
> > this case.
> >
> > D.
> >
> >
> >
> >
> >
> > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> >> Guys,
> >>
> >> To build a great product we should be open minded and look to the
> future,
> >> not to the past.
> >>
> >> Dima raised very valid point - why async is not async? Current
> programming
> >> culture and demanding performance requirements pushes users towards
> >> reactive-style programming. I do not want my thread to ever be blocked.
> >> Instead, I want to send a number of concurrent commands and optionally
> >> subscribe to final result. So trully async API makes total sense to me.
> >>
> >> But personally, my primary interest in this area is SQL. Oracle is
> >> preparing new async driver. ADBA - async database access. It was
> presented
> >> on recent JavaOne [1]. It is under active development right now - juse
> >> weave through the mailing list [2]. Some prototypes are already there
> [3].
> >> PostgreSQL community even started adopted it [4]!
> >>
> >> I am not pushing for immediate actions, but at least we should
> understand
> >> which way the wind is blowing. As a mid-term goals I would propose to
> >> finally remove thread ID from our PESSIMISTIC transactions to allow for
> >> suspend/resume in different threads. And as a next step I would think on
> >> adopting async cache and SQL APIs.
> >>
> >> Vladimir.
> >>
> >> [1]
> >> http://www.oracle.com/technetwork/database/
> application-development/jdbc/
> >> con1491-3961036.pdf
> >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> >> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
> >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> >>
> >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> [hidden email]>
> >> wrote:
> >>
> >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> >> > [hidden email]> wrote:
> >> >
> >> > > I will edit IGNITE-8475, and remove all part that belong to the
> public
> >> > api.
> >> > > Is it acceptable for you?
> >> > >
> >> >
> >> > Everything is acceptable, as long as the public API is safe :)
> >> >
> >>
>

Alexey Goncharuk

Re: async operation is not fair async

Another +1 for the true asynchronous approach. I remember a while ago one
of the Ignite users raised a similar question regarding the *async method
being blocked on establishing a TCP connection.

As far as deadlocks go, I have a counter-example. Currently, we check the
thread-local chain only for a single cache, so if I run the following code:
cache1.getAsync(k1);
cache2.getAsync(k2);
then the deadlock is still possible, and I did not see a single user
complaining about unexpected deadlocks. Rather than implementing this
cross-cache chain (which would probably add another overhead), I would make
it consistent and allow operations to be run in parallel.

There are many use-cases when having true async operations dramatically
improve performance. Consider, for example, a streaming example when keys
are being pushed by a client to a cluster. Currently, to run effective
processing, the user will have to use a data streamer with custom keys
receiver which may be a huge usability downside. Async operations can
utilize the cluster resources very efficiently.

Finally, if we want to be on the safe side, we can keep the operation chain
inside a transaction. I see absolutely no point in maintaining this chain
outside of transactions.

--AG

2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <[hidden email]>
:

> Andrey,
>
> Do you prefer change behavior at runtime?
> I guess will be better have different methods for getting cache instance
> with fair and not fair sync.
>
> On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <[hidden email]> wrote:
>
> > +1 for fair async operations.
> >
> > But I don't like idea use withFairSync() method. We added xxxAsync()
> > methods recently and withAsync() is deprecated.
> >
> > I think we should just make methods are async in nature and provide
> > ability of switching to the old behaviour using flag or property.
> >
> > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> > <[hidden email]> wrote:
> > > Vladimir,
> > >
> > > In general I agree, but I do get greatly *close-minded* (pun intended)
> > > whenever users' code that worked for the past several years all of a
> > sudden
> > > gets deadlocked after an upgrade. Making this feature optional is even
> > > worse and more confusing. In this case the best action is no action at
> > all.
> > >
> > > BTW, would be interesting to find out how Oracle async driver behaves
> in
> > > this case.
> > >
> > > D.
> > >
> > >
> > >
> > >
> > >
> > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <[hidden email]
> >
> > > wrote:
> > >
> > >> Guys,
> > >>
> > >> To build a great product we should be open minded and look to the
> > future,
> > >> not to the past.
> > >>
> > >> Dima raised very valid point - why async is not async? Current
> > programming
> > >> culture and demanding performance requirements pushes users towards
> > >> reactive-style programming. I do not want my thread to ever be
> blocked.
> > >> Instead, I want to send a number of concurrent commands and optionally
> > >> subscribe to final result. So trully async API makes total sense to
> me.
> > >>
> > >> But personally, my primary interest in this area is SQL. Oracle is
> > >> preparing new async driver. ADBA - async database access. It was
> > presented
> > >> on recent JavaOne [1]. It is under active development right now - juse
> > >> weave through the mailing list [2]. Some prototypes are already there
> > [3].
> > >> PostgreSQL community even started adopted it [4]!
> > >>
> > >> I am not pushing for immediate actions, but at least we should
> > understand
> > >> which way the wind is blowing. As a mid-term goals I would propose to
> > >> finally remove thread ID from our PESSIMISTIC transactions to allow
> for
> > >> suspend/resume in different threads. And as a next step I would think
> on
> > >> adopting async cache and SQL APIs.
> > >>
> > >> Vladimir.
> > >>
> > >> [1]
> > >> http://www.oracle.com/technetwork/database/
> > application-development/jdbc/
> > >> con1491-3961036.pdf
> > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
> > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> > >>
> > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> > [hidden email]>
> > >> wrote:
> > >>
> > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > >> > [hidden email]> wrote:
> > >> >
> > >> > > I will edit IGNITE-8475, and remove all part that belong to the
> > public
> > >> > api.
> > >> > > Is it acceptable for you?
> > >> > >
> > >> >
> > >> > Everything is acceptable, as long as the public API is safe :)
> > >> >
> > >>
> >
>

Dmitriy Govorukhin

Re: async operation is not fair async

Alexey,

Could you please add more description information for this task? [1]
Perhaps, base steps for implementation.

[1] https://issues.apache.org/jira/browse/IGNITE-8475

On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
[hidden email]> wrote:

> Another +1 for the true asynchronous approach. I remember a while ago one
> of the Ignite users raised a similar question regarding the *async method
> being blocked on establishing a TCP connection.
>
> As far as deadlocks go, I have a counter-example. Currently, we check the
> thread-local chain only for a single cache, so if I run the following code:
> cache1.getAsync(k1);
> cache2.getAsync(k2);
> then the deadlock is still possible, and I did not see a single user
> complaining about unexpected deadlocks. Rather than implementing this
> cross-cache chain (which would probably add another overhead), I would make
> it consistent and allow operations to be run in parallel.
>
> There are many use-cases when having true async operations dramatically
> improve performance. Consider, for example, a streaming example when keys
> are being pushed by a client to a cluster. Currently, to run effective
> processing, the user will have to use a data streamer with custom keys
> receiver which may be a huge usability downside. Async operations can
> utilize the cluster resources very efficiently.
>
> Finally, if we want to be on the safe side, we can keep the operation chain
> inside a transaction. I see absolutely no point in maintaining this chain
> outside of transactions.
>
> --AG
>
> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
> [hidden email]>
> :
>
> > Andrey,
> >
> > Do you prefer change behavior at runtime?
> > I guess will be better have different methods for getting cache instance
> > with fair and not fair sync.
> >
> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <[hidden email]> wrote:
> >
> > > +1 for fair async operations.
> > >
> > > But I don't like idea use withFairSync() method. We added xxxAsync()
> > > methods recently and withAsync() is deprecated.
> > >
> > > I think we should just make methods are async in nature and provide
> > > ability of switching to the old behaviour using flag or property.
> > >
> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> > > <[hidden email]> wrote:
> > > > Vladimir,
> > > >
> > > > In general I agree, but I do get greatly *close-minded* (pun
> intended)
> > > > whenever users' code that worked for the past several years all of a
> > > sudden
> > > > gets deadlocked after an upgrade. Making this feature optional is
> even
> > > > worse and more confusing. In this case the best action is no action
> at
> > > all.
> > > >
> > > > BTW, would be interesting to find out how Oracle async driver behaves
> > in
> > > > this case.
> > > >
> > > > D.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > >> Guys,
> > > >>
> > > >> To build a great product we should be open minded and look to the
> > > future,
> > > >> not to the past.
> > > >>
> > > >> Dima raised very valid point - why async is not async? Current
> > > programming
> > > >> culture and demanding performance requirements pushes users towards
> > > >> reactive-style programming. I do not want my thread to ever be
> > blocked.
> > > >> Instead, I want to send a number of concurrent commands and
> optionally
> > > >> subscribe to final result. So trully async API makes total sense to
> > me.
> > > >>
> > > >> But personally, my primary interest in this area is SQL. Oracle is
> > > >> preparing new async driver. ADBA - async database access. It was
> > > presented
> > > >> on recent JavaOne [1]. It is under active development right now -
> juse
> > > >> weave through the mailing list [2]. Some prototypes are already
> there
> > > [3].
> > > >> PostgreSQL community even started adopted it [4]!
> > > >>
> > > >> I am not pushing for immediate actions, but at least we should
> > > understand
> > > >> which way the wind is blowing. As a mid-term goals I would propose
> to
> > > >> finally remove thread ID from our PESSIMISTIC transactions to allow
> > for
> > > >> suspend/resume in different threads. And as a next step I would
> think
> > on
> > > >> adopting async cache and SQL APIs.
> > > >>
> > > >> Vladimir.
> > > >>
> > > >> [1]
> > > >> http://www.oracle.com/technetwork/database/
> > > application-development/jdbc/
> > > >> con1491-3961036.pdf
> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/
> master/java/AoJ
> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> > > >>
> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> > > [hidden email]>
> > > >> wrote:
> > > >>
> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > > >> > [hidden email]> wrote:
> > > >> >
> > > >> > > I will edit IGNITE-8475, and remove all part that belong to the
> > > public
> > > >> > api.
> > > >> > > Is it acceptable for you?
> > > >> > >
> > > >> >
> > > >> > Everything is acceptable, as long as the public API is safe :)
> > > >> >
> > > >>
> > >
> >
>

Dmitriy Govorukhin

Re: async operation is not fair async

Alexey,

Any updates?

On Mon, May 14, 2018 at 6:19 PM, Dmitriy Govorukhin <
[hidden email]> wrote:

> Alexey,
>
> Could you please add more description information for this task? [1]
> Perhaps, base steps for implementation.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8475
>
> On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
> [hidden email]> wrote:
>
>> Another +1 for the true asynchronous approach. I remember a while ago one
>> of the Ignite users raised a similar question regarding the *async method
>> being blocked on establishing a TCP connection.
>>
>> As far as deadlocks go, I have a counter-example. Currently, we check the
>> thread-local chain only for a single cache, so if I run the following
>> code:
>> cache1.getAsync(k1);
>> cache2.getAsync(k2);
>> then the deadlock is still possible, and I did not see a single user
>> complaining about unexpected deadlocks. Rather than implementing this
>> cross-cache chain (which would probably add another overhead), I would
>> make
>> it consistent and allow operations to be run in parallel.
>>
>> There are many use-cases when having true async operations dramatically
>> improve performance. Consider, for example, a streaming example when keys
>> are being pushed by a client to a cluster. Currently, to run effective
>> processing, the user will have to use a data streamer with custom keys
>> receiver which may be a huge usability downside. Async operations can
>> utilize the cluster resources very efficiently.
>>
>> Finally, if we want to be on the safe side, we can keep the operation
>> chain
>> inside a transaction. I see absolutely no point in maintaining this chain
>> outside of transactions.
>>
>> --AG
>>
>> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
>> [hidden email]>
>> :
>>
>> > Andrey,
>> >
>> > Do you prefer change behavior at runtime?
>> > I guess will be better have different methods for getting cache instance
>> > with fair and not fair sync.
>> >
>> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <[hidden email]> wrote:
>> >
>> > > +1 for fair async operations.
>> > >
>> > > But I don't like idea use withFairSync() method. We added xxxAsync()
>> > > methods recently and withAsync() is deprecated.
>> > >
>> > > I think we should just make methods are async in nature and provide
>> > > ability of switching to the old behaviour using flag or property.
>> > >
>> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
>> > > <[hidden email]> wrote:
>> > > > Vladimir,
>> > > >
>> > > > In general I agree, but I do get greatly *close-minded* (pun
>> intended)
>> > > > whenever users' code that worked for the past several years all of a
>> > > sudden
>> > > > gets deadlocked after an upgrade. Making this feature optional is
>> even
>> > > > worse and more confusing. In this case the best action is no action
>> at
>> > > all.
>> > > >
>> > > > BTW, would be interesting to find out how Oracle async driver
>> behaves
>> > in
>> > > > this case.
>> > > >
>> > > > D.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
>> [hidden email]
>> > >
>> > > > wrote:
>> > > >
>> > > >> Guys,
>> > > >>
>> > > >> To build a great product we should be open minded and look to the
>> > > future,
>> > > >> not to the past.
>> > > >>
>> > > >> Dima raised very valid point - why async is not async? Current
>> > > programming
>> > > >> culture and demanding performance requirements pushes users towards
>> > > >> reactive-style programming. I do not want my thread to ever be
>> > blocked.
>> > > >> Instead, I want to send a number of concurrent commands and
>> optionally
>> > > >> subscribe to final result. So trully async API makes total sense to
>> > me.
>> > > >>
>> > > >> But personally, my primary interest in this area is SQL. Oracle is
>> > > >> preparing new async driver. ADBA - async database access. It was
>> > > presented
>> > > >> on recent JavaOne [1]. It is under active development right now -
>> juse
>> > > >> weave through the mailing list [2]. Some prototypes are already
>> there
>> > > [3].
>> > > >> PostgreSQL community even started adopted it [4]!
>> > > >>
>> > > >> I am not pushing for immediate actions, but at least we should
>> > > understand
>> > > >> which way the wind is blowing. As a mid-term goals I would propose
>> to
>> > > >> finally remove thread ID from our PESSIMISTIC transactions to allow
>> > for
>> > > >> suspend/resume in different threads. And as a next step I would
>> think
>> > on
>> > > >> adopting async cache and SQL APIs.
>> > > >>
>> > > >> Vladimir.
>> > > >>
>> > > >> [1]
>> > > >> http://www.oracle.com/technetwork/database/
>> > > application-development/jdbc/
>> > > >> con1491-3961036.pdf
>> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
>> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/
>> java/AoJ
>> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
>> > > >>
>> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
>> > > [hidden email]>
>> > > >> wrote:
>> > > >>
>> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
>> > > >> > [hidden email]> wrote:
>> > > >> >
>> > > >> > > I will edit IGNITE-8475, and remove all part that belong to the
>> > > public
>> > > >> > api.
>> > > >> > > Is it acceptable for you?
>> > > >> > >
>> > > >> >
>> > > >> > Everything is acceptable, as long as the public API is safe :)
>> > > >> >
>> > > >>
>> > >
>> >
>>
>
>

Alexey Goncharuk

Re: async operation is not fair async

Dmitriy,

I will add technical details to the ticket, however, looks like there is
still no consensus on how this change should be presented to a user. It
would be ok if we changed this behavior in Ignite 3.0, but for one of the
next point releases we have to agree how this should be enabled/disabled
(or whether we should delay this change to 3.0 at all).

2018-05-15 22:13 GMT+03:00 Dmitriy Govorukhin <[hidden email]>
:

> Alexey,
>
> Any updates?
>
> On Mon, May 14, 2018 at 6:19 PM, Dmitriy Govorukhin <
> [hidden email]> wrote:
>
> > Alexey,
> >
> > Could you please add more description information for this task? [1]
> > Perhaps, base steps for implementation.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-8475
> >
> > On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
> > [hidden email]> wrote:
> >
> >> Another +1 for the true asynchronous approach. I remember a while ago
> one
> >> of the Ignite users raised a similar question regarding the *async
> method
> >> being blocked on establishing a TCP connection.
> >>
> >> As far as deadlocks go, I have a counter-example. Currently, we check
> the
> >> thread-local chain only for a single cache, so if I run the following
> >> code:
> >> cache1.getAsync(k1);
> >> cache2.getAsync(k2);
> >> then the deadlock is still possible, and I did not see a single user
> >> complaining about unexpected deadlocks. Rather than implementing this
> >> cross-cache chain (which would probably add another overhead), I would
> >> make
> >> it consistent and allow operations to be run in parallel.
> >>
> >> There are many use-cases when having true async operations dramatically
> >> improve performance. Consider, for example, a streaming example when
> keys
> >> are being pushed by a client to a cluster. Currently, to run effective
> >> processing, the user will have to use a data streamer with custom keys
> >> receiver which may be a huge usability downside. Async operations can
> >> utilize the cluster resources very efficiently.
> >>
> >> Finally, if we want to be on the safe side, we can keep the operation
> >> chain
> >> inside a transaction. I see absolutely no point in maintaining this
> chain
> >> outside of transactions.
> >>
> >> --AG
> >>
> >> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
> >> [hidden email]>
> >> :
> >>
> >> > Andrey,
> >> >
> >> > Do you prefer change behavior at runtime?
> >> > I guess will be better have different methods for getting cache
> instance
> >> > with fair and not fair sync.
> >> >
> >> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <[hidden email]>
> wrote:
> >> >
> >> > > +1 for fair async operations.
> >> > >
> >> > > But I don't like idea use withFairSync() method. We added xxxAsync()
> >> > > methods recently and withAsync() is deprecated.
> >> > >
> >> > > I think we should just make methods are async in nature and provide
> >> > > ability of switching to the old behaviour using flag or property.
> >> > >
> >> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> >> > > <[hidden email]> wrote:
> >> > > > Vladimir,
> >> > > >
> >> > > > In general I agree, but I do get greatly *close-minded* (pun
> >> intended)
> >> > > > whenever users' code that worked for the past several years all
> of a
> >> > > sudden
> >> > > > gets deadlocked after an upgrade. Making this feature optional is
> >> even
> >> > > > worse and more confusing. In this case the best action is no
> action
> >> at
> >> > > all.
> >> > > >
> >> > > > BTW, would be interesting to find out how Oracle async driver
> >> behaves
> >> > in
> >> > > > this case.
> >> > > >
> >> > > > D.
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
> >> [hidden email]
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > >> Guys,
> >> > > >>
> >> > > >> To build a great product we should be open minded and look to the
> >> > > future,
> >> > > >> not to the past.
> >> > > >>
> >> > > >> Dima raised very valid point - why async is not async? Current
> >> > > programming
> >> > > >> culture and demanding performance requirements pushes users
> towards
> >> > > >> reactive-style programming. I do not want my thread to ever be
> >> > blocked.
> >> > > >> Instead, I want to send a number of concurrent commands and
> >> optionally
> >> > > >> subscribe to final result. So trully async API makes total sense
> to
> >> > me.
> >> > > >>
> >> > > >> But personally, my primary interest in this area is SQL. Oracle
> is
> >> > > >> preparing new async driver. ADBA - async database access. It was
> >> > > presented
> >> > > >> on recent JavaOne [1]. It is under active development right now -
> >> juse
> >> > > >> weave through the mailing list [2]. Some prototypes are already
> >> there
> >> > > [3].
> >> > > >> PostgreSQL community even started adopted it [4]!
> >> > > >>
> >> > > >> I am not pushing for immediate actions, but at least we should
> >> > > understand
> >> > > >> which way the wind is blowing. As a mid-term goals I would
> propose
> >> to
> >> > > >> finally remove thread ID from our PESSIMISTIC transactions to
> allow
> >> > for
> >> > > >> suspend/resume in different threads. And as a next step I would
> >> think
> >> > on
> >> > > >> adopting async cache and SQL APIs.
> >> > > >>
> >> > > >> Vladimir.
> >> > > >>
> >> > > >> [1]
> >> > > >> http://www.oracle.com/technetwork/database/
> >> > > application-development/jdbc/
> >> > > >> con1491-3961036.pdf
> >> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> >> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/
> >> java/AoJ
> >> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> >> > > >>
> >> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> >> > > [hidden email]>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> >> > > >> > [hidden email]> wrote:
> >> > > >> >
> >> > > >> > > I will edit IGNITE-8475, and remove all part that belong to
> the
> >> > > public
> >> > > >> > api.
> >> > > >> > > Is it acceptable for you?
> >> > > >> > >
> >> > > >> >
> >> > > >> > Everything is acceptable, as long as the public API is safe :)
> >> > > >> >
> >> > > >>
> >> > >
> >> >
> >>
> >
> >
>

Dmitriy Govorukhin

Re: async operation is not fair async

So, in a current implementation, how I can perform the real async operation
in one thread? Any workaround?
What can I do if I have event loop thread model?

On Wed, May 16, 2018 at 12:14 PM, Alexey Goncharuk <
[hidden email]> wrote:

> Dmitriy,
>
> I will add technical details to the ticket, however, looks like there is
> still no consensus on how this change should be presented to a user. It
> would be ok if we changed this behavior in Ignite 3.0, but for one of the
> next point releases we have to agree how this should be enabled/disabled
> (or whether we should delay this change to 3.0 at all).
>
> 2018-05-15 22:13 GMT+03:00 Dmitriy Govorukhin <
> [hidden email]>
> :
>
> > Alexey,
> >
> > Any updates?
> >
> > On Mon, May 14, 2018 at 6:19 PM, Dmitriy Govorukhin <
> > [hidden email]> wrote:
> >
> > > Alexey,
> > >
> > > Could you please add more description information for this task? [1]
> > > Perhaps, base steps for implementation.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-8475
> > >
> > > On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
> > > [hidden email]> wrote:
> > >
> > >> Another +1 for the true asynchronous approach. I remember a while ago
> > one
> > >> of the Ignite users raised a similar question regarding the *async
> > method
> > >> being blocked on establishing a TCP connection.
> > >>
> > >> As far as deadlocks go, I have a counter-example. Currently, we check
> > the
> > >> thread-local chain only for a single cache, so if I run the following
> > >> code:
> > >> cache1.getAsync(k1);
> > >> cache2.getAsync(k2);
> > >> then the deadlock is still possible, and I did not see a single user
> > >> complaining about unexpected deadlocks. Rather than implementing this
> > >> cross-cache chain (which would probably add another overhead), I would
> > >> make
> > >> it consistent and allow operations to be run in parallel.
> > >>
> > >> There are many use-cases when having true async operations
> dramatically
> > >> improve performance. Consider, for example, a streaming example when
> > keys
> > >> are being pushed by a client to a cluster. Currently, to run effective
> > >> processing, the user will have to use a data streamer with custom keys
> > >> receiver which may be a huge usability downside. Async operations can
> > >> utilize the cluster resources very efficiently.
> > >>
> > >> Finally, if we want to be on the safe side, we can keep the operation
> > >> chain
> > >> inside a transaction. I see absolutely no point in maintaining this
> > chain
> > >> outside of transactions.
> > >>
> > >> --AG
> > >>
> > >> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
> > >> [hidden email]>
> > >> :
> > >>
> > >> > Andrey,
> > >> >
> > >> > Do you prefer change behavior at runtime?
> > >> > I guess will be better have different methods for getting cache
> > instance
> > >> > with fair and not fair sync.
> > >> >
> > >> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <[hidden email]>
> > wrote:
> > >> >
> > >> > > +1 for fair async operations.
> > >> > >
> > >> > > But I don't like idea use withFairSync() method. We added
> xxxAsync()
> > >> > > methods recently and withAsync() is deprecated.
> > >> > >
> > >> > > I think we should just make methods are async in nature and
> provide
> > >> > > ability of switching to the old behaviour using flag or property.
> > >> > >
> > >> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> > >> > > <[hidden email]> wrote:
> > >> > > > Vladimir,
> > >> > > >
> > >> > > > In general I agree, but I do get greatly *close-minded* (pun
> > >> intended)
> > >> > > > whenever users' code that worked for the past several years all
> > of a
> > >> > > sudden
> > >> > > > gets deadlocked after an upgrade. Making this feature optional
> is
> > >> even
> > >> > > > worse and more confusing. In this case the best action is no
> > action
> > >> at
> > >> > > all.
> > >> > > >
> > >> > > > BTW, would be interesting to find out how Oracle async driver
> > >> behaves
> > >> > in
> > >> > > > this case.
> > >> > > >
> > >> > > > D.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
> > >> [hidden email]
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Guys,
> > >> > > >>
> > >> > > >> To build a great product we should be open minded and look to
> the
> > >> > > future,
> > >> > > >> not to the past.
> > >> > > >>
> > >> > > >> Dima raised very valid point - why async is not async? Current
> > >> > > programming
> > >> > > >> culture and demanding performance requirements pushes users
> > towards
> > >> > > >> reactive-style programming. I do not want my thread to ever be
> > >> > blocked.
> > >> > > >> Instead, I want to send a number of concurrent commands and
> > >> optionally
> > >> > > >> subscribe to final result. So trully async API makes total
> sense
> > to
> > >> > me.
> > >> > > >>
> > >> > > >> But personally, my primary interest in this area is SQL. Oracle
> > is
> > >> > > >> preparing new async driver. ADBA - async database access. It
> was
> > >> > > presented
> > >> > > >> on recent JavaOne [1]. It is under active development right
> now -
> > >> juse
> > >> > > >> weave through the mailing list [2]. Some prototypes are already
> > >> there
> > >> > > [3].
> > >> > > >> PostgreSQL community even started adopted it [4]!
> > >> > > >>
> > >> > > >> I am not pushing for immediate actions, but at least we should
> > >> > > understand
> > >> > > >> which way the wind is blowing. As a mid-term goals I would
> > propose
> > >> to
> > >> > > >> finally remove thread ID from our PESSIMISTIC transactions to
> > allow
> > >> > for
> > >> > > >> suspend/resume in different threads. And as a next step I would
> > >> think
> > >> > on
> > >> > > >> adopting async cache and SQL APIs.
> > >> > > >>
> > >> > > >> Vladimir.
> > >> > > >>
> > >> > > >> [1]
> > >> > > >> http://www.oracle.com/technetwork/database/
> > >> > > application-development/jdbc/
> > >> > > >> con1491-3961036.pdf
> > >> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> > >> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/
> > >> java/AoJ
> > >> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> > >> > > >>
> > >> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> > >> > > [hidden email]>
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > >> > > >> > [hidden email]> wrote:
> > >> > > >> >
> > >> > > >> > > I will edit IGNITE-8475, and remove all part that belong to
> > the
> > >> > > public
> > >> > > >> > api.
> > >> > > >> > > Is it acceptable for you?
> > >> > > >> > >
> > >> > > >> >
> > >> > > >> > Everything is acceptable, as long as the public API is safe
> :)
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

yzhdanov

Re: async operation is not fair async

In reply to this post by Alexey Goncharuk

Alexey Goncharuk, I remember we started working on async connection
establishment. This should fix latency issue related to network which I
believe gives the most contribution to overall latency. Mapping logic and
other stuff can be ignored as it can very rarely be an issue at least on
stable tolopogies. What is the status with async connections? That would
really be a huge improvement!

Also please remember that uncontrolled async operations may lead to OOME,
therefore at some point when there are too many uncompleted async
operations newly invoked async operations should become synchronous, i.e.
we should return completed future, ignoring the fact that user expected us
to be async.

I would like to have very strong reasons to start reapproaching this.

--Yakov

Dmitriy Pavlov

Re: async operation is not fair async

Igniters,

I've re-read this thread in brief. As far as I know this change is not
coming from some company, so this change will be at least good for healthy
community building.

And I didn't find any obstacles why not to implement approach with new mode
.withFairAsync() for cases user is completely aware of consequences.

It could be not public API to avoid anyone will use it. It could be
used,e.g. in integrations and by qualified users to gain as much
throutghput as it is possible.

So I would like to be an sponsor here. If anyone or Dmitriy G. will
contribute this change, I will merge it. I hope PMCs are agree here and
will not veto this change.

Sincerely,
Dmitriy Pavlov

чт, 24 мая 2018 г. в 15:13, Yakov Zhdanov <[hidden email]>:

> Alexey Goncharuk, I remember we started working on async connection
> establishment. This should fix latency issue related to network which I
> believe gives the most contribution to overall latency. Mapping logic and
> other stuff can be ignored as it can very rarely be an issue at least on
> stable tolopogies. What is the status with async connections? That would
> really be a huge improvement!
>
> Also please remember that uncontrolled async operations may lead to OOME,
> therefore at some point when there are too many uncompleted async
> operations newly invoked async operations should become synchronous, i.e.
> we should return completed future, ignoring the fact that user expected us
> to be async.
>
> I would like to have very strong reasons to start reapproaching this.
>
> --Yakov
>

Dmitriy Govorukhin

Re: async operation is not fair async

Folks,
Any comments?
I will start to implement withFairAsync(); decorator soon.

On Wed, Aug 1, 2018 at 12:22 PM Dmitriy Pavlov <[hidden email]>
wrote:

> Igniters,
>
> I've re-read this thread in brief. As far as I know this change is not
> coming from some company, so this change will be at least good for healthy
> community building.
>
> And I didn't find any obstacles why not to implement approach with new mode
> .withFairAsync() for cases user is completely aware of consequences.
>
> It could be not public API to avoid anyone will use it. It could be
> used,e.g. in integrations and by qualified users to gain as much
> throutghput as it is possible.
>
> So I would like to be an sponsor here. If anyone or Dmitriy G. will
> contribute this change, I will merge it. I hope PMCs are agree here and
> will not veto this change.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 24 мая 2018 г. в 15:13, Yakov Zhdanov <[hidden email]>:
>
> > Alexey Goncharuk, I remember we started working on async connection
> > establishment. This should fix latency issue related to network which I
> > believe gives the most contribution to overall latency. Mapping logic and
> > other stuff can be ignored as it can very rarely be an issue at least on
> > stable tolopogies. What is the status with async connections? That would
> > really be a huge improvement!
> >
> > Also please remember that uncontrolled async operations may lead to OOME,
> > therefore at some point when there are too many uncompleted async
> > operations newly invoked async operations should become synchronous, i.e.
> > we should return completed future, ignoring the fact that user expected
> us
> > to be async.
> >
> > I would like to have very strong reasons to start reapproaching this.
> >
> > --Yakov
> >
>

Deepa Kolwalkar

can Ignite Cache automatically detect changes in multiple Underlying DBs and stay in sync with them

Hi

While searching for a solution on how to have DB updates to reflect
automatically in Ignite Cache, I came across this post dt Mar-2018 which
states that :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html

Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes

Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

Dmitriy Pavlov

Re: can Ignite Cache automatically detect changes in multiple Underlying DBs and stay in sync with them

Hi,

if this question is not related to some contribution than, probably, better
place to ask is [hidden email] mailing list.

Sincerely,
Dmitriy Pavlov

пт, 3 авг. 2018 г. в 11:44, Deepa Kolwalkar <[hidden email]>:

> Hi
>
> While searching for a solution on how to have DB updates to reflect
> automatically in Ignite Cache, I came across this post dt Mar-2018 which
> states that :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> https://docs.gridgain.com/docs/goldengate-replication
> ==============
>
> http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
>

dmagda

Re: can Ignite Cache automatically detect changes in multiple Underlying DBs and stay in sync with them

In reply to this post by Deepa Kolwalkar

Hi Deepa,

What is your use case? There might be a solution.

--
Denis

On Fri, Aug 3, 2018 at 1:44 AM Deepa Kolwalkar <[hidden email]>
wrote: