affinityCall in one distributed transaction

[Игорь Г]

Hi, igniters!

Before openning JIRA ticket, I want to ask question about affinityCall or
affinityRun transactions.

For example I have batch task to modify many values in someCache according
to someRule. I want to parallel this task to whole cluster and minimize
network traffic.
So the resonable choice is affinityCall feature.

But I want all this changes to be in one transactoin. i.e. with at least
atomicity property (of ACID). And if for some reason my task will be
canceled or failed on one node - it should change nothing at all.

So, can I achieve this with existing functionality, or how can I approach
to this task?

[Taras Ledkov]

Ignite compute has no relation to the cache's transaction.

I think that IgniteCache.invokeAll() is appropriate for described case.

On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]> wrote:

> Hi, igniters!
>
> Before openning JIRA ticket, I want to ask question about affinityCall or
> affinityRun transactions.
>
> For example I have batch task to modify many values in someCache according
> to someRule. I want to parallel this task to whole cluster and minimize
> network traffic.
> So the resonable choice is affinityCall feature.
>
> But I want all this changes to be in one transactoin. i.e. with at least
> atomicity property (of ACID). And if for some reason my task will be
> canceled or failed on one node - it should change nothing at all.
>
> So, can I achieve this with existing functionality, or how can I approach
> to this task?
>

[dsetrakyan]

Taras, is invokeAll() transactional? The javadoc is silent to this fact. If
it is indeed transactional, then we should update the javadoc.

D.

On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <[hidden email]> wrote:

> Ignite compute has no relation to the cache's transaction.
>
> I think that IgniteCache.invokeAll() is appropriate for described case.
>
> On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]> wrote:
>
> > Hi, igniters!
> >
> > Before openning JIRA ticket, I want to ask question about affinityCall or
> > affinityRun transactions.
> >
> > For example I have batch task to modify many values in someCache
> according
> > to someRule. I want to parallel this task to whole cluster and minimize
> > network traffic.
> > So the resonable choice is affinityCall feature.
> >
> > But I want all this changes to be in one transactoin. i.e. with at least
> > atomicity property (of ACID). And if for some reason my task will be
> > canceled or failed on one node - it should change nothing at all.
> >
> > So, can I achieve this with existing functionality, or how can I approach
> > to this task?
> >
>

[Alexei Scherbakov]

Hi.

It's unclear from your description what are you trying to achieve.

AffinityCall is unicast and wil be send to single node.

To parallelise task among the cluster I would recommend to use compute task
API [1]

But the task execution is not transactional. Nevertheless, each job
triggered by task can use it's own local transaction.

And please explain, why can't you use a generic Ignite transaction for you
task?

[1] https://apacheignite.readme.io/docs/compute-tasks

2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Taras, is invokeAll() transactional? The javadoc is silent to this fact. If
> it is indeed transactional, then we should update the javadoc.
>
> D.
>
> On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <[hidden email]> wrote:
>
> > Ignite compute has no relation to the cache's transaction.
> >
> > I think that IgniteCache.invokeAll() is appropriate for described case.
> >
> > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]> wrote:
> >
> > > Hi, igniters!
> > >
> > > Before openning JIRA ticket, I want to ask question about affinityCall
> or
> > > affinityRun transactions.
> > >
> > > For example I have batch task to modify many values in someCache
> > according
> > > to someRule. I want to parallel this task to whole cluster and minimize
> > > network traffic.
> > > So the resonable choice is affinityCall feature.
> > >
> > > But I want all this changes to be in one transactoin. i.e. with at
> least
> > > atomicity property (of ACID). And if for some reason my task will be
> > > canceled or failed on one node - it should change nothing at all.
> > >
> > > So, can I achieve this with existing functionality, or how can I
> approach
> > > to this task?
> > >
> >
>

--

Best regards,
Alexei Scherbakov

[dsetrakyan]

Just clarified that all invoke(..) and invokeAll(..) methods are
transactional. Can someone familiar with these methods update the javadoc?

Igor, will invoke(..) work for your use case?

D.

On Wed, Dec 7, 2016 at 5:09 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Taras, is invokeAll() transactional? The javadoc is silent to this fact.
> If it is indeed transactional, then we should update the javadoc.
>
> D.
>
> On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <[hidden email]> wrote:
>
>> Ignite compute has no relation to the cache's transaction.
>>
>> I think that IgniteCache.invokeAll() is appropriate for described case.
>>
>> On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]> wrote:
>>
>> > Hi, igniters!
>> >
>> > Before openning JIRA ticket, I want to ask question about affinityCall
>> or
>> > affinityRun transactions.
>> >
>> > For example I have batch task to modify many values in someCache
>> according
>> > to someRule. I want to parallel this task to whole cluster and minimize
>> > network traffic.
>> > So the resonable choice is affinityCall feature.
>> >
>> > But I want all this changes to be in one transactoin. i.e. with at least
>> > atomicity property (of ACID). And if for some reason my task will be
>> > canceled or failed on one node - it should change nothing at all.
>> >
>> > So, can I achieve this with existing functionality, or how can I
>> approach
>> > to this task?
>> >
>>
>
>

[Антон Чураев]

In some cases it is necessary to implement a transaction processing logic
in several different application servers. In this case, working with Ignite
cache will be performed within the various applications. But all these
changes must be made within the same distributed transaction.

In my opinion this will require context transfer between the threads within
a single node or multiple Ignite nodes.

2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <[hidden email]>:

> Hi.
>
> It's unclear from your description what are you trying to achieve.
>
> AffinityCall is unicast and wil be send to single node.
>
> To parallelise task among the cluster I would recommend to use compute task
> API [1]
>
> But the task execution is not transactional. Nevertheless, each job
> triggered by task can use it's own local transaction.
>
> And please explain, why can't you use a generic Ignite transaction for you
> task?
>
> [1] https://apacheignite.readme.io/docs/compute-tasks
>
> 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Taras, is invokeAll() transactional? The javadoc is silent to this fact.
> If
> > it is indeed transactional, then we should update the javadoc.
> >
> > D.
> >
> > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <[hidden email]>
> wrote:
> >
> > > Ignite compute has no relation to the cache's transaction.
> > >
> > > I think that IgniteCache.invokeAll() is appropriate for described case.
> > >
> > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]> wrote:
> > >
> > > > Hi, igniters!
> > > >
> > > > Before openning JIRA ticket, I want to ask question about
> affinityCall
> > or
> > > > affinityRun transactions.
> > > >
> > > > For example I have batch task to modify many values in someCache
> > > according
> > > > to someRule. I want to parallel this task to whole cluster and
> minimize
> > > > network traffic.
> > > > So the resonable choice is affinityCall feature.
> > > >
> > > > But I want all this changes to be in one transactoin. i.e. with at
> > least
> > > > atomicity property (of ACID). And if for some reason my task will be
> > > > canceled or failed on one node - it should change nothing at all.
> > > >
> > > > So, can I achieve this with existing functionality, or how can I
> > approach
> > > > to this task?
> > > >
> > >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

--
С уважением,
Чураев Антон

[dsetrakyan]

Sounds like you need a centralized JTA server for this type of purpose, no?
In that case, Ignite transactions can already merge into ongoing JTA
transactions.

I would prefer to see a distributed flow of events to fully understand the
issue. For example,

Client
  - start transaction
  - send compute

Server
  - receive compute
  - execute ...
  - execute ...

etc.

D.

On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <[hidden email]> wrote:

> In some cases it is necessary to implement a transaction processing logic
> in several different application servers. In this case, working with Ignite
> cache will be performed within the various applications. But all these
> changes must be made within the same distributed transaction.
>
> In my opinion this will require context transfer between the threads within
> a single node or multiple Ignite nodes.
>
> 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <[hidden email]
> >:
>
> > Hi.
> >
> > It's unclear from your description what are you trying to achieve.
> >
> > AffinityCall is unicast and wil be send to single node.
> >
> > To parallelise task among the cluster I would recommend to use compute
> task
> > API [1]
> >
> > But the task execution is not transactional. Nevertheless, each job
> > triggered by task can use it's own local transaction.
> >
> > And please explain, why can't you use a generic Ignite transaction for
> you
> > task?
> >
> > [1] https://apacheignite.readme.io/docs/compute-tasks
> >
> > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > Taras, is invokeAll() transactional? The javadoc is silent to this
> fact.
> > If
> > > it is indeed transactional, then we should update the javadoc.
> > >
> > > D.
> > >
> > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <[hidden email]>
> > wrote:
> > >
> > > > Ignite compute has no relation to the cache's transaction.
> > > >
> > > > I think that IgniteCache.invokeAll() is appropriate for described
> case.
> > > >
> > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]> wrote:
> > > >
> > > > > Hi, igniters!
> > > > >
> > > > > Before openning JIRA ticket, I want to ask question about
> > affinityCall
> > > or
> > > > > affinityRun transactions.
> > > > >
> > > > > For example I have batch task to modify many values in someCache
> > > > according
> > > > > to someRule. I want to parallel this task to whole cluster and
> > minimize
> > > > > network traffic.
> > > > > So the resonable choice is affinityCall feature.
> > > > >
> > > > > But I want all this changes to be in one transactoin. i.e. with at
> > > least
> > > > > atomicity property (of ACID). And if for some reason my task will
> be
> > > > > canceled or failed on one node - it should change nothing at all.
> > > > >
> > > > > So, can I achieve this with existing functionality, or how can I
> > > approach
> > > > > to this task?
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>
>
>
> --
> С уважением,
> Чураев Антон
>

[Антон Чураев]

For example:
1) Front-end sends a request to perform a complex transaction.
2) Some application (like a business transactional coordinator) receives
message via asynchronous transport. This application implements logic of
calling different services sequentially or in parallel via asynchronous
transport.
3) Each service implement some little changes in a cache.

Or:
1) Front-end sends a request to perform a complex transaction.
2) This transaction is implemented in microservice architecture (large
number microservices + asynchronous transport).
3) Each microservice implement some little changes in a cache.

I think it is possible to implement distributed transactional using XA
coordinator outside Ignite and local transaction in each service. But its
cost may be unacceptable especially in the case of using a large number of
services.

I think distributed transaction inside Ignite could be useful also for
using multiple ComputeTask in one transaction.

2016-12-09 21:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Sounds like you need a centralized JTA server for this type of purpose, no?
> In that case, Ignite transactions can already merge into ongoing JTA
> transactions.
>
> I would prefer to see a distributed flow of events to fully understand the
> issue. For example,
>
> Client
>   - start transaction
>   - send compute
>
> Server
>   - receive compute
>   - execute ...
>   - execute ...
>
> etc.
>
> D.
>
> On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <[hidden email]> wrote:
>
> > In some cases it is necessary to implement a transaction processing logic
> > in several different application servers. In this case, working with
> Ignite
> > cache will be performed within the various applications. But all these
> > changes must be made within the same distributed transaction.
> >
> > In my opinion this will require context transfer between the threads
> within
> > a single node or multiple Ignite nodes.
> >
> > 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <
> [hidden email]
> > >:
> >
> > > Hi.
> > >
> > > It's unclear from your description what are you trying to achieve.
> > >
> > > AffinityCall is unicast and wil be send to single node.
> > >
> > > To parallelise task among the cluster I would recommend to use compute
> > task
> > > API [1]
> > >
> > > But the task execution is not transactional. Nevertheless, each job
> > > triggered by task can use it's own local transaction.
> > >
> > > And please explain, why can't you use a generic Ignite transaction for
> > you
> > > task?
> > >
> > > [1] https://apacheignite.readme.io/docs/compute-tasks
> > >
> > > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > >
> > > > Taras, is invokeAll() transactional? The javadoc is silent to this
> > fact.
> > > If
> > > > it is indeed transactional, then we should update the javadoc.
> > > >
> > > > D.
> > > >
> > > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <[hidden email]>
> > > wrote:
> > > >
> > > > > Ignite compute has no relation to the cache's transaction.
> > > > >
> > > > > I think that IgniteCache.invokeAll() is appropriate for described
> > case.
> > > > >
> > > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]> wrote:
> > > > >
> > > > > > Hi, igniters!
> > > > > >
> > > > > > Before openning JIRA ticket, I want to ask question about
> > > affinityCall
> > > > or
> > > > > > affinityRun transactions.
> > > > > >
> > > > > > For example I have batch task to modify many values in someCache
> > > > > according
> > > > > > to someRule. I want to parallel this task to whole cluster and
> > > minimize
> > > > > > network traffic.
> > > > > > So the resonable choice is affinityCall feature.
> > > > > >
> > > > > > But I want all this changes to be in one transactoin. i.e. with
> at
> > > > least
> > > > > > atomicity property (of ACID). And if for some reason my task will
> > be
> > > > > > canceled or failed on one node - it should change nothing at all.
> > > > > >
> > > > > > So, can I achieve this with existing functionality, or how can I
> > > > approach
> > > > > > to this task?
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
> >
> >
> > --
> > С уважением,
> > Чураев Антон
> >
>

--
С уважением,
Чураев Антон

[dsetrakyan]

Anton,

Thanks for the explanation. I am sorry to keep asking questions on this.
Can you change your example to include concrete Ignite calls on Compute or
Cache APIs (or other APIs)? I am still struggling to understand the
boundaries between business and Ignite logic.

D.

On Sat, Dec 10, 2016 at 5:46 AM, Антон Чураев <[hidden email]> wrote:

> For example:
> 1) Front-end sends a request to perform a complex transaction.
> 2) Some application (like a business transactional coordinator) receives
> message via asynchronous transport. This application implements logic of
> calling different services sequentially or in parallel via asynchronous
> transport.
> 3) Each service implement some little changes in a cache.
>
> Or:
> 1) Front-end sends a request to perform a complex transaction.
> 2) This transaction is implemented in microservice architecture (large
> number microservices + asynchronous transport).
> 3) Each microservice implement some little changes in a cache.
>
> I think it is possible to implement distributed transactional using XA
> coordinator outside Ignite and local transaction in each service. But its
> cost may be unacceptable especially in the case of using a large number of
> services.
>
> I think distributed transaction inside Ignite could be useful also for
> using multiple ComputeTask in one transaction.
>
> 2016-12-09 21:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Sounds like you need a centralized JTA server for this type of purpose,
> no?
> > In that case, Ignite transactions can already merge into ongoing JTA
> > transactions.
> >
> > I would prefer to see a distributed flow of events to fully understand
> the
> > issue. For example,
> >
> > Client
> >   - start transaction
> >   - send compute
> >
> > Server
> >   - receive compute
> >   - execute ...
> >   - execute ...
> >
> > etc.
> >
> > D.
> >
> > On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <[hidden email]>
> wrote:
> >
> > > In some cases it is necessary to implement a transaction processing
> logic
> > > in several different application servers. In this case, working with
> > Ignite
> > > cache will be performed within the various applications. But all these
> > > changes must be made within the same distributed transaction.
> > >
> > > In my opinion this will require context transfer between the threads
> > within
> > > a single node or multiple Ignite nodes.
> > >
> > > 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <
> > [hidden email]
> > > >:
> > >
> > > > Hi.
> > > >
> > > > It's unclear from your description what are you trying to achieve.
> > > >
> > > > AffinityCall is unicast and wil be send to single node.
> > > >
> > > > To parallelise task among the cluster I would recommend to use
> compute
> > > task
> > > > API [1]
> > > >
> > > > But the task execution is not transactional. Nevertheless, each job
> > > > triggered by task can use it's own local transaction.
> > > >
> > > > And please explain, why can't you use a generic Ignite transaction
> for
> > > you
> > > > task?
> > > >
> > > > [1] https://apacheignite.readme.io/docs/compute-tasks
> > > >
> > > > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > > >
> > > > > Taras, is invokeAll() transactional? The javadoc is silent to this
> > > fact.
> > > > If
> > > > > it is indeed transactional, then we should update the javadoc.
> > > > >
> > > > > D.
> > > > >
> > > > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <[hidden email]
> >
> > > > wrote:
> > > > >
> > > > > > Ignite compute has no relation to the cache's transaction.
> > > > > >
> > > > > > I think that IgniteCache.invokeAll() is appropriate for described
> > > case.
> > > > > >
> > > > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]>
> wrote:
> > > > > >
> > > > > > > Hi, igniters!
> > > > > > >
> > > > > > > Before openning JIRA ticket, I want to ask question about
> > > > affinityCall
> > > > > or
> > > > > > > affinityRun transactions.
> > > > > > >
> > > > > > > For example I have batch task to modify many values in
> someCache
> > > > > > according
> > > > > > > to someRule. I want to parallel this task to whole cluster and
> > > > minimize
> > > > > > > network traffic.
> > > > > > > So the resonable choice is affinityCall feature.
> > > > > > >
> > > > > > > But I want all this changes to be in one transactoin. i.e. with
> > at
> > > > > least
> > > > > > > atomicity property (of ACID). And if for some reason my task
> will
> > > be
> > > > > > > canceled or failed on one node - it should change nothing at
> all.
> > > > > > >
> > > > > > > So, can I achieve this with existing functionality, or how can
> I
> > > > > approach
> > > > > > > to this task?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Alexei Scherbakov
> > > >
> > >
> > >
> > >
> > > --
> > > С уважением,
> > > Чураев Антон
> > >
> >
>
>
>
> --
> С уважением,
> Чураев Антон
>

[Антон Чураев]

Dmitriy, it's ok

To be abstract simple business transaction for execution payment
(preparation done before)  from the card looks like:
1) Create a payment document (cache API);
2) Write-off funds from the payer's card;
2.1) Change in register #1 (cache API);
2.2) Change in register #2 (cache API);
2.3) Change in register #... (cache API);
2.4) Change limits of card (cache API);
3) Payment to service provider;
3.1) Change in the balance of business unit having the contract with payer
(cache API);
3.2) Change in the balance of business unit having the contract with
provider (cache API);
3.3) Change in the balance of the account (cache API);
3.4.1) Some change in registers... (cache API);
3.4.2) ...;
3.3) ...
3.4) Invoke provider's API for billing payment of payer;
4) Formation of financial statements (it's possible to implement off-line -
in other transational)
4.1) ...
4.2) ...

And all steps have been designed into separate microservices. And, of
course, it have been designed via asynchronous transport.
On the other hand it seems to be that implementation of coordination of
10-20 local transactions is not a good idea

2016-12-10 23:30 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Anton,
>
> Thanks for the explanation. I am sorry to keep asking questions on this.
> Can you change your example to include concrete Ignite calls on Compute or
> Cache APIs (or other APIs)? I am still struggling to understand the
> boundaries between business and Ignite logic.
>
> D.
>
> On Sat, Dec 10, 2016 at 5:46 AM, Антон Чураев <[hidden email]>
> wrote:
>
> > For example:
> > 1) Front-end sends a request to perform a complex transaction.
> > 2) Some application (like a business transactional coordinator) receives
> > message via asynchronous transport. This application implements logic of
> > calling different services sequentially or in parallel via asynchronous
> > transport.
> > 3) Each service implement some little changes in a cache.
> >
> > Or:
> > 1) Front-end sends a request to perform a complex transaction.
> > 2) This transaction is implemented in microservice architecture (large
> > number microservices + asynchronous transport).
> > 3) Each microservice implement some little changes in a cache.
> >
> > I think it is possible to implement distributed transactional using XA
> > coordinator outside Ignite and local transaction in each service. But its
> > cost may be unacceptable especially in the case of using a large number
> of
> > services.
> >
> > I think distributed transaction inside Ignite could be useful also for
> > using multiple ComputeTask in one transaction.
> >
> > 2016-12-09 21:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > Sounds like you need a centralized JTA server for this type of purpose,
> > no?
> > > In that case, Ignite transactions can already merge into ongoing JTA
> > > transactions.
> > >
> > > I would prefer to see a distributed flow of events to fully understand
> > the
> > > issue. For example,
> > >
> > > Client
> > >   - start transaction
> > >   - send compute
> > >
> > > Server
> > >   - receive compute
> > >   - execute ...
> > >   - execute ...
> > >
> > > etc.
> > >
> > > D.
> > >
> > > On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <[hidden email]>
> > wrote:
> > >
> > > > In some cases it is necessary to implement a transaction processing
> > logic
> > > > in several different application servers. In this case, working with
> > > Ignite
> > > > cache will be performed within the various applications. But all
> these
> > > > changes must be made within the same distributed transaction.
> > > >
> > > > In my opinion this will require context transfer between the threads
> > > within
> > > > a single node or multiple Ignite nodes.
> > > >
> > > > 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <
> > > [hidden email]
> > > > >:
> > > >
> > > > > Hi.
> > > > >
> > > > > It's unclear from your description what are you trying to achieve.
> > > > >
> > > > > AffinityCall is unicast and wil be send to single node.
> > > > >
> > > > > To parallelise task among the cluster I would recommend to use
> > compute
> > > > task
> > > > > API [1]
> > > > >
> > > > > But the task execution is not transactional. Nevertheless, each job
> > > > > triggered by task can use it's own local transaction.
> > > > >
> > > > > And please explain, why can't you use a generic Ignite transaction
> > for
> > > > you
> > > > > task?
> > > > >
> > > > > [1] https://apacheignite.readme.io/docs/compute-tasks
> > > > >
> > > > > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <[hidden email]
> >:
> > > > >
> > > > > > Taras, is invokeAll() transactional? The javadoc is silent to
> this
> > > > fact.
> > > > > If
> > > > > > it is indeed transactional, then we should update the javadoc.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <
> [hidden email]
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Ignite compute has no relation to the cache's transaction.
> > > > > > >
> > > > > > > I think that IgniteCache.invokeAll() is appropriate for
> described
> > > > case.
> > > > > > >
> > > > > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]>
> > wrote:
> > > > > > >
> > > > > > > > Hi, igniters!
> > > > > > > >
> > > > > > > > Before openning JIRA ticket, I want to ask question about
> > > > > affinityCall
> > > > > > or
> > > > > > > > affinityRun transactions.
> > > > > > > >
> > > > > > > > For example I have batch task to modify many values in
> > someCache
> > > > > > > according
> > > > > > > > to someRule. I want to parallel this task to whole cluster
> and
> > > > > minimize
> > > > > > > > network traffic.
> > > > > > > > So the resonable choice is affinityCall feature.
> > > > > > > >
> > > > > > > > But I want all this changes to be in one transactoin. i.e.
> with
> > > at
> > > > > > least
> > > > > > > > atomicity property (of ACID). And if for some reason my task
> > will
> > > > be
> > > > > > > > canceled or failed on one node - it should change nothing at
> > all.
> > > > > > > >
> > > > > > > > So, can I achieve this with existing functionality, or how
> can
> > I
> > > > > > approach
> > > > > > > > to this task?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best regards,
> > > > > Alexei Scherbakov
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > С уважением,
> > > > Чураев Антон
> > > >
> > >
> >
> >
> >
> > --
> > С уважением,
> > Чураев Антон
> >
>

--
С уважением,
Чураев Антон

[dsetrakyan]

Anton,

Looking at this sequence, I don't see any way of achieving it other than
enrolling all transactions into one JTA transaction. If you seen another
way, can you please suggest it here?

D.

On Sat, Dec 10, 2016 at 2:07 PM, Антон Чураев <[hidden email]> wrote:

> Dmitriy, it's ok
>
> To be abstract simple business transaction for execution payment
> (preparation done before)  from the card looks like:
> 1) Create a payment document (cache API);
> 2) Write-off funds from the payer's card;
> 2.1) Change in register #1 (cache API);
> 2.2) Change in register #2 (cache API);
> 2.3) Change in register #... (cache API);
> 2.4) Change limits of card (cache API);
> 3) Payment to service provider;
> 3.1) Change in the balance of business unit having the contract with payer
> (cache API);
> 3.2) Change in the balance of business unit having the contract with
> provider (cache API);
> 3.3) Change in the balance of the account (cache API);
> 3.4.1) Some change in registers... (cache API);
> 3.4.2) ...;
> 3.3) ...
> 3.4) Invoke provider's API for billing payment of payer;
> 4) Formation of financial statements (it's possible to implement off-line -
> in other transational)
> 4.1) ...
> 4.2) ...
>
> And all steps have been designed into separate microservices. And, of
> course, it have been designed via asynchronous transport.
> On the other hand it seems to be that implementation of coordination of
> 10-20 local transactions is not a good idea
>
> 2016-12-10 23:30 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Anton,
> >
> > Thanks for the explanation. I am sorry to keep asking questions on this.
> > Can you change your example to include concrete Ignite calls on Compute
> or
> > Cache APIs (or other APIs)? I am still struggling to understand the
> > boundaries between business and Ignite logic.
> >
> > D.
> >
> > On Sat, Dec 10, 2016 at 5:46 AM, Антон Чураев <[hidden email]>
> > wrote:
> >
> > > For example:
> > > 1) Front-end sends a request to perform a complex transaction.
> > > 2) Some application (like a business transactional coordinator)
> receives
> > > message via asynchronous transport. This application implements logic
> of
> > > calling different services sequentially or in parallel via asynchronous
> > > transport.
> > > 3) Each service implement some little changes in a cache.
> > >
> > > Or:
> > > 1) Front-end sends a request to perform a complex transaction.
> > > 2) This transaction is implemented in microservice architecture (large
> > > number microservices + asynchronous transport).
> > > 3) Each microservice implement some little changes in a cache.
> > >
> > > I think it is possible to implement distributed transactional using XA
> > > coordinator outside Ignite and local transaction in each service. But
> its
> > > cost may be unacceptable especially in the case of using a large number
> > of
> > > services.
> > >
> > > I think distributed transaction inside Ignite could be useful also for
> > > using multiple ComputeTask in one transaction.
> > >
> > > 2016-12-09 21:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > >
> > > > Sounds like you need a centralized JTA server for this type of
> purpose,
> > > no?
> > > > In that case, Ignite transactions can already merge into ongoing JTA
> > > > transactions.
> > > >
> > > > I would prefer to see a distributed flow of events to fully
> understand
> > > the
> > > > issue. For example,
> > > >
> > > > Client
> > > >   - start transaction
> > > >   - send compute
> > > >
> > > > Server
> > > >   - receive compute
> > > >   - execute ...
> > > >   - execute ...
> > > >
> > > > etc.
> > > >
> > > > D.
> > > >
> > > > On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <[hidden email]>
> > > wrote:
> > > >
> > > > > In some cases it is necessary to implement a transaction processing
> > > logic
> > > > > in several different application servers. In this case, working
> with
> > > > Ignite
> > > > > cache will be performed within the various applications. But all
> > these
> > > > > changes must be made within the same distributed transaction.
> > > > >
> > > > > In my opinion this will require context transfer between the
> threads
> > > > within
> > > > > a single node or multiple Ignite nodes.
> > > > >
> > > > > 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <
> > > > [hidden email]
> > > > > >:
> > > > >
> > > > > > Hi.
> > > > > >
> > > > > > It's unclear from your description what are you trying to
> achieve.
> > > > > >
> > > > > > AffinityCall is unicast and wil be send to single node.
> > > > > >
> > > > > > To parallelise task among the cluster I would recommend to use
> > > compute
> > > > > task
> > > > > > API [1]
> > > > > >
> > > > > > But the task execution is not transactional. Nevertheless, each
> job
> > > > > > triggered by task can use it's own local transaction.
> > > > > >
> > > > > > And please explain, why can't you use a generic Ignite
> transaction
> > > for
> > > > > you
> > > > > > task?
> > > > > >
> > > > > > [1] https://apacheignite.readme.io/docs/compute-tasks
> > > > > >
> > > > > > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <
> [hidden email]
> > >:
> > > > > >
> > > > > > > Taras, is invokeAll() transactional? The javadoc is silent to
> > this
> > > > > fact.
> > > > > > If
> > > > > > > it is indeed transactional, then we should update the javadoc.
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Ignite compute has no relation to the cache's transaction.
> > > > > > > >
> > > > > > > > I think that IgniteCache.invokeAll() is appropriate for
> > described
> > > > > case.
> > > > > > > >
> > > > > > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi, igniters!
> > > > > > > > >
> > > > > > > > > Before openning JIRA ticket, I want to ask question about
> > > > > > affinityCall
> > > > > > > or
> > > > > > > > > affinityRun transactions.
> > > > > > > > >
> > > > > > > > > For example I have batch task to modify many values in
> > > someCache
> > > > > > > > according
> > > > > > > > > to someRule. I want to parallel this task to whole cluster
> > and
> > > > > > minimize
> > > > > > > > > network traffic.
> > > > > > > > > So the resonable choice is affinityCall feature.
> > > > > > > > >
> > > > > > > > > But I want all this changes to be in one transactoin. i.e.
> > with
> > > > at
> > > > > > > least
> > > > > > > > > atomicity property (of ACID). And if for some reason my
> task
> > > will
> > > > > be
> > > > > > > > > canceled or failed on one node - it should change nothing
> at
> > > all.
> > > > > > > > >
> > > > > > > > > So, can I achieve this with existing functionality, or how
> > can
> > > I
> > > > > > > approach
> > > > > > > > > to this task?
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best regards,
> > > > > > Alexei Scherbakov
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > С уважением,
> > > > > Чураев Антон
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > С уважением,
> > > Чураев Антон
> > >
> >
>
>
>
> --
> С уважением,
> Чураев Антон
>

[Антон Чураев]

Using JTA in current implementation Ignite is possible. But it is
expensive, because currently Ignite does not support distributed
transaction context within all grid.

I think it would be right to devide task into two:
1) Add support of switching transactional context between multiple thread
within single instance jvm;
2) Using distributed memory for keeping transaction context;

In my opinion, the first one is not so difficult to implement.

2016-12-13 1:29 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Anton,
>
> Looking at this sequence, I don't see any way of achieving it other than
> enrolling all transactions into one JTA transaction. If you seen another
> way, can you please suggest it here?
>
> D.
>
> On Sat, Dec 10, 2016 at 2:07 PM, Антон Чураев <[hidden email]>
> wrote:
>
> > Dmitriy, it's ok
> >
> > To be abstract simple business transaction for execution payment
> > (preparation done before)  from the card looks like:
> > 1) Create a payment document (cache API);
> > 2) Write-off funds from the payer's card;
> > 2.1) Change in register #1 (cache API);
> > 2.2) Change in register #2 (cache API);
> > 2.3) Change in register #... (cache API);
> > 2.4) Change limits of card (cache API);
> > 3) Payment to service provider;
> > 3.1) Change in the balance of business unit having the contract with
> payer
> > (cache API);
> > 3.2) Change in the balance of business unit having the contract with
> > provider (cache API);
> > 3.3) Change in the balance of the account (cache API);
> > 3.4.1) Some change in registers... (cache API);
> > 3.4.2) ...;
> > 3.3) ...
> > 3.4) Invoke provider's API for billing payment of payer;
> > 4) Formation of financial statements (it's possible to implement
> off-line -
> > in other transational)
> > 4.1) ...
> > 4.2) ...
> >
> > And all steps have been designed into separate microservices. And, of
> > course, it have been designed via asynchronous transport.
> > On the other hand it seems to be that implementation of coordination of
> > 10-20 local transactions is not a good idea
> >
> > 2016-12-10 23:30 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > Anton,
> > >
> > > Thanks for the explanation. I am sorry to keep asking questions on
> this.
> > > Can you change your example to include concrete Ignite calls on Compute
> > or
> > > Cache APIs (or other APIs)? I am still struggling to understand the
> > > boundaries between business and Ignite logic.
> > >
> > > D.
> > >
> > > On Sat, Dec 10, 2016 at 5:46 AM, Антон Чураев <[hidden email]>
> > > wrote:
> > >
> > > > For example:
> > > > 1) Front-end sends a request to perform a complex transaction.
> > > > 2) Some application (like a business transactional coordinator)
> > receives
> > > > message via asynchronous transport. This application implements logic
> > of
> > > > calling different services sequentially or in parallel via
> asynchronous
> > > > transport.
> > > > 3) Each service implement some little changes in a cache.
> > > >
> > > > Or:
> > > > 1) Front-end sends a request to perform a complex transaction.
> > > > 2) This transaction is implemented in microservice architecture
> (large
> > > > number microservices + asynchronous transport).
> > > > 3) Each microservice implement some little changes in a cache.
> > > >
> > > > I think it is possible to implement distributed transactional using
> XA
> > > > coordinator outside Ignite and local transaction in each service. But
> > its
> > > > cost may be unacceptable especially in the case of using a large
> number
> > > of
> > > > services.
> > > >
> > > > I think distributed transaction inside Ignite could be useful also
> for
> > > > using multiple ComputeTask in one transaction.
> > > >
> > > > 2016-12-09 21:45 GMT+03:00 Dmitriy Setrakyan <[hidden email]
> >:
> > > >
> > > > > Sounds like you need a centralized JTA server for this type of
> > purpose,
> > > > no?
> > > > > In that case, Ignite transactions can already merge into ongoing
> JTA
> > > > > transactions.
> > > > >
> > > > > I would prefer to see a distributed flow of events to fully
> > understand
> > > > the
> > > > > issue. For example,
> > > > >
> > > > > Client
> > > > >   - start transaction
> > > > >   - send compute
> > > > >
> > > > > Server
> > > > >   - receive compute
> > > > >   - execute ...
> > > > >   - execute ...
> > > > >
> > > > > etc.
> > > > >
> > > > > D.
> > > > >
> > > > > On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <[hidden email]
> >
> > > > wrote:
> > > > >
> > > > > > In some cases it is necessary to implement a transaction
> processing
> > > > logic
> > > > > > in several different application servers. In this case, working
> > with
> > > > > Ignite
> > > > > > cache will be performed within the various applications. But all
> > > these
> > > > > > changes must be made within the same distributed transaction.
> > > > > >
> > > > > > In my opinion this will require context transfer between the
> > threads
> > > > > within
> > > > > > a single node or multiple Ignite nodes.
> > > > > >
> > > > > > 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <
> > > > > [hidden email]
> > > > > > >:
> > > > > >
> > > > > > > Hi.
> > > > > > >
> > > > > > > It's unclear from your description what are you trying to
> > achieve.
> > > > > > >
> > > > > > > AffinityCall is unicast and wil be send to single node.
> > > > > > >
> > > > > > > To parallelise task among the cluster I would recommend to use
> > > > compute
> > > > > > task
> > > > > > > API [1]
> > > > > > >
> > > > > > > But the task execution is not transactional. Nevertheless, each
> > job
> > > > > > > triggered by task can use it's own local transaction.
> > > > > > >
> > > > > > > And please explain, why can't you use a generic Ignite
> > transaction
> > > > for
> > > > > > you
> > > > > > > task?
> > > > > > >
> > > > > > > [1] https://apacheignite.readme.io/docs/compute-tasks
> > > > > > >
> > > > > > > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <
> > [hidden email]
> > > >:
> > > > > > >
> > > > > > > > Taras, is invokeAll() transactional? The javadoc is silent to
> > > this
> > > > > > fact.
> > > > > > > If
> > > > > > > > it is indeed transactional, then we should update the
> javadoc.
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Ignite compute has no relation to the cache's transaction.
> > > > > > > > >
> > > > > > > > > I think that IgniteCache.invokeAll() is appropriate for
> > > described
> > > > > > case.
> > > > > > > > >
> > > > > > > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <[hidden email]>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, igniters!
> > > > > > > > > >
> > > > > > > > > > Before openning JIRA ticket, I want to ask question about
> > > > > > > affinityCall
> > > > > > > > or
> > > > > > > > > > affinityRun transactions.
> > > > > > > > > >
> > > > > > > > > > For example I have batch task to modify many values in
> > > > someCache
> > > > > > > > > according
> > > > > > > > > > to someRule. I want to parallel this task to whole
> cluster
> > > and
> > > > > > > minimize
> > > > > > > > > > network traffic.
> > > > > > > > > > So the resonable choice is affinityCall feature.
> > > > > > > > > >
> > > > > > > > > > But I want all this changes to be in one transactoin.
> i.e.
> > > with
> > > > > at
> > > > > > > > least
> > > > > > > > > > atomicity property (of ACID). And if for some reason my
> > task
> > > > will
> > > > > > be
> > > > > > > > > > canceled or failed on one node - it should change nothing
> > at
> > > > all.
> > > > > > > > > >
> > > > > > > > > > So, can I achieve this with existing functionality, or
> how
> > > can
> > > > I
> > > > > > > > approach
> > > > > > > > > > to this task?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Alexei Scherbakov
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > С уважением,
> > > > > > Чураев Антон
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > С уважением,
> > > > Чураев Антон
> > > >
> > >
> >
> >
> >
> > --
> > С уважением,
> > Чураев Антон
> >
>

--
С уважением,
Чураев Антон

[dsetrakyan]

On Tue, Dec 13, 2016 at 1:29 AM, Антон Чураев <[hidden email]> wrote:

> Using JTA in current implementation Ignite is possible. But it is
> expensive, because currently Ignite does not support distributed
> transaction context within all grid.
>
> I think it would be right to devide task into two:
> 1) Add support of switching transactional context between multiple thread
> within single instance jvm;
>

Can you please suggest an API for it?


> 2) Using distributed memory for keeping transaction context;
>

What do you mean by distributed memory here?

>
> In my opinion, the first one is not so difficult to implement.
>
> 2016-12-13 1:29 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Anton,
> >
> > Looking at this sequence, I don't see any way of achieving it other than
> > enrolling all transactions into one JTA transaction. If you seen another
> > way, can you please suggest it here?
> >
> > D.
> >
> > On Sat, Dec 10, 2016 at 2:07 PM, Антон Чураев <[hidden email]>
> > wrote:
> >
> > > Dmitriy, it's ok
> > >
> > > To be abstract simple business transaction for execution payment
> > > (preparation done before)  from the card looks like:
> > > 1) Create a payment document (cache API);
> > > 2) Write-off funds from the payer's card;
> > > 2.1) Change in register #1 (cache API);
> > > 2.2) Change in register #2 (cache API);
> > > 2.3) Change in register #... (cache API);
> > > 2.4) Change limits of card (cache API);
> > > 3) Payment to service provider;
> > > 3.1) Change in the balance of business unit having the contract with
> > payer
> > > (cache API);
> > > 3.2) Change in the balance of business unit having the contract with
> > > provider (cache API);
> > > 3.3) Change in the balance of the account (cache API);
> > > 3.4.1) Some change in registers... (cache API);
> > > 3.4.2) ...;
> > > 3.3) ...
> > > 3.4) Invoke provider's API for billing payment of payer;
> > > 4) Formation of financial statements (it's possible to implement
> > off-line -
> > > in other transational)
> > > 4.1) ...
> > > 4.2) ...
> > >
> > > And all steps have been designed into separate microservices. And, of
> > > course, it have been designed via asynchronous transport.
> > > On the other hand it seems to be that implementation of coordination of
> > > 10-20 local transactions is not a good idea
> > >
> > > 2016-12-10 23:30 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > >
> > > > Anton,
> > > >
> > > > Thanks for the explanation. I am sorry to keep asking questions on
> > this.
> > > > Can you change your example to include concrete Ignite calls on
> Compute
> > > or
> > > > Cache APIs (or other APIs)? I am still struggling to understand the
> > > > boundaries between business and Ignite logic.
> > > >
> > > > D.
> > > >
> > > > On Sat, Dec 10, 2016 at 5:46 AM, Антон Чураев <[hidden email]>
> > > > wrote:
> > > >
> > > > > For example:
> > > > > 1) Front-end sends a request to perform a complex transaction.
> > > > > 2) Some application (like a business transactional coordinator)
> > > receives
> > > > > message via asynchronous transport. This application implements
> logic
> > > of
> > > > > calling different services sequentially or in parallel via
> > asynchronous
> > > > > transport.
> > > > > 3) Each service implement some little changes in a cache.
> > > > >
> > > > > Or:
> > > > > 1) Front-end sends a request to perform a complex transaction.
> > > > > 2) This transaction is implemented in microservice architecture
> > (large
> > > > > number microservices + asynchronous transport).
> > > > > 3) Each microservice implement some little changes in a cache.
> > > > >
> > > > > I think it is possible to implement distributed transactional using
> > XA
> > > > > coordinator outside Ignite and local transaction in each service.
> But
> > > its
> > > > > cost may be unacceptable especially in the case of using a large
> > number
> > > > of
> > > > > services.
> > > > >
> > > > > I think distributed transaction inside Ignite could be useful also
> > for
> > > > > using multiple ComputeTask in one transaction.
> > > > >
> > > > > 2016-12-09 21:45 GMT+03:00 Dmitriy Setrakyan <
> [hidden email]
> > >:
> > > > >
> > > > > > Sounds like you need a centralized JTA server for this type of
> > > purpose,
> > > > > no?
> > > > > > In that case, Ignite transactions can already merge into ongoing
> > JTA
> > > > > > transactions.
> > > > > >
> > > > > > I would prefer to see a distributed flow of events to fully
> > > understand
> > > > > the
> > > > > > issue. For example,
> > > > > >
> > > > > > Client
> > > > > >   - start transaction
> > > > > >   - send compute
> > > > > >
> > > > > > Server
> > > > > >   - receive compute
> > > > > >   - execute ...
> > > > > >   - execute ...
> > > > > >
> > > > > > etc.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <
> [hidden email]
> > >
> > > > > wrote:
> > > > > >
> > > > > > > In some cases it is necessary to implement a transaction
> > processing
> > > > > logic
> > > > > > > in several different application servers. In this case, working
> > > with
> > > > > > Ignite
> > > > > > > cache will be performed within the various applications. But
> all
> > > > these
> > > > > > > changes must be made within the same distributed transaction.
> > > > > > >
> > > > > > > In my opinion this will require context transfer between the
> > > threads
> > > > > > within
> > > > > > > a single node or multiple Ignite nodes.
> > > > > > >
> > > > > > > 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <
> > > > > > [hidden email]
> > > > > > > >:
> > > > > > >
> > > > > > > > Hi.
> > > > > > > >
> > > > > > > > It's unclear from your description what are you trying to
> > > achieve.
> > > > > > > >
> > > > > > > > AffinityCall is unicast and wil be send to single node.
> > > > > > > >
> > > > > > > > To parallelise task among the cluster I would recommend to
> use
> > > > > compute
> > > > > > > task
> > > > > > > > API [1]
> > > > > > > >
> > > > > > > > But the task execution is not transactional. Nevertheless,
> each
> > > job
> > > > > > > > triggered by task can use it's own local transaction.
> > > > > > > >
> > > > > > > > And please explain, why can't you use a generic Ignite
> > > transaction
> > > > > for
> > > > > > > you
> > > > > > > > task?
> > > > > > > >
> > > > > > > > [1] https://apacheignite.readme.io/docs/compute-tasks
> > > > > > > >
> > > > > > > > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <
> > > [hidden email]
> > > > >:
> > > > > > > >
> > > > > > > > > Taras, is invokeAll() transactional? The javadoc is silent
> to
> > > > this
> > > > > > > fact.
> > > > > > > > If
> > > > > > > > > it is indeed transactional, then we should update the
> > javadoc.
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Ignite compute has no relation to the cache's
> transaction.
> > > > > > > > > >
> > > > > > > > > > I think that IgniteCache.invokeAll() is appropriate for
> > > > described
> > > > > > > case.
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <
> [hidden email]>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, igniters!
> > > > > > > > > > >
> > > > > > > > > > > Before openning JIRA ticket, I want to ask question
> about
> > > > > > > > affinityCall
> > > > > > > > > or
> > > > > > > > > > > affinityRun transactions.
> > > > > > > > > > >
> > > > > > > > > > > For example I have batch task to modify many values in
> > > > > someCache
> > > > > > > > > > according
> > > > > > > > > > > to someRule. I want to parallel this task to whole
> > cluster
> > > > and
> > > > > > > > minimize
> > > > > > > > > > > network traffic.
> > > > > > > > > > > So the resonable choice is affinityCall feature.
> > > > > > > > > > >
> > > > > > > > > > > But I want all this changes to be in one transactoin.
> > i.e.
> > > > with
> > > > > > at
> > > > > > > > > least
> > > > > > > > > > > atomicity property (of ACID). And if for some reason my
> > > task
> > > > > will
> > > > > > > be
> > > > > > > > > > > canceled or failed on one node - it should change
> nothing
> > > at
> > > > > all.
> > > > > > > > > > >
> > > > > > > > > > > So, can I achieve this with existing functionality, or
> > how
> > > > can
> > > > > I
> > > > > > > > > approach
> > > > > > > > > > > to this task?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Alexei Scherbakov
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > С уважением,
> > > > > > > Чураев Антон
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > С уважением,
> > > > > Чураев Антон
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > С уважением,
> > > Чураев Антон
> > >
> >
>
>
>
> --
> С уважением,
> Чураев Антон
>

[Игорь Г]

Dmitriy, for p.1 we can enroll it in one JTA transaction, yes.
But is there a cheaper way to do it? I think in p.2 Anton means that if we
can start ignite transaction on one client node and "join" it on other
client node (by some id) so that all changes made on 2nd client were inside
this transaction it will work much faster.




2016-12-13 12:31 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> On Tue, Dec 13, 2016 at 1:29 AM, Антон Чураев <[hidden email]>
> wrote:
>
> > Using JTA in current implementation Ignite is possible. But it is
> > expensive, because currently Ignite does not support distributed
> > transaction context within all grid.
> >
> > I think it would be right to devide task into two:
> > 1) Add support of switching transactional context between multiple thread
> > within single instance jvm;
> >
>
> Can you please suggest an API for it?
>
>
> > 2) Using distributed memory for keeping transaction context;
> >
>
> What do you mean by distributed memory here?
>
>
> >
> > In my opinion, the first one is not so difficult to implement.
> >
> > 2016-12-13 1:29 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > Anton,
> > >
> > > Looking at this sequence, I don't see any way of achieving it other
> than
> > > enrolling all transactions into one JTA transaction. If you seen
> another
> > > way, can you please suggest it here?
> > >
> > > D.
> > >
> > > On Sat, Dec 10, 2016 at 2:07 PM, Антон Чураев <[hidden email]>
> > > wrote:
> > >
> > > > Dmitriy, it's ok
> > > >
> > > > To be abstract simple business transaction for execution payment
> > > > (preparation done before)  from the card looks like:
> > > > 1) Create a payment document (cache API);
> > > > 2) Write-off funds from the payer's card;
> > > > 2.1) Change in register #1 (cache API);
> > > > 2.2) Change in register #2 (cache API);
> > > > 2.3) Change in register #... (cache API);
> > > > 2.4) Change limits of card (cache API);
> > > > 3) Payment to service provider;
> > > > 3.1) Change in the balance of business unit having the contract with
> > > payer
> > > > (cache API);
> > > > 3.2) Change in the balance of business unit having the contract with
> > > > provider (cache API);
> > > > 3.3) Change in the balance of the account (cache API);
> > > > 3.4.1) Some change in registers... (cache API);
> > > > 3.4.2) ...;
> > > > 3.3) ...
> > > > 3.4) Invoke provider's API for billing payment of payer;
> > > > 4) Formation of financial statements (it's possible to implement
> > > off-line -
> > > > in other transational)
> > > > 4.1) ...
> > > > 4.2) ...
> > > >
> > > > And all steps have been designed into separate microservices. And, of
> > > > course, it have been designed via asynchronous transport.
> > > > On the other hand it seems to be that implementation of coordination
> of
> > > > 10-20 local transactions is not a good idea
> > > >
> > > > 2016-12-10 23:30 GMT+03:00 Dmitriy Setrakyan <[hidden email]
> >:
> > > >
> > > > > Anton,
> > > > >
> > > > > Thanks for the explanation. I am sorry to keep asking questions on
> > > this.
> > > > > Can you change your example to include concrete Ignite calls on
> > Compute
> > > > or
> > > > > Cache APIs (or other APIs)? I am still struggling to understand the
> > > > > boundaries between business and Ignite logic.
> > > > >
> > > > > D.
> > > > >
> > > > > On Sat, Dec 10, 2016 at 5:46 AM, Антон Чураев <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > For example:
> > > > > > 1) Front-end sends a request to perform a complex transaction.
> > > > > > 2) Some application (like a business transactional coordinator)
> > > > receives
> > > > > > message via asynchronous transport. This application implements
> > logic
> > > > of
> > > > > > calling different services sequentially or in parallel via
> > > asynchronous
> > > > > > transport.
> > > > > > 3) Each service implement some little changes in a cache.
> > > > > >
> > > > > > Or:
> > > > > > 1) Front-end sends a request to perform a complex transaction.
> > > > > > 2) This transaction is implemented in microservice architecture
> > > (large
> > > > > > number microservices + asynchronous transport).
> > > > > > 3) Each microservice implement some little changes in a cache.
> > > > > >
> > > > > > I think it is possible to implement distributed transactional
> using
> > > XA
> > > > > > coordinator outside Ignite and local transaction in each service.
> > But
> > > > its
> > > > > > cost may be unacceptable especially in the case of using a large
> > > number
> > > > > of
> > > > > > services.
> > > > > >
> > > > > > I think distributed transaction inside Ignite could be useful
> also
> > > for
> > > > > > using multiple ComputeTask in one transaction.
> > > > > >
> > > > > > 2016-12-09 21:45 GMT+03:00 Dmitriy Setrakyan <
> > [hidden email]
> > > >:
> > > > > >
> > > > > > > Sounds like you need a centralized JTA server for this type of
> > > > purpose,
> > > > > > no?
> > > > > > > In that case, Ignite transactions can already merge into
> ongoing
> > > JTA
> > > > > > > transactions.
> > > > > > >
> > > > > > > I would prefer to see a distributed flow of events to fully
> > > > understand
> > > > > > the
> > > > > > > issue. For example,
> > > > > > >
> > > > > > > Client
> > > > > > >   - start transaction
> > > > > > >   - send compute
> > > > > > >
> > > > > > > Server
> > > > > > >   - receive compute
> > > > > > >   - execute ...
> > > > > > >   - execute ...
> > > > > > >
> > > > > > > etc.
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > In some cases it is necessary to implement a transaction
> > > processing
> > > > > > logic
> > > > > > > > in several different application servers. In this case,
> working
> > > > with
> > > > > > > Ignite
> > > > > > > > cache will be performed within the various applications. But
> > all
> > > > > these
> > > > > > > > changes must be made within the same distributed transaction.
> > > > > > > >
> > > > > > > > In my opinion this will require context transfer between the
> > > > threads
> > > > > > > within
> > > > > > > > a single node or multiple Ignite nodes.
> > > > > > > >
> > > > > > > > 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <
> > > > > > > [hidden email]
> > > > > > > > >:
> > > > > > > >
> > > > > > > > > Hi.
> > > > > > > > >
> > > > > > > > > It's unclear from your description what are you trying to
> > > > achieve.
> > > > > > > > >
> > > > > > > > > AffinityCall is unicast and wil be send to single node.
> > > > > > > > >
> > > > > > > > > To parallelise task among the cluster I would recommend to
> > use
> > > > > > compute
> > > > > > > > task
> > > > > > > > > API [1]
> > > > > > > > >
> > > > > > > > > But the task execution is not transactional. Nevertheless,
> > each
> > > > job
> > > > > > > > > triggered by task can use it's own local transaction.
> > > > > > > > >
> > > > > > > > > And please explain, why can't you use a generic Ignite
> > > > transaction
> > > > > > for
> > > > > > > > you
> > > > > > > > > task?
> > > > > > > > >
> > > > > > > > > [1] https://apacheignite.readme.io/docs/compute-tasks
> > > > > > > > >
> > > > > > > > > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >:
> > > > > > > > >
> > > > > > > > > > Taras, is invokeAll() transactional? The javadoc is
> silent
> > to
> > > > > this
> > > > > > > > fact.
> > > > > > > > > If
> > > > > > > > > > it is indeed transactional, then we should update the
> > > javadoc.
> > > > > > > > > >
> > > > > > > > > > D.
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Ignite compute has no relation to the cache's
> > transaction.
> > > > > > > > > > >
> > > > > > > > > > > I think that IgniteCache.invokeAll() is appropriate for
> > > > > described
> > > > > > > > case.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi, igniters!
> > > > > > > > > > > >
> > > > > > > > > > > > Before openning JIRA ticket, I want to ask question
> > about
> > > > > > > > > affinityCall
> > > > > > > > > > or
> > > > > > > > > > > > affinityRun transactions.
> > > > > > > > > > > >
> > > > > > > > > > > > For example I have batch task to modify many values
> in
> > > > > > someCache
> > > > > > > > > > > according
> > > > > > > > > > > > to someRule. I want to parallel this task to whole
> > > cluster
> > > > > and
> > > > > > > > > minimize
> > > > > > > > > > > > network traffic.
> > > > > > > > > > > > So the resonable choice is affinityCall feature.
> > > > > > > > > > > >
> > > > > > > > > > > > But I want all this changes to be in one transactoin.
> > > i.e.
> > > > > with
> > > > > > > at
> > > > > > > > > > least
> > > > > > > > > > > > atomicity property (of ACID). And if for some reason
> my
> > > > task
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > > canceled or failed on one node - it should change
> > nothing
> > > > at
> > > > > > all.
> > > > > > > > > > > >
> > > > > > > > > > > > So, can I achieve this with existing functionality,
> or
> > > how
> > > > > can
> > > > > > I
> > > > > > > > > > approach
> > > > > > > > > > > > to this task?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Alexei Scherbakov
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > С уважением,
> > > > > > > > Чураев Антон
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > С уважением,
> > > > > > Чураев Антон
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > С уважением,
> > > > Чураев Антон
> > > >
> > >
> >
> >
> >
> > --
> > С уважением,
> > Чураев Антон
> >
>

[dsetrakyan]

On Tue, Dec 13, 2016 at 11:22 AM, Игорь Г <[hidden email]> wrote:

> Dmitriy, for p.1 we can enroll it in one JTA transaction, yes.
> But is there a cheaper way to do it?


We have to pick whether we use JTA for this use case at all. If yes, then
it should be used across the whole system and no additional development is
required. If we do not use JTA, then we need to come up with proper API to
support it.


> I think in p.2 Anton means that if we
> can start ignite transaction on one client node and "join" it on other
> client node (by some id) so that all changes made on 2nd client were inside
> this transaction it will work much faster.
>

This is only valid if we decide not to use JTA. Can you please suggest an
API here?

>
>
>
>
> 2016-12-13 12:31 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > On Tue, Dec 13, 2016 at 1:29 AM, Антон Чураев <[hidden email]>
> > wrote:
> >
> > > Using JTA in current implementation Ignite is possible. But it is
> > > expensive, because currently Ignite does not support distributed
> > > transaction context within all grid.
> > >
> > > I think it would be right to devide task into two:
> > > 1) Add support of switching transactional context between multiple
> thread
> > > within single instance jvm;
> > >
> >
> > Can you please suggest an API for it?
> >
> >
> > > 2) Using distributed memory for keeping transaction context;
> > >
> >
> > What do you mean by distributed memory here?
> >
> >
> > >
> > > In my opinion, the first one is not so difficult to implement.
> > >
> > > 2016-12-13 1:29 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > >
> > > > Anton,
> > > >
> > > > Looking at this sequence, I don't see any way of achieving it other
> > than
> > > > enrolling all transactions into one JTA transaction. If you seen
> > another
> > > > way, can you please suggest it here?
> > > >
> > > > D.
> > > >
> > > > On Sat, Dec 10, 2016 at 2:07 PM, Антон Чураев <[hidden email]>
> > > > wrote:
> > > >
> > > > > Dmitriy, it's ok
> > > > >
> > > > > To be abstract simple business transaction for execution payment
> > > > > (preparation done before)  from the card looks like:
> > > > > 1) Create a payment document (cache API);
> > > > > 2) Write-off funds from the payer's card;
> > > > > 2.1) Change in register #1 (cache API);
> > > > > 2.2) Change in register #2 (cache API);
> > > > > 2.3) Change in register #... (cache API);
> > > > > 2.4) Change limits of card (cache API);
> > > > > 3) Payment to service provider;
> > > > > 3.1) Change in the balance of business unit having the contract
> with
> > > > payer
> > > > > (cache API);
> > > > > 3.2) Change in the balance of business unit having the contract
> with
> > > > > provider (cache API);
> > > > > 3.3) Change in the balance of the account (cache API);
> > > > > 3.4.1) Some change in registers... (cache API);
> > > > > 3.4.2) ...;
> > > > > 3.3) ...
> > > > > 3.4) Invoke provider's API for billing payment of payer;
> > > > > 4) Formation of financial statements (it's possible to implement
> > > > off-line -
> > > > > in other transational)
> > > > > 4.1) ...
> > > > > 4.2) ...
> > > > >
> > > > > And all steps have been designed into separate microservices. And,
> of
> > > > > course, it have been designed via asynchronous transport.
> > > > > On the other hand it seems to be that implementation of
> coordination
> > of
> > > > > 10-20 local transactions is not a good idea
> > > > >
> > > > > 2016-12-10 23:30 GMT+03:00 Dmitriy Setrakyan <
> [hidden email]
> > >:
> > > > >
> > > > > > Anton,
> > > > > >
> > > > > > Thanks for the explanation. I am sorry to keep asking questions
> on
> > > > this.
> > > > > > Can you change your example to include concrete Ignite calls on
> > > Compute
> > > > > or
> > > > > > Cache APIs (or other APIs)? I am still struggling to understand
> the
> > > > > > boundaries between business and Ignite logic.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Sat, Dec 10, 2016 at 5:46 AM, Антон Чураев <
> > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > For example:
> > > > > > > 1) Front-end sends a request to perform a complex transaction.
> > > > > > > 2) Some application (like a business transactional coordinator)
> > > > > receives
> > > > > > > message via asynchronous transport. This application implements
> > > logic
> > > > > of
> > > > > > > calling different services sequentially or in parallel via
> > > > asynchronous
> > > > > > > transport.
> > > > > > > 3) Each service implement some little changes in a cache.
> > > > > > >
> > > > > > > Or:
> > > > > > > 1) Front-end sends a request to perform a complex transaction.
> > > > > > > 2) This transaction is implemented in microservice architecture
> > > > (large
> > > > > > > number microservices + asynchronous transport).
> > > > > > > 3) Each microservice implement some little changes in a cache.
> > > > > > >
> > > > > > > I think it is possible to implement distributed transactional
> > using
> > > > XA
> > > > > > > coordinator outside Ignite and local transaction in each
> service.
> > > But
> > > > > its
> > > > > > > cost may be unacceptable especially in the case of using a
> large
> > > > number
> > > > > > of
> > > > > > > services.
> > > > > > >
> > > > > > > I think distributed transaction inside Ignite could be useful
> > also
> > > > for
> > > > > > > using multiple ComputeTask in one transaction.
> > > > > > >
> > > > > > > 2016-12-09 21:45 GMT+03:00 Dmitriy Setrakyan <
> > > [hidden email]
> > > > >:
> > > > > > >
> > > > > > > > Sounds like you need a centralized JTA server for this type
> of
> > > > > purpose,
> > > > > > > no?
> > > > > > > > In that case, Ignite transactions can already merge into
> > ongoing
> > > > JTA
> > > > > > > > transactions.
> > > > > > > >
> > > > > > > > I would prefer to see a distributed flow of events to fully
> > > > > understand
> > > > > > > the
> > > > > > > > issue. For example,
> > > > > > > >
> > > > > > > > Client
> > > > > > > >   - start transaction
> > > > > > > >   - send compute
> > > > > > > >
> > > > > > > > Server
> > > > > > > >   - receive compute
> > > > > > > >   - execute ...
> > > > > > > >   - execute ...
> > > > > > > >
> > > > > > > > etc.
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Fri, Dec 9, 2016 at 1:26 AM, Антон Чураев <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > In some cases it is necessary to implement a transaction
> > > > processing
> > > > > > > logic
> > > > > > > > > in several different application servers. In this case,
> > working
> > > > > with
> > > > > > > > Ignite
> > > > > > > > > cache will be performed within the various applications.
> But
> > > all
> > > > > > these
> > > > > > > > > changes must be made within the same distributed
> transaction.
> > > > > > > > >
> > > > > > > > > In my opinion this will require context transfer between
> the
> > > > > threads
> > > > > > > > within
> > > > > > > > > a single node or multiple Ignite nodes.
> > > > > > > > >
> > > > > > > > > 2016-12-08 12:53 GMT+03:00 Alexei Scherbakov <
> > > > > > > > [hidden email]
> > > > > > > > > >:
> > > > > > > > >
> > > > > > > > > > Hi.
> > > > > > > > > >
> > > > > > > > > > It's unclear from your description what are you trying to
> > > > > achieve.
> > > > > > > > > >
> > > > > > > > > > AffinityCall is unicast and wil be send to single node.
> > > > > > > > > >
> > > > > > > > > > To parallelise task among the cluster I would recommend
> to
> > > use
> > > > > > > compute
> > > > > > > > > task
> > > > > > > > > > API [1]
> > > > > > > > > >
> > > > > > > > > > But the task execution is not transactional.
> Nevertheless,
> > > each
> > > > > job
> > > > > > > > > > triggered by task can use it's own local transaction.
> > > > > > > > > >
> > > > > > > > > > And please explain, why can't you use a generic Ignite
> > > > > transaction
> > > > > > > for
> > > > > > > > > you
> > > > > > > > > > task?
> > > > > > > > > >
> > > > > > > > > > [1] https://apacheignite.readme.io/docs/compute-tasks
> > > > > > > > > >
> > > > > > > > > > 2016-12-08 4:09 GMT+03:00 Dmitriy Setrakyan <
> > > > > [hidden email]
> > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Taras, is invokeAll() transactional? The javadoc is
> > silent
> > > to
> > > > > > this
> > > > > > > > > fact.
> > > > > > > > > > If
> > > > > > > > > > > it is indeed transactional, then we should update the
> > > > javadoc.
> > > > > > > > > > >
> > > > > > > > > > > D.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 7, 2016 at 5:32 AM, Taras Ledkov <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Ignite compute has no relation to the cache's
> > > transaction.
> > > > > > > > > > > >
> > > > > > > > > > > > I think that IgniteCache.invokeAll() is appropriate
> for
> > > > > > described
> > > > > > > > > case.
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Dec 7, 2016 at 4:00 PM, Игорь Г <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi, igniters!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Before openning JIRA ticket, I want to ask question
> > > about
> > > > > > > > > > affinityCall
> > > > > > > > > > > or
> > > > > > > > > > > > > affinityRun transactions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For example I have batch task to modify many values
> > in
> > > > > > > someCache
> > > > > > > > > > > > according
> > > > > > > > > > > > > to someRule. I want to parallel this task to whole
> > > > cluster
> > > > > > and
> > > > > > > > > > minimize
> > > > > > > > > > > > > network traffic.
> > > > > > > > > > > > > So the resonable choice is affinityCall feature.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But I want all this changes to be in one
> transactoin.
> > > > i.e.
> > > > > > with
> > > > > > > > at
> > > > > > > > > > > least
> > > > > > > > > > > > > atomicity property (of ACID). And if for some
> reason
> > my
> > > > > task
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > canceled or failed on one node - it should change
> > > nothing
> > > > > at
> > > > > > > all.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So, can I achieve this with existing functionality,
> > or
> > > > how
> > > > > > can
> > > > > > > I
> > > > > > > > > > > approach
> > > > > > > > > > > > > to this task?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Alexei Scherbakov
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > С уважением,
> > > > > > > > > Чураев Антон
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > С уважением,
> > > > > > > Чураев Антон
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > С уважением,
> > > > > Чураев Антон
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > С уважением,
> > > Чураев Антон
> > >
> >
>