distributed transaction of non-single coordinator
Locked
123
123
 
|
orchestrator is a custom thing. He is responsible for managing business
scenarios flows. Many nodes are involved in scenarios. They exchange data and folow one another. If you acquinted with BPMN framework, so orchestrator is like bpmn engine. вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <[hidden email]>: > What is Orchestrator for you? Is it a thing from Microsoft or your custom > in-house software? > > Sergi > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > Fine. Let's say we've got multiple servers which fulfills custom logic. > > This servers compound oriented graph (BPMN process) which controlled by > > Orchestrator. > > For instance, *server1 *creates *variable A *with value 1, persists it > to > > IGNITE cache and creates *variable B *and sends it to* server2. *The > > latests receives *variable B*, do some logic with it and stores to > IGNITE. > > All the work made by both servers must be fulfilled in *one* transaction. > > Because we need all information done, or nothing(rollbacked). The > scenario > > is managed by orchestrator. > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin <[hidden email]>: > > > > > Ok, it is not a business case, it is your wrong solution for it. > > > Lets try again, what is the business case? > > > > > > Sergi > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > > > The case is the following, One starts transaction in one node, and > > commit > > > > this transaction in another jvm node(or rollback it remotely). > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > [hidden email] > > >: > > > > > > > > > Because even if you make it work for some simplistic scenario, get > > > ready > > > > to > > > > > write many fault tolerance tests and make sure that you TXs work > > > > gracefully > > > > > in all modes in case of crashes. Also make sure that we do not have > > any > > > > > performance drops after all your changes in existing benchmarks. > All > > in > > > > all > > > > > I don't believe these conditions will be met and your contribution > > will > > > > be > > > > > accepted. > > > > > > > > > > Better solution to what problem? Sending TX to another node? The > > > problem > > > > > statement itself is already wrong. What business case you are > trying > > to > > > > > solve? I'm sure everything you need can be done in a much more > simple > > > and > > > > > efficient way at the application level. > > > > > > > > > > Sergi > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > [hidden email] > > > >: > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Just serializing TX object and deserializing it on another node > > is > > > > > > > meaningless, because other nodes participating in the TX have > to > > > know > > > > > > about > > > > > > > the new coordinator. This will require protocol changes, we > > > > definitely > > > > > > will > > > > > > > have fault tolerance and performance issues. IMO the whole idea > > is > > > > > wrong > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > IgniteTransactionState implememntation contains > IgniteTxEntry's > > > > which > > > > > > is > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are passing > > transaction > > > > > > objects > > > > > > > > > around. Such object may contain all sorts of Ignite > context. > > If > > > > > some > > > > > > > data > > > > > > > > > needs to be passed across, we should create a special > > transfer > > > > > object > > > > > > > in > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing transaction > > > > proceeding. > > > > > > > > > > At first, After transaction serialization and > > deserialization > > > > on > > > > > > the > > > > > > > > > remote > > > > > > > > > > server, there is no txState. So im going to put it in > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction lacks of shared > > > cache > > > > > > > context > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it must be > injected > > > by > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY KUZNETSOV < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > while starting and continuing transaction in different > > jvms > > > > in > > > > > > run > > > > > > > > into > > > > > > > > > > > serialization exception in writeExternalMeta : > > > > > > > > > > > > > > > > > > > > > > @Override public void writeExternal(ObjectOutput out) > > > throws > > > > > > > > > IOException > > > > > > > > > > { > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey Goncharuk < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you want, but I have > a > > > few > > > > > > > > concerns: > > > > > > > > > > > - What is the API for the proposed change? In your > test, > > > you > > > > > > pass > > > > > > > an > > > > > > > > > > > instance of transaction created on ignite(0) to the > > ignite > > > > > > instance > > > > > > > > > > > ignite(1). This is obviously not possible in a truly > > > > > distributed > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > - How will you synchronize cache update actions and > > > > transaction > > > > > > > > commit? > > > > > > > > > > > Say, you have one node that decided to commit, but > > another > > > > node > > > > > > is > > > > > > > > > still > > > > > > > > > > > writing within this transaction. How do you make sure > > that > > > > two > > > > > > > nodes > > > > > > > > > will > > > > > > > > > > > not call commit() and rollback() simultaneously? > > > > > > > > > > > - How do you make sure that either commit() or > > rollback() > > > is > > > > > > > called > > > > > > > > if > > > > > > > > > > an > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial understanding was > > that > > > > > > > > transferring > > > > > > > > > > of > > > > > > > > > > > tx > > > > > > > > > > > > ownership from one node to another will be happened > > > > > > automatically > > > > > > > > > when > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on multiple threads, > > > nodes, > > > > > > > > > jvms(soon). > > > > > > > > > > > So > > > > > > > > > > > > > every node is able to rollback, or commit common > > > > > > transaction.It > > > > > > > > > > turned > > > > > > > > > > > > up i > > > > > > > > > > > > > need to transfer tx between nodes in order to > commit > > > > > > > transaction > > > > > > > > in > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey Goncharuk < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a concept of > transferring > > > of > > > > tx > > > > > > > > > ownership > > > > > > > > > > > > from > > > > > > > > > > > > > > one node to another? My initial understanding was > > > that > > > > > you > > > > > > > want > > > > > > > > > to > > > > > > > > > > be > > > > > > > > > > > > > able > > > > > > > > > > > > > > to update keys in a transaction from multiple > > threads > > > > in > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction started in one node, > > and > > > > > > > continued > > > > > > > > > in > > > > > > > > > > > > > another > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > The following test describes my idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions = > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> cache = > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = transactions.txStart( > > concurrency, > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> fut = > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > -> > > > > > > > > > > { > > > > > > > > > > > > > > > IgniteTransactions ts = > > > ignite(1).transactions(); > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > Assert.assertEquals( > > TransactionState.STOPPED, > > > > > > > > tx.state()); > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > tx.state()); > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.COMMITTED, > > > > > > > tx.state()); > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > Assert.assertFalse(cache.containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we just rebind *tx* > > to > > > > > > current > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction tx) { > > > > > > > > > > > > > > > TransactionProxyImpl transactionProxy = > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > cctx.tm().reopenTx(transactionProxy.tx()); > > > > > > > > > > > > > > > transactionProxy.bindToCurrentThread(); > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter *threadMap* so > that > > > it > > > > > > binds > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis Magda < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational behind this and the > > > > > thoughts, > > > > > > > > > design > > > > > > > > > > > > ideas > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, ALEKSEY > > KUZNETSOV < > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing distributed > transaction > > > > which > > > > > > can > > > > > > > be > > > > > > > > > > > started > > > > > > > > > > > > > at > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > node, and continued at other one. Has > anybody > > > > > > thoughts > > > > > > > on > > > > > > > > > it > > > > > > > > > > ? > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > *Best Regards,* *Kuznetsov Aleksey* |
|
|
Good! Basically your orchestrator just takes some predefined graph of
distributed services to be invoked, calls them by some kind of RPC and passes the needed parameters between them, right? Sergi 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > orchestrator is a custom thing. He is responsible for managing business > scenarios flows. Many nodes are involved in scenarios. They exchange data > and folow one another. If you acquinted with BPMN framework, so > orchestrator is like bpmn engine. > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <[hidden email]>: > > > What is Orchestrator for you? Is it a thing from Microsoft or your custom > > in-house software? > > > > Sergi > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > > > Fine. Let's say we've got multiple servers which fulfills custom logic. > > > This servers compound oriented graph (BPMN process) which controlled by > > > Orchestrator. > > > For instance, *server1 *creates *variable A *with value 1, persists it > > to > > > IGNITE cache and creates *variable B *and sends it to* server2. *The > > > latests receives *variable B*, do some logic with it and stores to > > IGNITE. > > > All the work made by both servers must be fulfilled in *one* > transaction. > > > Because we need all information done, or nothing(rollbacked). The > > scenario > > > is managed by orchestrator. > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin <[hidden email] > >: > > > > > > > Ok, it is not a business case, it is your wrong solution for it. > > > > Lets try again, what is the business case? > > > > > > > > Sergi > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > > >: > > > > > > > > > The case is the following, One starts transaction in one node, and > > > commit > > > > > this transaction in another jvm node(or rollback it remotely). > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > [hidden email] > > > >: > > > > > > > > > > > Because even if you make it work for some simplistic scenario, > get > > > > ready > > > > > to > > > > > > write many fault tolerance tests and make sure that you TXs work > > > > > gracefully > > > > > > in all modes in case of crashes. Also make sure that we do not > have > > > any > > > > > > performance drops after all your changes in existing benchmarks. > > All > > > in > > > > > all > > > > > > I don't believe these conditions will be met and your > contribution > > > will > > > > > be > > > > > > accepted. > > > > > > > > > > > > Better solution to what problem? Sending TX to another node? The > > > > problem > > > > > > statement itself is already wrong. What business case you are > > trying > > > to > > > > > > solve? I'm sure everything you need can be done in a much more > > simple > > > > and > > > > > > efficient way at the application level. > > > > > > > > > > > > Sergi > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > Just serializing TX object and deserializing it on another > node > > > is > > > > > > > > meaningless, because other nodes participating in the TX have > > to > > > > know > > > > > > > about > > > > > > > > the new coordinator. This will require protocol changes, we > > > > > definitely > > > > > > > will > > > > > > > > have fault tolerance and performance issues. IMO the whole > idea > > > is > > > > > > wrong > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation contains > > IgniteTxEntry's > > > > > which > > > > > > > is > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are passing > > > transaction > > > > > > > objects > > > > > > > > > > around. Such object may contain all sorts of Ignite > > context. > > > If > > > > > > some > > > > > > > > data > > > > > > > > > > needs to be passed across, we should create a special > > > transfer > > > > > > object > > > > > > > > in > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing transaction > > > > > proceeding. > > > > > > > > > > > At first, After transaction serialization and > > > deserialization > > > > > on > > > > > > > the > > > > > > > > > > remote > > > > > > > > > > > server, there is no txState. So im going to put it in > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction lacks of > shared > > > > cache > > > > > > > > context > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it must be > > injected > > > > by > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing transaction in > different > > > jvms > > > > > in > > > > > > > run > > > > > > > > > into > > > > > > > > > > > > serialization exception in writeExternalMeta : > > > > > > > > > > > > > > > > > > > > > > > > @Override public void writeExternal(ObjectOutput out) > > > > throws > > > > > > > > > > IOException > > > > > > > > > > > { > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey Goncharuk < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you want, but I > have > > a > > > > few > > > > > > > > > concerns: > > > > > > > > > > > > - What is the API for the proposed change? In your > > test, > > > > you > > > > > > > pass > > > > > > > > an > > > > > > > > > > > > instance of transaction created on ignite(0) to the > > > ignite > > > > > > > instance > > > > > > > > > > > > ignite(1). This is obviously not possible in a truly > > > > > > distributed > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > - How will you synchronize cache update actions and > > > > > transaction > > > > > > > > > commit? > > > > > > > > > > > > Say, you have one node that decided to commit, but > > > another > > > > > node > > > > > > > is > > > > > > > > > > still > > > > > > > > > > > > writing within this transaction. How do you make sure > > > that > > > > > two > > > > > > > > nodes > > > > > > > > > > will > > > > > > > > > > > > not call commit() and rollback() simultaneously? > > > > > > > > > > > > - How do you make sure that either commit() or > > > rollback() > > > > is > > > > > > > > called > > > > > > > > > if > > > > > > > > > > > an > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial understanding was > > > that > > > > > > > > > transferring > > > > > > > > > > > of > > > > > > > > > > > > tx > > > > > > > > > > > > > ownership from one node to another will be happened > > > > > > > automatically > > > > > > > > > > when > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on multiple > threads, > > > > nodes, > > > > > > > > > > jvms(soon). > > > > > > > > > > > > So > > > > > > > > > > > > > > every node is able to rollback, or commit common > > > > > > > transaction.It > > > > > > > > > > > turned > > > > > > > > > > > > > up i > > > > > > > > > > > > > > need to transfer tx between nodes in order to > > commit > > > > > > > > transaction > > > > > > > > > in > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey Goncharuk < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a concept of > > transferring > > > > of > > > > > tx > > > > > > > > > > ownership > > > > > > > > > > > > > from > > > > > > > > > > > > > > > one node to another? My initial understanding > was > > > > that > > > > > > you > > > > > > > > want > > > > > > > > > > to > > > > > > > > > > > be > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > to update keys in a transaction from multiple > > > threads > > > > > in > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction started in one > node, > > > and > > > > > > > > continued > > > > > > > > > > in > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > The following test describes my idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions = > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> cache = > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = transactions.txStart( > > > concurrency, > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> fut = > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > -> > > > > > > > > > > > { > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > Assert.assertEquals( > > > TransactionState.STOPPED, > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > TransactionState.COMMITTED, > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we just rebind > *tx* > > > to > > > > > > > current > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction tx) { > > > > > > > > > > > > > > > > TransactionProxyImpl transactionProxy = > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > transactionProxy.tx()); > > > > > > > > > > > > > > > > transactionProxy.bindToCurrentThread(); > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter *threadMap* so > > that > > > > it > > > > > > > binds > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis Magda < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational behind this and > the > > > > > > thoughts, > > > > > > > > > > design > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, ALEKSEY > > > KUZNETSOV < > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing distributed > > transaction > > > > > which > > > > > > > can > > > > > > > > be > > > > > > > > > > > > started > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > node, and continued at other one. Has > > anybody > > > > > > > thoughts > > > > > > > > on > > > > > > > > > > it > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Best Regards,* > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > -- > > > > > > *Best Regards,* > > > > > > *Kuznetsov Aleksey* > > > > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > |
|
|
Right
ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin <[hidden email]>: > Good! Basically your orchestrator just takes some predefined graph of > distributed services to be invoked, calls them by some kind of RPC and > passes the needed parameters between them, right? > > Sergi > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > orchestrator is a custom thing. He is responsible for managing business > > scenarios flows. Many nodes are involved in scenarios. They exchange data > > and folow one another. If you acquinted with BPMN framework, so > > orchestrator is like bpmn engine. > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <[hidden email]>: > > > > > What is Orchestrator for you? Is it a thing from Microsoft or your > custom > > > in-house software? > > > > > > Sergi > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > > > Fine. Let's say we've got multiple servers which fulfills custom > logic. > > > > This servers compound oriented graph (BPMN process) which controlled > by > > > > Orchestrator. > > > > For instance, *server1 *creates *variable A *with value 1, persists > it > > > to > > > > IGNITE cache and creates *variable B *and sends it to* server2. *The > > > > latests receives *variable B*, do some logic with it and stores to > > > IGNITE. > > > > All the work made by both servers must be fulfilled in *one* > > transaction. > > > > Because we need all information done, or nothing(rollbacked). The > > > scenario > > > > is managed by orchestrator. > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > [hidden email] > > >: > > > > > > > > > Ok, it is not a business case, it is your wrong solution for it. > > > > > Lets try again, what is the business case? > > > > > > > > > > Sergi > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > [hidden email] > > > >: > > > > > > > > > > > The case is the following, One starts transaction in one node, > and > > > > commit > > > > > > this transaction in another jvm node(or rollback it remotely). > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Because even if you make it work for some simplistic scenario, > > get > > > > > ready > > > > > > to > > > > > > > write many fault tolerance tests and make sure that you TXs > work > > > > > > gracefully > > > > > > > in all modes in case of crashes. Also make sure that we do not > > have > > > > any > > > > > > > performance drops after all your changes in existing > benchmarks. > > > All > > > > in > > > > > > all > > > > > > > I don't believe these conditions will be met and your > > contribution > > > > will > > > > > > be > > > > > > > accepted. > > > > > > > > > > > > > > Better solution to what problem? Sending TX to another node? > The > > > > > problem > > > > > > > statement itself is already wrong. What business case you are > > > trying > > > > to > > > > > > > solve? I'm sure everything you need can be done in a much more > > > simple > > > > > and > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing it on another > > node > > > > is > > > > > > > > > meaningless, because other nodes participating in the TX > have > > > to > > > > > know > > > > > > > > about > > > > > > > > > the new coordinator. This will require protocol changes, we > > > > > > definitely > > > > > > > > will > > > > > > > > > have fault tolerance and performance issues. IMO the whole > > idea > > > > is > > > > > > > wrong > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation contains > > > IgniteTxEntry's > > > > > > which > > > > > > > > is > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are passing > > > > transaction > > > > > > > > objects > > > > > > > > > > > around. Such object may contain all sorts of Ignite > > > context. > > > > If > > > > > > > some > > > > > > > > > data > > > > > > > > > > > needs to be passed across, we should create a special > > > > transfer > > > > > > > object > > > > > > > > > in > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing transaction > > > > > > proceeding. > > > > > > > > > > > > At first, After transaction serialization and > > > > deserialization > > > > > > on > > > > > > > > the > > > > > > > > > > > remote > > > > > > > > > > > > server, there is no txState. So im going to put it in > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction lacks of > > shared > > > > > cache > > > > > > > > > context > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it must be > > > injected > > > > > by > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing transaction in > > different > > > > jvms > > > > > > in > > > > > > > > run > > > > > > > > > > into > > > > > > > > > > > > > serialization exception in writeExternalMeta : > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void writeExternal(ObjectOutput > out) > > > > > throws > > > > > > > > > > > IOException > > > > > > > > > > > > { > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey Goncharuk < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you want, but I > > have > > > a > > > > > few > > > > > > > > > > concerns: > > > > > > > > > > > > > - What is the API for the proposed change? In your > > > test, > > > > > you > > > > > > > > pass > > > > > > > > > an > > > > > > > > > > > > > instance of transaction created on ignite(0) to the > > > > ignite > > > > > > > > instance > > > > > > > > > > > > > ignite(1). This is obviously not possible in a > truly > > > > > > > distributed > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > - How will you synchronize cache update actions and > > > > > > transaction > > > > > > > > > > commit? > > > > > > > > > > > > > Say, you have one node that decided to commit, but > > > > another > > > > > > node > > > > > > > > is > > > > > > > > > > > still > > > > > > > > > > > > > writing within this transaction. How do you make > sure > > > > that > > > > > > two > > > > > > > > > nodes > > > > > > > > > > > will > > > > > > > > > > > > > not call commit() and rollback() simultaneously? > > > > > > > > > > > > > - How do you make sure that either commit() or > > > > rollback() > > > > > is > > > > > > > > > called > > > > > > > > > > if > > > > > > > > > > > > an > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial understanding > was > > > > that > > > > > > > > > > transferring > > > > > > > > > > > > of > > > > > > > > > > > > > tx > > > > > > > > > > > > > > ownership from one node to another will be > happened > > > > > > > > automatically > > > > > > > > > > > when > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on multiple > > threads, > > > > > nodes, > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > So > > > > > > > > > > > > > > > every node is able to rollback, or commit > common > > > > > > > > transaction.It > > > > > > > > > > > > turned > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > need to transfer tx between nodes in order to > > > commit > > > > > > > > > transaction > > > > > > > > > > in > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey Goncharuk < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a concept of > > > transferring > > > > > of > > > > > > tx > > > > > > > > > > > ownership > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > one node to another? My initial understanding > > was > > > > > that > > > > > > > you > > > > > > > > > want > > > > > > > > > > > to > > > > > > > > > > > > be > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > to update keys in a transaction from multiple > > > > threads > > > > > > in > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY KUZNETSOV > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction started in one > > node, > > > > and > > > > > > > > > continued > > > > > > > > > > > in > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > The following test describes my idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions = > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> cache = > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = transactions.txStart( > > > > concurrency, > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> fut = > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > -> > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > TransactionState.STOPPED, > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > TransactionState.COMMITTED, > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we just rebind > > *tx* > > > > to > > > > > > > > current > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction tx) { > > > > > > > > > > > > > > > > > TransactionProxyImpl transactionProxy = > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > transactionProxy.bindToCurrentThread(); > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter *threadMap* > so > > > that > > > > > it > > > > > > > > binds > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis Magda < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational behind this and > > the > > > > > > > thoughts, > > > > > > > > > > > design > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, ALEKSEY > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing distributed > > > transaction > > > > > > which > > > > > > > > can > > > > > > > > > be > > > > > > > > > > > > > started > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > node, and continued at other one. Has > > > anybody > > > > > > > > thoughts > > > > > > > > > on > > > > > > > > > > > it > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > *Best Regards,* *Kuznetsov Aleksey* |
Re: distributed transaction of non-single coordinator
|
|
IMO the use case makes sense. However, as Sergi already mentioned, the
problem is far more complex, than simply passing TX state over a wire. Most probably a kind of coordinator will be required still to manage all kinds of failures. This task should be started with clean design proposal explaining how we handle all these concurrent events. And only then, when we understand all implications, we should move to development stage. On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < [hidden email]> wrote: > Right > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin <[hidden email]>: > > > Good! Basically your orchestrator just takes some predefined graph of > > distributed services to be invoked, calls them by some kind of RPC and > > passes the needed parameters between them, right? > > > > Sergi > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > > > orchestrator is a custom thing. He is responsible for managing business > > > scenarios flows. Many nodes are involved in scenarios. They exchange > data > > > and folow one another. If you acquinted with BPMN framework, so > > > orchestrator is like bpmn engine. > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <[hidden email]>: > > > > > > > What is Orchestrator for you? Is it a thing from Microsoft or your > > custom > > > > in-house software? > > > > > > > > Sergi > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > > >: > > > > > > > > > Fine. Let's say we've got multiple servers which fulfills custom > > logic. > > > > > This servers compound oriented graph (BPMN process) which > controlled > > by > > > > > Orchestrator. > > > > > For instance, *server1 *creates *variable A *with value 1, > persists > > it > > > > to > > > > > IGNITE cache and creates *variable B *and sends it to* server2. > *The > > > > > latests receives *variable B*, do some logic with it and stores to > > > > IGNITE. > > > > > All the work made by both servers must be fulfilled in *one* > > > transaction. > > > > > Because we need all information done, or nothing(rollbacked). The > > > > scenario > > > > > is managed by orchestrator. > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > [hidden email] > > > >: > > > > > > > > > > > Ok, it is not a business case, it is your wrong solution for it. > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > Sergi > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > [hidden email] > > > > >: > > > > > > > > > > > > > The case is the following, One starts transaction in one node, > > and > > > > > commit > > > > > > > this transaction in another jvm node(or rollback it remotely). > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > Because even if you make it work for some simplistic > scenario, > > > get > > > > > > ready > > > > > > > to > > > > > > > > write many fault tolerance tests and make sure that you TXs > > work > > > > > > > gracefully > > > > > > > > in all modes in case of crashes. Also make sure that we do > not > > > have > > > > > any > > > > > > > > performance drops after all your changes in existing > > benchmarks. > > > > All > > > > > in > > > > > > > all > > > > > > > > I don't believe these conditions will be met and your > > > contribution > > > > > will > > > > > > > be > > > > > > > > accepted. > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to another node? > > The > > > > > > problem > > > > > > > > statement itself is already wrong. What business case you are > > > > trying > > > > > to > > > > > > > > solve? I'm sure everything you need can be done in a much > more > > > > simple > > > > > > and > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing it on > another > > > node > > > > > is > > > > > > > > > > meaningless, because other nodes participating in the TX > > have > > > > to > > > > > > know > > > > > > > > > about > > > > > > > > > > the new coordinator. This will require protocol changes, > we > > > > > > > definitely > > > > > > > > > will > > > > > > > > > > have fault tolerance and performance issues. IMO the > whole > > > idea > > > > > is > > > > > > > > wrong > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation contains > > > > IgniteTxEntry's > > > > > > > which > > > > > > > > > is > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are passing > > > > > transaction > > > > > > > > > objects > > > > > > > > > > > > around. Such object may contain all sorts of Ignite > > > > context. > > > > > If > > > > > > > > some > > > > > > > > > > data > > > > > > > > > > > > needs to be passed across, we should create a special > > > > > transfer > > > > > > > > object > > > > > > > > > > in > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing > transaction > > > > > > > proceeding. > > > > > > > > > > > > > At first, After transaction serialization and > > > > > deserialization > > > > > > > on > > > > > > > > > the > > > > > > > > > > > > remote > > > > > > > > > > > > > server, there is no txState. So im going to put it > in > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction lacks of > > > shared > > > > > > cache > > > > > > > > > > context > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it must be > > > > injected > > > > > > by > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing transaction in > > > different > > > > > jvms > > > > > > > in > > > > > > > > > run > > > > > > > > > > > into > > > > > > > > > > > > > > serialization exception in writeExternalMeta : > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void writeExternal(ObjectOutput > > out) > > > > > > throws > > > > > > > > > > > > IOException > > > > > > > > > > > > > { > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey Goncharuk < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you want, but I > > > have > > > > a > > > > > > few > > > > > > > > > > > concerns: > > > > > > > > > > > > > > - What is the API for the proposed change? In > your > > > > test, > > > > > > you > > > > > > > > > pass > > > > > > > > > > an > > > > > > > > > > > > > > instance of transaction created on ignite(0) to > the > > > > > ignite > > > > > > > > > instance > > > > > > > > > > > > > > ignite(1). This is obviously not possible in a > > truly > > > > > > > > distributed > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > - How will you synchronize cache update actions > and > > > > > > > transaction > > > > > > > > > > > commit? > > > > > > > > > > > > > > Say, you have one node that decided to commit, > but > > > > > another > > > > > > > node > > > > > > > > > is > > > > > > > > > > > > still > > > > > > > > > > > > > > writing within this transaction. How do you make > > sure > > > > > that > > > > > > > two > > > > > > > > > > nodes > > > > > > > > > > > > will > > > > > > > > > > > > > > not call commit() and rollback() simultaneously? > > > > > > > > > > > > > > - How do you make sure that either commit() or > > > > > rollback() > > > > > > is > > > > > > > > > > called > > > > > > > > > > > if > > > > > > > > > > > > > an > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial understanding > > was > > > > > that > > > > > > > > > > > transferring > > > > > > > > > > > > > of > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > ownership from one node to another will be > > happened > > > > > > > > > automatically > > > > > > > > > > > > when > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on multiple > > > threads, > > > > > > nodes, > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > every node is able to rollback, or commit > > common > > > > > > > > > transaction.It > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > need to transfer tx between nodes in order to > > > > commit > > > > > > > > > > transaction > > > > > > > > > > > in > > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey > Goncharuk < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a concept of > > > > transferring > > > > > > of > > > > > > > tx > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > one node to another? My initial > understanding > > > was > > > > > > that > > > > > > > > you > > > > > > > > > > want > > > > > > > > > > > > to > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > to update keys in a transaction from > multiple > > > > > threads > > > > > > > in > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY > KUZNETSOV > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction started in one > > > node, > > > > > and > > > > > > > > > > continued > > > > > > > > > > > > in > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > The following test describes my idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions = > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> cache = > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = transactions.txStart( > > > > > concurrency, > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> fut = > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > -> > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > TransactionState.STOPPED, > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > TransactionState.COMMITTED, > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we just > rebind > > > *tx* > > > > > to > > > > > > > > > current > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction tx) { > > > > > > > > > > > > > > > > > > TransactionProxyImpl > transactionProxy = > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > transactionProxy. > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter *threadMap* > > so > > > > that > > > > > > it > > > > > > > > > binds > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis Magda < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational behind this > and > > > the > > > > > > > > thoughts, > > > > > > > > > > > > design > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, ALEKSEY > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing distributed > > > > transaction > > > > > > > which > > > > > > > > > can > > > > > > > > > > be > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > node, and continued at other one. Has > > > > anybody > > > > > > > > > thoughts > > > > > > > > > > on > > > > > > > > > > > > it > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Best Regards,* > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > -- > > > > > > *Best Regards,* > > > > > > *Kuznetsov Aleksey* > > > > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > |
|
|
Okay, we are open for proposals on business task. I mean, we can make use
of some other thing, not distributed transaction. Not transaction yet. ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <[hidden email]>: > IMO the use case makes sense. However, as Sergi already mentioned, the > problem is far more complex, than simply passing TX state over a wire. Most > probably a kind of coordinator will be required still to manage all kinds > of failures. This task should be started with clean design proposal > explaining how we handle all these concurrent events. And only then, when > we understand all implications, we should move to development stage. > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > [hidden email]> wrote: > > > Right > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin <[hidden email]>: > > > > > Good! Basically your orchestrator just takes some predefined graph of > > > distributed services to be invoked, calls them by some kind of RPC and > > > passes the needed parameters between them, right? > > > > > > Sergi > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > > > orchestrator is a custom thing. He is responsible for managing > business > > > > scenarios flows. Many nodes are involved in scenarios. They exchange > > data > > > > and folow one another. If you acquinted with BPMN framework, so > > > > orchestrator is like bpmn engine. > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <[hidden email]>: > > > > > > > > > What is Orchestrator for you? Is it a thing from Microsoft or your > > > custom > > > > > in-house software? > > > > > > > > > > Sergi > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > [hidden email] > > > >: > > > > > > > > > > > Fine. Let's say we've got multiple servers which fulfills custom > > > logic. > > > > > > This servers compound oriented graph (BPMN process) which > > controlled > > > by > > > > > > Orchestrator. > > > > > > For instance, *server1 *creates *variable A *with value 1, > > persists > > > it > > > > > to > > > > > > IGNITE cache and creates *variable B *and sends it to* server2. > > *The > > > > > > latests receives *variable B*, do some logic with it and stores > to > > > > > IGNITE. > > > > > > All the work made by both servers must be fulfilled in *one* > > > > transaction. > > > > > > Because we need all information done, or nothing(rollbacked). The > > > > > scenario > > > > > > is managed by orchestrator. > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong solution for > it. > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > The case is the following, One starts transaction in one > node, > > > and > > > > > > commit > > > > > > > > this transaction in another jvm node(or rollback it > remotely). > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > Because even if you make it work for some simplistic > > scenario, > > > > get > > > > > > > ready > > > > > > > > to > > > > > > > > > write many fault tolerance tests and make sure that you TXs > > > work > > > > > > > > gracefully > > > > > > > > > in all modes in case of crashes. Also make sure that we do > > not > > > > have > > > > > > any > > > > > > > > > performance drops after all your changes in existing > > > benchmarks. > > > > > All > > > > > > in > > > > > > > > all > > > > > > > > > I don't believe these conditions will be met and your > > > > contribution > > > > > > will > > > > > > > > be > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to another > node? > > > The > > > > > > > problem > > > > > > > > > statement itself is already wrong. What business case you > are > > > > > trying > > > > > > to > > > > > > > > > solve? I'm sure everything you need can be done in a much > > more > > > > > simple > > > > > > > and > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing it on > > another > > > > node > > > > > > is > > > > > > > > > > > meaningless, because other nodes participating in the > TX > > > have > > > > > to > > > > > > > know > > > > > > > > > > about > > > > > > > > > > > the new coordinator. This will require protocol > changes, > > we > > > > > > > > definitely > > > > > > > > > > will > > > > > > > > > > > have fault tolerance and performance issues. IMO the > > whole > > > > idea > > > > > > is > > > > > > > > > wrong > > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation contains > > > > > IgniteTxEntry's > > > > > > > > which > > > > > > > > > > is > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are passing > > > > > > transaction > > > > > > > > > > objects > > > > > > > > > > > > > around. Such object may contain all sorts of Ignite > > > > > context. > > > > > > If > > > > > > > > > some > > > > > > > > > > > data > > > > > > > > > > > > > needs to be passed across, we should create a > special > > > > > > transfer > > > > > > > > > object > > > > > > > > > > > in > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY KUZNETSOV > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing > > transaction > > > > > > > > proceeding. > > > > > > > > > > > > > > At first, After transaction serialization and > > > > > > deserialization > > > > > > > > on > > > > > > > > > > the > > > > > > > > > > > > > remote > > > > > > > > > > > > > > server, there is no txState. So im going to put > it > > in > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction lacks of > > > > shared > > > > > > > cache > > > > > > > > > > > context > > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it must > be > > > > > injected > > > > > > > by > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing transaction in > > > > different > > > > > > jvms > > > > > > > > in > > > > > > > > > > run > > > > > > > > > > > > into > > > > > > > > > > > > > > > serialization exception in writeExternalMeta : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > writeExternal(ObjectOutput > > > out) > > > > > > > throws > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey Goncharuk < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you want, > but I > > > > have > > > > > a > > > > > > > few > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > - What is the API for the proposed change? In > > your > > > > > test, > > > > > > > you > > > > > > > > > > pass > > > > > > > > > > > an > > > > > > > > > > > > > > > instance of transaction created on ignite(0) to > > the > > > > > > ignite > > > > > > > > > > instance > > > > > > > > > > > > > > > ignite(1). This is obviously not possible in a > > > truly > > > > > > > > > distributed > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > - How will you synchronize cache update actions > > and > > > > > > > > transaction > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > Say, you have one node that decided to commit, > > but > > > > > > another > > > > > > > > node > > > > > > > > > > is > > > > > > > > > > > > > still > > > > > > > > > > > > > > > writing within this transaction. How do you > make > > > sure > > > > > > that > > > > > > > > two > > > > > > > > > > > nodes > > > > > > > > > > > > > will > > > > > > > > > > > > > > > not call commit() and rollback() > simultaneously? > > > > > > > > > > > > > > > - How do you make sure that either commit() or > > > > > > rollback() > > > > > > > is > > > > > > > > > > > called > > > > > > > > > > > > if > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial > understanding > > > was > > > > > > that > > > > > > > > > > > > transferring > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > ownership from one node to another will be > > > happened > > > > > > > > > > automatically > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY KUZNETSOV > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on multiple > > > > threads, > > > > > > > nodes, > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > every node is able to rollback, or commit > > > common > > > > > > > > > > transaction.It > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > need to transfer tx between nodes in order > to > > > > > commit > > > > > > > > > > > transaction > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey > > Goncharuk < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a concept of > > > > > transferring > > > > > > > of > > > > > > > > tx > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > one node to another? My initial > > understanding > > > > was > > > > > > > that > > > > > > > > > you > > > > > > > > > > > want > > > > > > > > > > > > > to > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > to update keys in a transaction from > > multiple > > > > > > threads > > > > > > > > in > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY > > KUZNETSOV > > > < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction started in > one > > > > node, > > > > > > and > > > > > > > > > > > continued > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > The following test describes my idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions = > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> cache = > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = transactions.txStart( > > > > > > concurrency, > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> fut = > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > -> > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > TransactionState.COMMITTED, > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we just > > rebind > > > > *tx* > > > > > > to > > > > > > > > > > current > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction tx) { > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > transactionProxy = > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > transactionProxy. > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter > *threadMap* > > > so > > > > > that > > > > > > > it > > > > > > > > > > binds > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis > Magda < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational behind this > > and > > > > the > > > > > > > > > thoughts, > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, ALEKSEY > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing distributed > > > > > transaction > > > > > > > > which > > > > > > > > > > can > > > > > > > > > > > be > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > node, and continued at other one. > Has > > > > > anybody > > > > > > > > > > thoughts > > > > > > > > > > > on > > > > > > > > > > > > > it > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > *Best Regards,* *Kuznetsov Aleksey* |
|
|
Ok, here is what you actually need to implement at the application level.
Lets say we have to call 2 services in the following order: - Service A: wants to update keys [k1 => v1, k2 => v2] to [k1 => v1a, k2 => v2a] - Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 => v2ab, k3 => v3b] The change from [ k1 => v1, k2 => v2, k3 => v3 ] to [ k1 => v1a, k2 => v2ab, k3 => v3b ] must happen in a single transaction. Optimistic protocol to solve this: Each cache key must have a field `otx`, which is a unique orchestrator TX identifier - it must be a parameter passed to all the services. If `otx` is set to some value it means that it is an intermediate key and is visible only inside of some transaction, for the finalized key `otx` must be null - it means the key is committed and visible for everyone. Each cache value must have a field `ver` which is a version of that value. For both fields (`otx` and `ver`) the safest way is to use UUID. Workflow is the following: Orchestrator starts the distributed transaction with `otx` = x and passes this parameter to all the services. Service A: - does some computations - stores [k1x => v1a, k2x => v2a] with TTL = Za where Za - left time from max Orchestrator TX duration after Service A end k1x, k2x - new temporary keys with field `otx` = x v2a has updated version `ver` - returns a set of updated keys and all the old versions to the orchestrator or just stores it in some special atomic cache like [x => (k1 -> ver1, k2 -> ver2)] TTL = Za Service B: - retrieves the updated value k2x => v2a because it knows `otx` = x - does computations - stores [k2x => v2ab, k3x => v3b] TTL = Zb - updates the set of updated keys like [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] TTL = Zb Service Committer (may be embedded into Orchestrator): - takes all the updated keys and versions for `otx` = x [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] - in a single transaction checks value versions for all the old values and replaces them with calculated new ones - does cleanup of temporary keys and values - in case of version mismatch or TX timeout just rollbacks and signals to Orchestrator to restart the job with new `otx` PROFIT!! This approach even allows you to run independent parts of the graph in parallel (with TX transfer you will always run only one at a time). Also it does not require inventing any special fault tolerance technics because Ignite caches are already fault tolerant and all the intermediate results are virtually invisible and stored with TTL, thus in case of any crash you will not have inconsistent state or garbage. Sergi 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > Okay, we are open for proposals on business task. I mean, we can make use > of some other thing, not distributed transaction. Not transaction yet. > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <[hidden email]>: > > > IMO the use case makes sense. However, as Sergi already mentioned, the > > problem is far more complex, than simply passing TX state over a wire. > Most > > probably a kind of coordinator will be required still to manage all kinds > > of failures. This task should be started with clean design proposal > > explaining how we handle all these concurrent events. And only then, when > > we understand all implications, we should move to development stage. > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > [hidden email]> wrote: > > > > > Right > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin <[hidden email] > >: > > > > > > > Good! Basically your orchestrator just takes some predefined graph of > > > > distributed services to be invoked, calls them by some kind of RPC > and > > > > passes the needed parameters between them, right? > > > > > > > > Sergi > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > > >: > > > > > > > > > orchestrator is a custom thing. He is responsible for managing > > business > > > > > scenarios flows. Many nodes are involved in scenarios. They > exchange > > > data > > > > > and folow one another. If you acquinted with BPMN framework, so > > > > > orchestrator is like bpmn engine. > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <[hidden email] > >: > > > > > > > > > > > What is Orchestrator for you? Is it a thing from Microsoft or > your > > > > custom > > > > > > in-house software? > > > > > > > > > > > > Sergi > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which fulfills > custom > > > > logic. > > > > > > > This servers compound oriented graph (BPMN process) which > > > controlled > > > > by > > > > > > > Orchestrator. > > > > > > > For instance, *server1 *creates *variable A *with value 1, > > > persists > > > > it > > > > > > to > > > > > > > IGNITE cache and creates *variable B *and sends it to* server2. > > > *The > > > > > > > latests receives *variable B*, do some logic with it and stores > > to > > > > > > IGNITE. > > > > > > > All the work made by both servers must be fulfilled in *one* > > > > > transaction. > > > > > > > Because we need all information done, or nothing(rollbacked). > The > > > > > > scenario > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong solution for > > it. > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > The case is the following, One starts transaction in one > > node, > > > > and > > > > > > > commit > > > > > > > > > this transaction in another jvm node(or rollback it > > remotely). > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > Because even if you make it work for some simplistic > > > scenario, > > > > > get > > > > > > > > ready > > > > > > > > > to > > > > > > > > > > write many fault tolerance tests and make sure that you > TXs > > > > work > > > > > > > > > gracefully > > > > > > > > > > in all modes in case of crashes. Also make sure that we > do > > > not > > > > > have > > > > > > > any > > > > > > > > > > performance drops after all your changes in existing > > > > benchmarks. > > > > > > All > > > > > > > in > > > > > > > > > all > > > > > > > > > > I don't believe these conditions will be met and your > > > > > contribution > > > > > > > will > > > > > > > > > be > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to another > > node? > > > > The > > > > > > > > problem > > > > > > > > > > statement itself is already wrong. What business case you > > are > > > > > > trying > > > > > > > to > > > > > > > > > > solve? I'm sure everything you need can be done in a much > > > more > > > > > > simple > > > > > > > > and > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing it on > > > another > > > > > node > > > > > > > is > > > > > > > > > > > > meaningless, because other nodes participating in the > > TX > > > > have > > > > > > to > > > > > > > > know > > > > > > > > > > > about > > > > > > > > > > > > the new coordinator. This will require protocol > > changes, > > > we > > > > > > > > > definitely > > > > > > > > > > > will > > > > > > > > > > > > have fault tolerance and performance issues. IMO the > > > whole > > > > > idea > > > > > > > is > > > > > > > > > > wrong > > > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation contains > > > > > > IgniteTxEntry's > > > > > > > > > which > > > > > > > > > > > is > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are > passing > > > > > > > transaction > > > > > > > > > > > objects > > > > > > > > > > > > > > around. Such object may contain all sorts of > Ignite > > > > > > context. > > > > > > > If > > > > > > > > > > some > > > > > > > > > > > > data > > > > > > > > > > > > > > needs to be passed across, we should create a > > special > > > > > > > transfer > > > > > > > > > > object > > > > > > > > > > > > in > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY > KUZNETSOV > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing > > > transaction > > > > > > > > > proceeding. > > > > > > > > > > > > > > > At first, After transaction serialization and > > > > > > > deserialization > > > > > > > > > on > > > > > > > > > > > the > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > server, there is no txState. So im going to put > > it > > > in > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction lacks > of > > > > > shared > > > > > > > > cache > > > > > > > > > > > > context > > > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it must > > be > > > > > > injected > > > > > > > > by > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY KUZNETSOV > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing transaction in > > > > > different > > > > > > > jvms > > > > > > > > > in > > > > > > > > > > > run > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > serialization exception in writeExternalMeta > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > writeExternal(ObjectOutput > > > > out) > > > > > > > > throws > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey > Goncharuk < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you want, > > but I > > > > > have > > > > > > a > > > > > > > > few > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > - What is the API for the proposed change? > In > > > your > > > > > > test, > > > > > > > > you > > > > > > > > > > > pass > > > > > > > > > > > > an > > > > > > > > > > > > > > > > instance of transaction created on ignite(0) > to > > > the > > > > > > > ignite > > > > > > > > > > > instance > > > > > > > > > > > > > > > > ignite(1). This is obviously not possible in > a > > > > truly > > > > > > > > > > distributed > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > - How will you synchronize cache update > actions > > > and > > > > > > > > > transaction > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > Say, you have one node that decided to > commit, > > > but > > > > > > > another > > > > > > > > > node > > > > > > > > > > > is > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > writing within this transaction. How do you > > make > > > > sure > > > > > > > that > > > > > > > > > two > > > > > > > > > > > > nodes > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > not call commit() and rollback() > > simultaneously? > > > > > > > > > > > > > > > > - How do you make sure that either commit() > or > > > > > > > rollback() > > > > > > > > is > > > > > > > > > > > > called > > > > > > > > > > > > > if > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial > > understanding > > > > was > > > > > > > that > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > ownership from one node to another will be > > > > happened > > > > > > > > > > > automatically > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY > KUZNETSOV > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on multiple > > > > > threads, > > > > > > > > nodes, > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > every node is able to rollback, or commit > > > > common > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > need to transfer tx between nodes in > order > > to > > > > > > commit > > > > > > > > > > > > transaction > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey > > > Goncharuk < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a concept of > > > > > > transferring > > > > > > > > of > > > > > > > > > tx > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > one node to another? My initial > > > understanding > > > > > was > > > > > > > > that > > > > > > > > > > you > > > > > > > > > > > > want > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > to update keys in a transaction from > > > multiple > > > > > > > threads > > > > > > > > > in > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY > > > KUZNETSOV > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction started in > > one > > > > > node, > > > > > > > and > > > > > > > > > > > > continued > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > The following test describes my idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions = > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> cache = > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > transactions.txStart( > > > > > > > concurrency, > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> fut = > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we just > > > rebind > > > > > *tx* > > > > > > > to > > > > > > > > > > > current > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction tx) { > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > transactionProxy = > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter > > *threadMap* > > > > so > > > > > > that > > > > > > > > it > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis > > Magda < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational behind > this > > > and > > > > > the > > > > > > > > > > thoughts, > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, > ALEKSEY > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing distributed > > > > > > transaction > > > > > > > > > which > > > > > > > > > > > can > > > > > > > > > > > > be > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > node, and continued at other one. > > Has > > > > > > anybody > > > > > > > > > > > thoughts > > > > > > > > > > > > on > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Best Regards,* > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > -- > > > > > > *Best Regards,* > > > > > > *Kuznetsov Aleksey* > > > > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > |
|
|
- what do u mean by saying "
*in a single transaction checks value versions for all the old values and replaces them with calculated new ones *"? Every time you change value(in some service), you store it to *some special atomic cache* , so when all services ceased working, Service commiter got a values with the last versions. - After "*does cleanup of temporary keys and values*" Service commiter persists them into permanent store, isn't it ? - I cant grasp your though, you say "*in case of version mismatch or TX timeout just rollbacks*". But what versions would it match? ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin <[hidden email]>: > Ok, here is what you actually need to implement at the application level. > > Lets say we have to call 2 services in the following order: > - Service A: wants to update keys [k1 => v1, k2 => v2] to [k1 => v1a, > k2 => v2a] > - Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 => v2ab, > k3 => v3b] > > The change > from [ k1 => v1, k2 => v2, k3 => v3 ] > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > must happen in a single transaction. > > > Optimistic protocol to solve this: > > Each cache key must have a field `otx`, which is a unique orchestrator TX > identifier - it must be a parameter passed to all the services. If `otx` is > set to some value it means that it is an intermediate key and is visible > only inside of some transaction, for the finalized key `otx` must be null - > it means the key is committed and visible for everyone. > > Each cache value must have a field `ver` which is a version of that value. > > For both fields (`otx` and `ver`) the safest way is to use UUID. > > Workflow is the following: > > Orchestrator starts the distributed transaction with `otx` = x and passes > this parameter to all the services. > > Service A: > - does some computations > - stores [k1x => v1a, k2x => v2a] with TTL = Za > where > Za - left time from max Orchestrator TX duration after Service A > end > k1x, k2x - new temporary keys with field `otx` = x > v2a has updated version `ver` > - returns a set of updated keys and all the old versions to the > orchestrator > or just stores it in some special atomic cache like > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > Service B: > - retrieves the updated value k2x => v2a because it knows `otx` = x > - does computations > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > - updates the set of updated keys like [x => (k1 -> ver1, k2 -> ver2, k3 > -> ver3)] TTL = Zb > > Service Committer (may be embedded into Orchestrator): > - takes all the updated keys and versions for `otx` = x > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > - in a single transaction checks value versions for all the old values > and replaces them with calculated new ones > - does cleanup of temporary keys and values > - in case of version mismatch or TX timeout just rollbacks and signals > to Orchestrator to restart the job with new `otx` > > PROFIT!! > > This approach even allows you to run independent parts of the graph in > parallel (with TX transfer you will always run only one at a time). Also it > does not require inventing any special fault tolerance technics because > Ignite caches are already fault tolerant and all the intermediate results > are virtually invisible and stored with TTL, thus in case of any crash you > will not have inconsistent state or garbage. > > Sergi > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > Okay, we are open for proposals on business task. I mean, we can make use > > of some other thing, not distributed transaction. Not transaction yet. > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <[hidden email]>: > > > > > IMO the use case makes sense. However, as Sergi already mentioned, the > > > problem is far more complex, than simply passing TX state over a wire. > > Most > > > probably a kind of coordinator will be required still to manage all > kinds > > > of failures. This task should be started with clean design proposal > > > explaining how we handle all these concurrent events. And only then, > when > > > we understand all implications, we should move to development stage. > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > [hidden email]> wrote: > > > > > > > Right > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > [hidden email] > > >: > > > > > > > > > Good! Basically your orchestrator just takes some predefined graph > of > > > > > distributed services to be invoked, calls them by some kind of RPC > > and > > > > > passes the needed parameters between them, right? > > > > > > > > > > Sergi > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > [hidden email] > > > >: > > > > > > > > > > > orchestrator is a custom thing. He is responsible for managing > > > business > > > > > > scenarios flows. Many nodes are involved in scenarios. They > > exchange > > > > data > > > > > > and folow one another. If you acquinted with BPMN framework, so > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > [hidden email] > > >: > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from Microsoft or > > your > > > > > custom > > > > > > > in-house software? > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which fulfills > > custom > > > > > logic. > > > > > > > > This servers compound oriented graph (BPMN process) which > > > > controlled > > > > > by > > > > > > > > Orchestrator. > > > > > > > > For instance, *server1 *creates *variable A *with value 1, > > > > persists > > > > > it > > > > > > > to > > > > > > > > IGNITE cache and creates *variable B *and sends it to* > server2. > > > > *The > > > > > > > > latests receives *variable B*, do some logic with it and > stores > > > to > > > > > > > IGNITE. > > > > > > > > All the work made by both servers must be fulfilled in *one* > > > > > > transaction. > > > > > > > > Because we need all information done, or nothing(rollbacked). > > The > > > > > > > scenario > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong solution > for > > > it. > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > The case is the following, One starts transaction in one > > > node, > > > > > and > > > > > > > > commit > > > > > > > > > > this transaction in another jvm node(or rollback it > > > remotely). > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some simplistic > > > > scenario, > > > > > > get > > > > > > > > > ready > > > > > > > > > > to > > > > > > > > > > > write many fault tolerance tests and make sure that you > > TXs > > > > > work > > > > > > > > > > gracefully > > > > > > > > > > > in all modes in case of crashes. Also make sure that we > > do > > > > not > > > > > > have > > > > > > > > any > > > > > > > > > > > performance drops after all your changes in existing > > > > > benchmarks. > > > > > > > All > > > > > > > > in > > > > > > > > > > all > > > > > > > > > > > I don't believe these conditions will be met and your > > > > > > contribution > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to another > > > node? > > > > > The > > > > > > > > > problem > > > > > > > > > > > statement itself is already wrong. What business case > you > > > are > > > > > > > trying > > > > > > > > to > > > > > > > > > > > solve? I'm sure everything you need can be done in a > much > > > > more > > > > > > > simple > > > > > > > > > and > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing it on > > > > another > > > > > > node > > > > > > > > is > > > > > > > > > > > > > meaningless, because other nodes participating in > the > > > TX > > > > > have > > > > > > > to > > > > > > > > > know > > > > > > > > > > > > about > > > > > > > > > > > > > the new coordinator. This will require protocol > > > changes, > > > > we > > > > > > > > > > definitely > > > > > > > > > > > > will > > > > > > > > > > > > > have fault tolerance and performance issues. IMO > the > > > > whole > > > > > > idea > > > > > > > > is > > > > > > > > > > > wrong > > > > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation contains > > > > > > > IgniteTxEntry's > > > > > > > > > > which > > > > > > > > > > > > is > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are > > passing > > > > > > > > transaction > > > > > > > > > > > > objects > > > > > > > > > > > > > > > around. Such object may contain all sorts of > > Ignite > > > > > > > context. > > > > > > > > If > > > > > > > > > > > some > > > > > > > > > > > > > data > > > > > > > > > > > > > > > needs to be passed across, we should create a > > > special > > > > > > > > transfer > > > > > > > > > > > object > > > > > > > > > > > > > in > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY > > KUZNETSOV > > > < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing > > > > transaction > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > At first, After transaction serialization and > > > > > > > > deserialization > > > > > > > > > > on > > > > > > > > > > > > the > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > server, there is no txState. So im going to > put > > > it > > > > in > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction > lacks > > of > > > > > > shared > > > > > > > > > cache > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it > must > > > be > > > > > > > injected > > > > > > > > > by > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY > KUZNETSOV > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing transaction > in > > > > > > different > > > > > > > > jvms > > > > > > > > > > in > > > > > > > > > > > > run > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > serialization exception in > writeExternalMeta > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > writeExternal(ObjectOutput > > > > > out) > > > > > > > > > throws > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey > > Goncharuk < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you want, > > > but I > > > > > > have > > > > > > > a > > > > > > > > > few > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > - What is the API for the proposed change? > > In > > > > your > > > > > > > test, > > > > > > > > > you > > > > > > > > > > > > pass > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > instance of transaction created on > ignite(0) > > to > > > > the > > > > > > > > ignite > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > ignite(1). This is obviously not possible > in > > a > > > > > truly > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > - How will you synchronize cache update > > actions > > > > and > > > > > > > > > > transaction > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > Say, you have one node that decided to > > commit, > > > > but > > > > > > > > another > > > > > > > > > > node > > > > > > > > > > > > is > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > writing within this transaction. How do you > > > make > > > > > sure > > > > > > > > that > > > > > > > > > > two > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > not call commit() and rollback() > > > simultaneously? > > > > > > > > > > > > > > > > > - How do you make sure that either > commit() > > or > > > > > > > > rollback() > > > > > > > > > is > > > > > > > > > > > > > called > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial > > > understanding > > > > > was > > > > > > > > that > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > ownership from one node to another will > be > > > > > happened > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY > > KUZNETSOV > > > < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on > multiple > > > > > > threads, > > > > > > > > > nodes, > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > every node is able to rollback, or > commit > > > > > common > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > need to transfer tx between nodes in > > order > > > to > > > > > > > commit > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey > > > > Goncharuk < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a concept > of > > > > > > > transferring > > > > > > > > > of > > > > > > > > > > tx > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > one node to another? My initial > > > > understanding > > > > > > was > > > > > > > > > that > > > > > > > > > > > you > > > > > > > > > > > > > want > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > to update keys in a transaction from > > > > multiple > > > > > > > > threads > > > > > > > > > > in > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY > > > > KUZNETSOV > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction started > in > > > one > > > > > > node, > > > > > > > > and > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > The following test describes my > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions = > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> cache > = > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > transactions.txStart( > > > > > > > > concurrency, > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> fut = > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we just > > > > rebind > > > > > > *tx* > > > > > > > > to > > > > > > > > > > > > current > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction > tx) { > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > transactionProxy = > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter > > > *threadMap* > > > > > so > > > > > > > that > > > > > > > > > it > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis > > > Magda < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational behind > > this > > > > and > > > > > > the > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, > > ALEKSEY > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing > distributed > > > > > > > transaction > > > > > > > > > > which > > > > > > > > > > > > can > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > node, and continued at other > one. > > > Has > > > > > > > anybody > > > > > > > > > > > > thoughts > > > > > > > > > > > > > on > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > *Best Regards,* *Kuznetsov Aleksey* |
|
|
All the services do not update key in place, but only generate new keys
augmented by otx and store the updated value in the same cache + remember the keys and versions participating in the transaction in some separate atomic cache. Follow this sequence of changes applied to cache contents by each Service: Initial cache contents: [k1 => v1] [k2 => v2] [k3 => v3] Cache contents after Service A: [k1 => v1] [k2 => v2] [k3 => v3] [k1x => v1a] [k2x => v2a] + [x => (k1 -> ver1, k2 -> ver2)] in some separate atomic cache Cache contents after Service B: [k1 => v1] [k2 => v2] [k3 => v3] [k1x => v1a] [k2x => v2ab] [k3x => v3b] + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate atomic cache Finally the Committer Service takes this map of updated keys and their versions from some separate atomic cache, starts Ignite transaction and replaces all the values for k* keys to values taken from k*x keys. The successful result must be the following: [k1 => v1a] [k2 => v2ab] [k3 => v3b] [k1x => v1a] [k2x => v2ab] [k3x => v3b] + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate atomic cache But Committer Service also has to check that no one updated the original values before us, because otherwise we can not give any serializability guarantee for these distributed transactions. Here we may need to check not only versions of the updated keys, but also versions of any other keys end result depends on. After that Committer Service has to do a cleanup (may be outside of the committing tx) to come to the following final state: [k1 => v1a] [k2 => v2ab] [k3 => v3b] Makes sense? Sergi 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > - what do u mean by saying " > *in a single transaction checks value versions for all the old values > and replaces them with calculated new ones *"? Every time you change > value(in some service), you store it to *some special atomic cache* , so > when all services ceased working, Service commiter got a values with the > last versions. > - After "*does cleanup of temporary keys and values*" Service commiter > persists them into permanent store, isn't it ? > - I cant grasp your though, you say "*in case of version mismatch or TX > timeout just rollbacks*". But what versions would it match? > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin <[hidden email]>: > > > Ok, here is what you actually need to implement at the application level. > > > > Lets say we have to call 2 services in the following order: > > - Service A: wants to update keys [k1 => v1, k2 => v2] to [k1 => > v1a, > > k2 => v2a] > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 => > v2ab, > > k3 => v3b] > > > > The change > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > must happen in a single transaction. > > > > > > Optimistic protocol to solve this: > > > > Each cache key must have a field `otx`, which is a unique orchestrator TX > > identifier - it must be a parameter passed to all the services. If `otx` > is > > set to some value it means that it is an intermediate key and is visible > > only inside of some transaction, for the finalized key `otx` must be > null - > > it means the key is committed and visible for everyone. > > > > Each cache value must have a field `ver` which is a version of that > value. > > > > For both fields (`otx` and `ver`) the safest way is to use UUID. > > > > Workflow is the following: > > > > Orchestrator starts the distributed transaction with `otx` = x and passes > > this parameter to all the services. > > > > Service A: > > - does some computations > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > where > > Za - left time from max Orchestrator TX duration after Service > A > > end > > k1x, k2x - new temporary keys with field `otx` = x > > v2a has updated version `ver` > > - returns a set of updated keys and all the old versions to the > > orchestrator > > or just stores it in some special atomic cache like > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > Service B: > > - retrieves the updated value k2x => v2a because it knows `otx` = x > > - does computations > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > - updates the set of updated keys like [x => (k1 -> ver1, k2 -> ver2, k3 > > -> ver3)] TTL = Zb > > > > Service Committer (may be embedded into Orchestrator): > > - takes all the updated keys and versions for `otx` = x > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > - in a single transaction checks value versions for all the old values > > and replaces them with calculated new ones > > - does cleanup of temporary keys and values > > - in case of version mismatch or TX timeout just rollbacks and signals > > to Orchestrator to restart the job with new `otx` > > > > PROFIT!! > > > > This approach even allows you to run independent parts of the graph in > > parallel (with TX transfer you will always run only one at a time). Also > it > > does not require inventing any special fault tolerance technics because > > Ignite caches are already fault tolerant and all the intermediate results > > are virtually invisible and stored with TTL, thus in case of any crash > you > > will not have inconsistent state or garbage. > > > > Sergi > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > > > Okay, we are open for proposals on business task. I mean, we can make > use > > > of some other thing, not distributed transaction. Not transaction yet. > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <[hidden email]>: > > > > > > > IMO the use case makes sense. However, as Sergi already mentioned, > the > > > > problem is far more complex, than simply passing TX state over a > wire. > > > Most > > > > probably a kind of coordinator will be required still to manage all > > kinds > > > > of failures. This task should be started with clean design proposal > > > > explaining how we handle all these concurrent events. And only then, > > when > > > > we understand all implications, we should move to development stage. > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > [hidden email]> wrote: > > > > > > > > > Right > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > [hidden email] > > > >: > > > > > > > > > > > Good! Basically your orchestrator just takes some predefined > graph > > of > > > > > > distributed services to be invoked, calls them by some kind of > RPC > > > and > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > Sergi > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > [hidden email] > > > > >: > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible for managing > > > > business > > > > > > > scenarios flows. Many nodes are involved in scenarios. They > > > exchange > > > > > data > > > > > > > and folow one another. If you acquinted with BPMN framework, so > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > [hidden email] > > > >: > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from Microsoft or > > > your > > > > > > custom > > > > > > > > in-house software? > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which fulfills > > > custom > > > > > > logic. > > > > > > > > > This servers compound oriented graph (BPMN process) which > > > > > controlled > > > > > > by > > > > > > > > > Orchestrator. > > > > > > > > > For instance, *server1 *creates *variable A *with value 1, > > > > > persists > > > > > > it > > > > > > > > to > > > > > > > > > IGNITE cache and creates *variable B *and sends it to* > > server2. > > > > > *The > > > > > > > > > latests receives *variable B*, do some logic with it and > > stores > > > > to > > > > > > > > IGNITE. > > > > > > > > > All the work made by both servers must be fulfilled in > *one* > > > > > > > transaction. > > > > > > > > > Because we need all information done, or > nothing(rollbacked). > > > The > > > > > > > > scenario > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong solution > > for > > > > it. > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts transaction in > one > > > > node, > > > > > > and > > > > > > > > > commit > > > > > > > > > > > this transaction in another jvm node(or rollback it > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some simplistic > > > > > scenario, > > > > > > > get > > > > > > > > > > ready > > > > > > > > > > > to > > > > > > > > > > > > write many fault tolerance tests and make sure that > you > > > TXs > > > > > > work > > > > > > > > > > > gracefully > > > > > > > > > > > > in all modes in case of crashes. Also make sure that > we > > > do > > > > > not > > > > > > > have > > > > > > > > > any > > > > > > > > > > > > performance drops after all your changes in existing > > > > > > benchmarks. > > > > > > > > All > > > > > > > > > in > > > > > > > > > > > all > > > > > > > > > > > > I don't believe these conditions will be met and your > > > > > > > contribution > > > > > > > > > will > > > > > > > > > > > be > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to > another > > > > node? > > > > > > The > > > > > > > > > > problem > > > > > > > > > > > > statement itself is already wrong. What business case > > you > > > > are > > > > > > > > trying > > > > > > > > > to > > > > > > > > > > > > solve? I'm sure everything you need can be done in a > > much > > > > > more > > > > > > > > simple > > > > > > > > > > and > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing it > on > > > > > another > > > > > > > node > > > > > > > > > is > > > > > > > > > > > > > > meaningless, because other nodes participating in > > the > > > > TX > > > > > > have > > > > > > > > to > > > > > > > > > > know > > > > > > > > > > > > > about > > > > > > > > > > > > > > the new coordinator. This will require protocol > > > > changes, > > > > > we > > > > > > > > > > > definitely > > > > > > > > > > > > > will > > > > > > > > > > > > > > have fault tolerance and performance issues. IMO > > the > > > > > whole > > > > > > > idea > > > > > > > > > is > > > > > > > > > > > > wrong > > > > > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation contains > > > > > > > > IgniteTxEntry's > > > > > > > > > > > which > > > > > > > > > > > > > is > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan > < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are > > > passing > > > > > > > > > transaction > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > around. Such object may contain all sorts of > > > Ignite > > > > > > > > context. > > > > > > > > > If > > > > > > > > > > > > some > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > needs to be passed across, we should create a > > > > special > > > > > > > > > transfer > > > > > > > > > > > > object > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY > > > KUZNETSOV > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing > > > > > transaction > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > At first, After transaction serialization > and > > > > > > > > > deserialization > > > > > > > > > > > on > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > server, there is no txState. So im going to > > put > > > > it > > > > > in > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction > > lacks > > > of > > > > > > > shared > > > > > > > > > > cache > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, it > > must > > > > be > > > > > > > > injected > > > > > > > > > > by > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY > > KUZNETSOV > > > < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing transaction > > in > > > > > > > different > > > > > > > > > jvms > > > > > > > > > > > in > > > > > > > > > > > > > run > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > serialization exception in > > writeExternalMeta > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > writeExternal(ObjectOutput > > > > > > out) > > > > > > > > > > throws > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey > > > Goncharuk < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you > want, > > > > but I > > > > > > > have > > > > > > > > a > > > > > > > > > > few > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > - What is the API for the proposed > change? > > > In > > > > > your > > > > > > > > test, > > > > > > > > > > you > > > > > > > > > > > > > pass > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > instance of transaction created on > > ignite(0) > > > to > > > > > the > > > > > > > > > ignite > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > ignite(1). This is obviously not possible > > in > > > a > > > > > > truly > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > - How will you synchronize cache update > > > actions > > > > > and > > > > > > > > > > > transaction > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > Say, you have one node that decided to > > > commit, > > > > > but > > > > > > > > > another > > > > > > > > > > > node > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > writing within this transaction. How do > you > > > > make > > > > > > sure > > > > > > > > > that > > > > > > > > > > > two > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > not call commit() and rollback() > > > > simultaneously? > > > > > > > > > > > > > > > > > > - How do you make sure that either > > commit() > > > or > > > > > > > > > rollback() > > > > > > > > > > is > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов > < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial > > > > understanding > > > > > > was > > > > > > > > > that > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > ownership from one node to another will > > be > > > > > > happened > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY > > > KUZNETSOV > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on > > multiple > > > > > > > threads, > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > every node is able to rollback, or > > commit > > > > > > common > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > need to transfer tx between nodes in > > > order > > > > to > > > > > > > > commit > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a concept > > of > > > > > > > > transferring > > > > > > > > > > of > > > > > > > > > > > tx > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > one node to another? My initial > > > > > understanding > > > > > > > was > > > > > > > > > > that > > > > > > > > > > > > you > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > to update keys in a transaction > from > > > > > multiple > > > > > > > > > threads > > > > > > > > > > > in > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY > > > > > KUZNETSOV > > > > > > < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction > started > > in > > > > one > > > > > > > node, > > > > > > > > > and > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > The following test describes my > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions transactions = > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> > cache > > = > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > transactions.txStart( > > > > > > > > > concurrency, > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > fut = > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we > just > > > > > rebind > > > > > > > *tx* > > > > > > > > > to > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction > > tx) { > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter > > > > *threadMap* > > > > > > so > > > > > > > > that > > > > > > > > > > it > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, Denis > > > > Magda < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational > behind > > > this > > > > > and > > > > > > > the > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, > > > ALEKSEY > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing > > distributed > > > > > > > > transaction > > > > > > > > > > > which > > > > > > > > > > > > > can > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > node, and continued at other > > one. > > > > Has > > > > > > > > anybody > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Best Regards,* > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > -- > > > > > > *Best Regards,* > > > > > > *Kuznetsov Aleksey* > > > > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > |
|
|
Thank you very much for help. I will answer later.
ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin <[hidden email]>: > All the services do not update key in place, but only generate new keys > augmented by otx and store the updated value in the same cache + remember > the keys and versions participating in the transaction in some separate > atomic cache. > > Follow this sequence of changes applied to cache contents by each Service: > > Initial cache contents: > [k1 => v1] > [k2 => v2] > [k3 => v3] > > Cache contents after Service A: > [k1 => v1] > [k2 => v2] > [k3 => v3] > [k1x => v1a] > [k2x => v2a] > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate atomic cache > > Cache contents after Service B: > [k1 => v1] > [k2 => v2] > [k3 => v3] > [k1x => v1a] > [k2x => v2ab] > [k3x => v3b] > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate > atomic cache > > Finally the Committer Service takes this map of updated keys and their > versions from some separate atomic cache, starts Ignite transaction and > replaces all the values for k* keys to values taken from k*x keys. The > successful result must be the following: > > [k1 => v1a] > [k2 => v2ab] > [k3 => v3b] > [k1x => v1a] > [k2x => v2ab] > [k3x => v3b] > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate > atomic cache > > But Committer Service also has to check that no one updated the original > values before us, because otherwise we can not give any serializability > guarantee for these distributed transactions. Here we may need to check not > only versions of the updated keys, but also versions of any other keys end > result depends on. > > After that Committer Service has to do a cleanup (may be outside of the > committing tx) to come to the following final state: > > [k1 => v1a] > [k2 => v2ab] > [k3 => v3b] > > Makes sense? > > Sergi > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > - what do u mean by saying " > > *in a single transaction checks value versions for all the old values > > and replaces them with calculated new ones *"? Every time you change > > value(in some service), you store it to *some special atomic cache* , > so > > when all services ceased working, Service commiter got a values with > the > > last versions. > > - After "*does cleanup of temporary keys and values*" Service commiter > > persists them into permanent store, isn't it ? > > - I cant grasp your though, you say "*in case of version mismatch or > TX > > timeout just rollbacks*". But what versions would it match? > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin <[hidden email]>: > > > > > Ok, here is what you actually need to implement at the application > level. > > > > > > Lets say we have to call 2 services in the following order: > > > - Service A: wants to update keys [k1 => v1, k2 => v2] to [k1 => > > v1a, > > > k2 => v2a] > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 => > > v2ab, > > > k3 => v3b] > > > > > > The change > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > must happen in a single transaction. > > > > > > > > > Optimistic protocol to solve this: > > > > > > Each cache key must have a field `otx`, which is a unique orchestrator > TX > > > identifier - it must be a parameter passed to all the services. If > `otx` > > is > > > set to some value it means that it is an intermediate key and is > visible > > > only inside of some transaction, for the finalized key `otx` must be > > null - > > > it means the key is committed and visible for everyone. > > > > > > Each cache value must have a field `ver` which is a version of that > > value. > > > > > > For both fields (`otx` and `ver`) the safest way is to use UUID. > > > > > > Workflow is the following: > > > > > > Orchestrator starts the distributed transaction with `otx` = x and > passes > > > this parameter to all the services. > > > > > > Service A: > > > - does some computations > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > where > > > Za - left time from max Orchestrator TX duration after > Service > > A > > > end > > > k1x, k2x - new temporary keys with field `otx` = x > > > v2a has updated version `ver` > > > - returns a set of updated keys and all the old versions to the > > > orchestrator > > > or just stores it in some special atomic cache like > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > Service B: > > > - retrieves the updated value k2x => v2a because it knows `otx` = x > > > - does computations > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > - updates the set of updated keys like [x => (k1 -> ver1, k2 -> ver2, > k3 > > > -> ver3)] TTL = Zb > > > > > > Service Committer (may be embedded into Orchestrator): > > > - takes all the updated keys and versions for `otx` = x > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > - in a single transaction checks value versions for all the old values > > > and replaces them with calculated new ones > > > - does cleanup of temporary keys and values > > > - in case of version mismatch or TX timeout just rollbacks and signals > > > to Orchestrator to restart the job with new `otx` > > > > > > PROFIT!! > > > > > > This approach even allows you to run independent parts of the graph in > > > parallel (with TX transfer you will always run only one at a time). > Also > > it > > > does not require inventing any special fault tolerance technics because > > > Ignite caches are already fault tolerant and all the intermediate > results > > > are virtually invisible and stored with TTL, thus in case of any crash > > you > > > will not have inconsistent state or garbage. > > > > > > Sergi > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > > > Okay, we are open for proposals on business task. I mean, we can make > > use > > > > of some other thing, not distributed transaction. Not transaction > yet. > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <[hidden email]>: > > > > > > > > > IMO the use case makes sense. However, as Sergi already mentioned, > > the > > > > > problem is far more complex, than simply passing TX state over a > > wire. > > > > Most > > > > > probably a kind of coordinator will be required still to manage all > > > kinds > > > > > of failures. This task should be started with clean design proposal > > > > > explaining how we handle all these concurrent events. And only > then, > > > when > > > > > we understand all implications, we should move to development > stage. > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > [hidden email]> wrote: > > > > > > > > > > > Right > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Good! Basically your orchestrator just takes some predefined > > graph > > > of > > > > > > > distributed services to be invoked, calls them by some kind of > > RPC > > > > and > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible for > managing > > > > > business > > > > > > > > scenarios flows. Many nodes are involved in scenarios. They > > > > exchange > > > > > > data > > > > > > > > and folow one another. If you acquinted with BPMN framework, > so > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from Microsoft > or > > > > your > > > > > > > custom > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which fulfills > > > > custom > > > > > > > logic. > > > > > > > > > > This servers compound oriented graph (BPMN process) which > > > > > > controlled > > > > > > > by > > > > > > > > > > Orchestrator. > > > > > > > > > > For instance, *server1 *creates *variable A *with value > 1, > > > > > > persists > > > > > > > it > > > > > > > > > to > > > > > > > > > > IGNITE cache and creates *variable B *and sends it to* > > > server2. > > > > > > *The > > > > > > > > > > latests receives *variable B*, do some logic with it and > > > stores > > > > > to > > > > > > > > > IGNITE. > > > > > > > > > > All the work made by both servers must be fulfilled in > > *one* > > > > > > > > transaction. > > > > > > > > > > Because we need all information done, or > > nothing(rollbacked). > > > > The > > > > > > > > > scenario > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong > solution > > > for > > > > > it. > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts transaction in > > one > > > > > node, > > > > > > > and > > > > > > > > > > commit > > > > > > > > > > > > this transaction in another jvm node(or rollback it > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some > simplistic > > > > > > scenario, > > > > > > > > get > > > > > > > > > > > ready > > > > > > > > > > > > to > > > > > > > > > > > > > write many fault tolerance tests and make sure that > > you > > > > TXs > > > > > > > work > > > > > > > > > > > > gracefully > > > > > > > > > > > > > in all modes in case of crashes. Also make sure > that > > we > > > > do > > > > > > not > > > > > > > > have > > > > > > > > > > any > > > > > > > > > > > > > performance drops after all your changes in > existing > > > > > > > benchmarks. > > > > > > > > > All > > > > > > > > > > in > > > > > > > > > > > > all > > > > > > > > > > > > > I don't believe these conditions will be met and > your > > > > > > > > contribution > > > > > > > > > > will > > > > > > > > > > > > be > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to > > another > > > > > node? > > > > > > > The > > > > > > > > > > > problem > > > > > > > > > > > > > statement itself is already wrong. What business > case > > > you > > > > > are > > > > > > > > > trying > > > > > > > > > > to > > > > > > > > > > > > > solve? I'm sure everything you need can be done in > a > > > much > > > > > > more > > > > > > > > > simple > > > > > > > > > > > and > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing it > > on > > > > > > another > > > > > > > > node > > > > > > > > > > is > > > > > > > > > > > > > > > meaningless, because other nodes participating > in > > > the > > > > > TX > > > > > > > have > > > > > > > > > to > > > > > > > > > > > know > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > the new coordinator. This will require protocol > > > > > changes, > > > > > > we > > > > > > > > > > > > definitely > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > have fault tolerance and performance issues. > IMO > > > the > > > > > > whole > > > > > > > > idea > > > > > > > > > > is > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation > contains > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > which > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy > Setrakyan > > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are > > > > passing > > > > > > > > > > transaction > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > around. Such object may contain all sorts > of > > > > Ignite > > > > > > > > > context. > > > > > > > > > > If > > > > > > > > > > > > > some > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > needs to be passed across, we should > create a > > > > > special > > > > > > > > > > transfer > > > > > > > > > > > > > object > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY > > > > KUZNETSOV > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing > > > > > > transaction > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > At first, After transaction serialization > > and > > > > > > > > > > deserialization > > > > > > > > > > > > on > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > server, there is no txState. So im going > to > > > put > > > > > it > > > > > > in > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction > > > lacks > > > > of > > > > > > > > shared > > > > > > > > > > > cache > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, > it > > > must > > > > > be > > > > > > > > > injected > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY > > > KUZNETSOV > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing > transaction > > > in > > > > > > > > different > > > > > > > > > > jvms > > > > > > > > > > > > in > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > serialization exception in > > > writeExternalMeta > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > writeExternal(ObjectOutput > > > > > > > out) > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey > > > > Goncharuk < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you > > want, > > > > > but I > > > > > > > > have > > > > > > > > > a > > > > > > > > > > > few > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > - What is the API for the proposed > > change? > > > > In > > > > > > your > > > > > > > > > test, > > > > > > > > > > > you > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > instance of transaction created on > > > ignite(0) > > > > to > > > > > > the > > > > > > > > > > ignite > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > ignite(1). This is obviously not > possible > > > in > > > > a > > > > > > > truly > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > - How will you synchronize cache update > > > > actions > > > > > > and > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > Say, you have one node that decided to > > > > commit, > > > > > > but > > > > > > > > > > another > > > > > > > > > > > > node > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > writing within this transaction. How do > > you > > > > > make > > > > > > > sure > > > > > > > > > > that > > > > > > > > > > > > two > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > not call commit() and rollback() > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > - How do you make sure that either > > > commit() > > > > or > > > > > > > > > > rollback() > > > > > > > > > > > is > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий > Рябов > > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial > > > > > understanding > > > > > > > was > > > > > > > > > > that > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > ownership from one node to another > will > > > be > > > > > > > happened > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY > > > > KUZNETSOV > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on > > > multiple > > > > > > > > threads, > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > every node is able to rollback, or > > > commit > > > > > > > common > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > need to transfer tx between nodes > in > > > > order > > > > > to > > > > > > > > > commit > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a > concept > > > of > > > > > > > > > transferring > > > > > > > > > > > of > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > one node to another? My initial > > > > > > understanding > > > > > > > > was > > > > > > > > > > > that > > > > > > > > > > > > > you > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > to update keys in a transaction > > from > > > > > > multiple > > > > > > > > > > threads > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 > ALEKSEY > > > > > > KUZNETSOV > > > > > > > < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction > > started > > > in > > > > > one > > > > > > > > node, > > > > > > > > > > and > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > The following test describes my > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > transactions = > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> > > cache > > > = > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > transactions.txStart( > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > fut = > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we > > just > > > > > > rebind > > > > > > > > *tx* > > > > > > > > > > to > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter > > > > > *threadMap* > > > > > > > so > > > > > > > > > that > > > > > > > > > > > it > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, > Denis > > > > > Magda < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational > > behind > > > > this > > > > > > and > > > > > > > > the > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, > > > > ALEKSEY > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing > > > distributed > > > > > > > > > transaction > > > > > > > > > > > > which > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > node, and continued at > other > > > one. > > > > > Has > > > > > > > > > anybody > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > *Best Regards,* *Kuznetsov Aleksey* |
|
|
Yeah, now i got it.
There are some doubts on this approach 1) During optimistic commit phase, when you assure no one altered the original values, you must check versions of other dependent keys. How could we obtain those keys(in an automative manner, of course) ? 2) How could we lock a key before some Service A introduce changes? So no other service is allowed to change this key-value?(sort of pessimistic blocking) May be you know some implementations of such approach ? ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV <[hidden email]>: > Thank you very much for help. I will answer later. > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin <[hidden email]>: > > All the services do not update key in place, but only generate new keys > augmented by otx and store the updated value in the same cache + remember > the keys and versions participating in the transaction in some separate > atomic cache. > > Follow this sequence of changes applied to cache contents by each Service: > > Initial cache contents: > [k1 => v1] > [k2 => v2] > [k3 => v3] > > Cache contents after Service A: > [k1 => v1] > [k2 => v2] > [k3 => v3] > [k1x => v1a] > [k2x => v2a] > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate atomic cache > > Cache contents after Service B: > [k1 => v1] > [k2 => v2] > [k3 => v3] > [k1x => v1a] > [k2x => v2ab] > [k3x => v3b] > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate > atomic cache > > Finally the Committer Service takes this map of updated keys and their > versions from some separate atomic cache, starts Ignite transaction and > replaces all the values for k* keys to values taken from k*x keys. The > successful result must be the following: > > [k1 => v1a] > [k2 => v2ab] > [k3 => v3b] > [k1x => v1a] > [k2x => v2ab] > [k3x => v3b] > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate > atomic cache > > But Committer Service also has to check that no one updated the original > values before us, because otherwise we can not give any serializability > guarantee for these distributed transactions. Here we may need to check not > only versions of the updated keys, but also versions of any other keys end > result depends on. > > After that Committer Service has to do a cleanup (may be outside of the > committing tx) to come to the following final state: > > [k1 => v1a] > [k2 => v2ab] > [k3 => v3b] > > Makes sense? > > Sergi > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > - what do u mean by saying " > > *in a single transaction checks value versions for all the old values > > and replaces them with calculated new ones *"? Every time you change > > value(in some service), you store it to *some special atomic cache* , > so > > when all services ceased working, Service commiter got a values with > the > > last versions. > > - After "*does cleanup of temporary keys and values*" Service commiter > > persists them into permanent store, isn't it ? > > - I cant grasp your though, you say "*in case of version mismatch or > TX > > timeout just rollbacks*". But what versions would it match? > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin <[hidden email]>: > > > > > Ok, here is what you actually need to implement at the application > level. > > > > > > Lets say we have to call 2 services in the following order: > > > - Service A: wants to update keys [k1 => v1, k2 => v2] to [k1 => > > v1a, > > > k2 => v2a] > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 => > > v2ab, > > > k3 => v3b] > > > > > > The change > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > must happen in a single transaction. > > > > > > > > > Optimistic protocol to solve this: > > > > > > Each cache key must have a field `otx`, which is a unique orchestrator > TX > > > identifier - it must be a parameter passed to all the services. If > `otx` > > is > > > set to some value it means that it is an intermediate key and is > visible > > > only inside of some transaction, for the finalized key `otx` must be > > null - > > > it means the key is committed and visible for everyone. > > > > > > Each cache value must have a field `ver` which is a version of that > > value. > > > > > > For both fields (`otx` and `ver`) the safest way is to use UUID. > > > > > > Workflow is the following: > > > > > > Orchestrator starts the distributed transaction with `otx` = x and > passes > > > this parameter to all the services. > > > > > > Service A: > > > - does some computations > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > where > > > Za - left time from max Orchestrator TX duration after > Service > > A > > > end > > > k1x, k2x - new temporary keys with field `otx` = x > > > v2a has updated version `ver` > > > - returns a set of updated keys and all the old versions to the > > > orchestrator > > > or just stores it in some special atomic cache like > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > Service B: > > > - retrieves the updated value k2x => v2a because it knows `otx` = x > > > - does computations > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > - updates the set of updated keys like [x => (k1 -> ver1, k2 -> ver2, > k3 > > > -> ver3)] TTL = Zb > > > > > > Service Committer (may be embedded into Orchestrator): > > > - takes all the updated keys and versions for `otx` = x > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > - in a single transaction checks value versions for all the old values > > > and replaces them with calculated new ones > > > - does cleanup of temporary keys and values > > > - in case of version mismatch or TX timeout just rollbacks and signals > > > to Orchestrator to restart the job with new `otx` > > > > > > PROFIT!! > > > > > > This approach even allows you to run independent parts of the graph in > > > parallel (with TX transfer you will always run only one at a time). > Also > > it > > > does not require inventing any special fault tolerance technics because > > > Ignite caches are already fault tolerant and all the intermediate > results > > > are virtually invisible and stored with TTL, thus in case of any crash > > you > > > will not have inconsistent state or garbage. > > > > > > Sergi > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > > > Okay, we are open for proposals on business task. I mean, we can make > > use > > > > of some other thing, not distributed transaction. Not transaction > yet. > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <[hidden email]>: > > > > > > > > > IMO the use case makes sense. However, as Sergi already mentioned, > > the > > > > > problem is far more complex, than simply passing TX state over a > > wire. > > > > Most > > > > > probably a kind of coordinator will be required still to manage all > > > kinds > > > > > of failures. This task should be started with clean design proposal > > > > > explaining how we handle all these concurrent events. And only > then, > > > when > > > > > we understand all implications, we should move to development > stage. > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > [hidden email]> wrote: > > > > > > > > > > > Right > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Good! Basically your orchestrator just takes some predefined > > graph > > > of > > > > > > > distributed services to be invoked, calls them by some kind of > > RPC > > > > and > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible for > managing > > > > > business > > > > > > > > scenarios flows. Many nodes are involved in scenarios. They > > > > exchange > > > > > > data > > > > > > > > and folow one another. If you acquinted with BPMN framework, > so > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from Microsoft > or > > > > your > > > > > > > custom > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which fulfills > > > > custom > > > > > > > logic. > > > > > > > > > > This servers compound oriented graph (BPMN process) which > > > > > > controlled > > > > > > > by > > > > > > > > > > Orchestrator. > > > > > > > > > > For instance, *server1 *creates *variable A *with value > 1, > > > > > > persists > > > > > > > it > > > > > > > > > to > > > > > > > > > > IGNITE cache and creates *variable B *and sends it to* > > > server2. > > > > > > *The > > > > > > > > > > latests receives *variable B*, do some logic with it and > > > stores > > > > > to > > > > > > > > > IGNITE. > > > > > > > > > > All the work made by both servers must be fulfilled in > > *one* > > > > > > > > transaction. > > > > > > > > > > Because we need all information done, or > > nothing(rollbacked). > > > > The > > > > > > > > > scenario > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong > solution > > > for > > > > > it. > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts transaction in > > one > > > > > node, > > > > > > > and > > > > > > > > > > commit > > > > > > > > > > > > this transaction in another jvm node(or rollback it > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some > simplistic > > > > > > scenario, > > > > > > > > get > > > > > > > > > > > ready > > > > > > > > > > > > to > > > > > > > > > > > > > write many fault tolerance tests and make sure that > > you > > > > TXs > > > > > > > work > > > > > > > > > > > > gracefully > > > > > > > > > > > > > in all modes in case of crashes. Also make sure > that > > we > > > > do > > > > > > not > > > > > > > > have > > > > > > > > > > any > > > > > > > > > > > > > performance drops after all your changes in > existing > > > > > > > benchmarks. > > > > > > > > > All > > > > > > > > > > in > > > > > > > > > > > > all > > > > > > > > > > > > > I don't believe these conditions will be met and > your > > > > > > > > contribution > > > > > > > > > > will > > > > > > > > > > > > be > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to > > another > > > > > node? > > > > > > > The > > > > > > > > > > > problem > > > > > > > > > > > > > statement itself is already wrong. What business > case > > > you > > > > > are > > > > > > > > > trying > > > > > > > > > > to > > > > > > > > > > > > > solve? I'm sure everything you need can be done in > a > > > much > > > > > > more > > > > > > > > > simple > > > > > > > > > > > and > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing it > > on > > > > > > another > > > > > > > > node > > > > > > > > > > is > > > > > > > > > > > > > > > meaningless, because other nodes participating > in > > > the > > > > > TX > > > > > > > have > > > > > > > > > to > > > > > > > > > > > know > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > the new coordinator. This will require protocol > > > > > changes, > > > > > > we > > > > > > > > > > > > definitely > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > have fault tolerance and performance issues. > IMO > > > the > > > > > > whole > > > > > > > > idea > > > > > > > > > > is > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation > contains > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > which > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy > Setrakyan > > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we are > > > > passing > > > > > > > > > > transaction > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > around. Such object may contain all sorts > of > > > > Ignite > > > > > > > > > context. > > > > > > > > > > If > > > > > > > > > > > > > some > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > needs to be passed across, we should > create a > > > > > special > > > > > > > > > > transfer > > > > > > > > > > > > > object > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY > > > > KUZNETSOV > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues preventing > > > > > > transaction > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > At first, After transaction serialization > > and > > > > > > > > > > deserialization > > > > > > > > > > > > on > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > server, there is no txState. So im going > to > > > put > > > > > it > > > > > > in > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized transaction > > > lacks > > > > of > > > > > > > > shared > > > > > > > > > > > cache > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, > it > > > must > > > > > be > > > > > > > > > injected > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY > > > KUZNETSOV > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing > transaction > > > in > > > > > > > > different > > > > > > > > > > jvms > > > > > > > > > > > > in > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > serialization exception in > > > writeExternalMeta > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > writeExternal(ObjectOutput > > > > > > > out) > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey > > > > Goncharuk < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you > > want, > > > > > but I > > > > > > > > have > > > > > > > > > a > > > > > > > > > > > few > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > - What is the API for the proposed > > change? > > > > In > > > > > > your > > > > > > > > > test, > > > > > > > > > > > you > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > instance of transaction created on > > > ignite(0) > > > > to > > > > > > the > > > > > > > > > > ignite > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > ignite(1). This is obviously not > possible > > > in > > > > a > > > > > > > truly > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > - How will you synchronize cache update > > > > actions > > > > > > and > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > Say, you have one node that decided to > > > > commit, > > > > > > but > > > > > > > > > > another > > > > > > > > > > > > node > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > writing within this transaction. How do > > you > > > > > make > > > > > > > sure > > > > > > > > > > that > > > > > > > > > > > > two > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > not call commit() and rollback() > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > - How do you make sure that either > > > commit() > > > > or > > > > > > > > > > rollback() > > > > > > > > > > > is > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий > Рябов > > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial > > > > > understanding > > > > > > > was > > > > > > > > > > that > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > ownership from one node to another > will > > > be > > > > > > > happened > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY > > > > KUZNETSOV > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on > > > multiple > > > > > > > > threads, > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > every node is able to rollback, or > > > commit > > > > > > > common > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > need to transfer tx between nodes > in > > > > order > > > > > to > > > > > > > > > commit > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, Alexey > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a > concept > > > of > > > > > > > > > transferring > > > > > > > > > > > of > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > one node to another? My initial > > > > > > understanding > > > > > > > > was > > > > > > > > > > > that > > > > > > > > > > > > > you > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > to update keys in a transaction > > from > > > > > > multiple > > > > > > > > > > threads > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 > ALEKSEY > > > > > > KUZNETSOV > > > > > > > < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction > > started > > > in > > > > > one > > > > > > > > node, > > > > > > > > > > and > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > The following test describes my > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > transactions = > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> > > cache > > > = > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > transactions.txStart( > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > fut = > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* we > > just > > > > > > rebind > > > > > > > > *tx* > > > > > > > > > > to > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void txStart(Transaction > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter > > > > > *threadMap* > > > > > > > so > > > > > > > > > that > > > > > > > > > > > it > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, > Denis > > > > > Magda < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational > > behind > > > > this > > > > > > and > > > > > > > > the > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 AM, > > > > ALEKSEY > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing > > > distributed > > > > > > > > > transaction > > > > > > > > > > > > which > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > node, and continued at > other > > > one. > > > > > Has > > > > > > > > > anybody > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > *Best Regards,* *Kuznetsov Aleksey* |
|
|
1. Actually you have to check versions on all the values you have read
during the tx. For example if we have [k1 => v1, k2 => v2] and do: put(k1, get(k2) + 5) We have to remember the version for k2. This logic can be relatively easily encapsulated in a framework atop of Ignite. You need to implement one to make all this stuff usable. 2. I suggest to avoid any locking here, because you easily will end up with deadlocks. If you do not have too frequent updates for your keys, optimistic approach will work just fine. Theoretically in the Committer Service you can start a thread for the lifetime of the whole distributed transaction, take a lock on the key using IgniteCache.lock(K key) before executing any Services, wait for all the services to complete, execute optimistic commit in the same thread while keeping this lock and then release it. Notice that all the Ignite transactions inside of all Services must be optimistic here to be able to read this locked key. But again I do not recommend you using this approach until you have a reliable deadlock avoidance scheme. Sergi 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > Yeah, now i got it. > There are some doubts on this approach > 1) During optimistic commit phase, when you assure no one altered the > original values, you must check versions of other dependent keys. How could > we obtain those keys(in an automative manner, of course) ? > 2) How could we lock a key before some Service A introduce changes? So no > other service is allowed to change this key-value?(sort of pessimistic > blocking) > May be you know some implementations of such approach ? > > ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV <[hidden email]>: > > > Thank you very much for help. I will answer later. > > > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin <[hidden email]>: > > > > All the services do not update key in place, but only generate new keys > > augmented by otx and store the updated value in the same cache + remember > > the keys and versions participating in the transaction in some separate > > atomic cache. > > > > Follow this sequence of changes applied to cache contents by each > Service: > > > > Initial cache contents: > > [k1 => v1] > > [k2 => v2] > > [k3 => v3] > > > > Cache contents after Service A: > > [k1 => v1] > > [k2 => v2] > > [k3 => v3] > > [k1x => v1a] > > [k2x => v2a] > > > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate atomic cache > > > > Cache contents after Service B: > > [k1 => v1] > > [k2 => v2] > > [k3 => v3] > > [k1x => v1a] > > [k2x => v2ab] > > [k3x => v3b] > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate > > atomic cache > > > > Finally the Committer Service takes this map of updated keys and their > > versions from some separate atomic cache, starts Ignite transaction and > > replaces all the values for k* keys to values taken from k*x keys. The > > successful result must be the following: > > > > [k1 => v1a] > > [k2 => v2ab] > > [k3 => v3b] > > [k1x => v1a] > > [k2x => v2ab] > > [k3x => v3b] > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate > > atomic cache > > > > But Committer Service also has to check that no one updated the original > > values before us, because otherwise we can not give any serializability > > guarantee for these distributed transactions. Here we may need to check > not > > only versions of the updated keys, but also versions of any other keys > end > > result depends on. > > > > After that Committer Service has to do a cleanup (may be outside of the > > committing tx) to come to the following final state: > > > > [k1 => v1a] > > [k2 => v2ab] > > [k3 => v3b] > > > > Makes sense? > > > > Sergi > > > > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > > > - what do u mean by saying " > > > *in a single transaction checks value versions for all the old values > > > and replaces them with calculated new ones *"? Every time you > change > > > value(in some service), you store it to *some special atomic cache* > , > > so > > > when all services ceased working, Service commiter got a values with > > the > > > last versions. > > > - After "*does cleanup of temporary keys and values*" Service > commiter > > > persists them into permanent store, isn't it ? > > > - I cant grasp your though, you say "*in case of version mismatch or > > TX > > > timeout just rollbacks*". But what versions would it match? > > > > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin <[hidden email] > >: > > > > > > > Ok, here is what you actually need to implement at the application > > level. > > > > > > > > Lets say we have to call 2 services in the following order: > > > > - Service A: wants to update keys [k1 => v1, k2 => v2] to [k1 => > > > v1a, > > > > k2 => v2a] > > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 => > > > v2ab, > > > > k3 => v3b] > > > > > > > > The change > > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > > must happen in a single transaction. > > > > > > > > > > > > Optimistic protocol to solve this: > > > > > > > > Each cache key must have a field `otx`, which is a unique > orchestrator > > TX > > > > identifier - it must be a parameter passed to all the services. If > > `otx` > > > is > > > > set to some value it means that it is an intermediate key and is > > visible > > > > only inside of some transaction, for the finalized key `otx` must be > > > null - > > > > it means the key is committed and visible for everyone. > > > > > > > > Each cache value must have a field `ver` which is a version of that > > > value. > > > > > > > > For both fields (`otx` and `ver`) the safest way is to use UUID. > > > > > > > > Workflow is the following: > > > > > > > > Orchestrator starts the distributed transaction with `otx` = x and > > passes > > > > this parameter to all the services. > > > > > > > > Service A: > > > > - does some computations > > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > > where > > > > Za - left time from max Orchestrator TX duration after > > Service > > > A > > > > end > > > > k1x, k2x - new temporary keys with field `otx` = x > > > > v2a has updated version `ver` > > > > - returns a set of updated keys and all the old versions to the > > > > orchestrator > > > > or just stores it in some special atomic cache like > > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > > > Service B: > > > > - retrieves the updated value k2x => v2a because it knows `otx` = x > > > > - does computations > > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > > - updates the set of updated keys like [x => (k1 -> ver1, k2 -> > ver2, > > k3 > > > > -> ver3)] TTL = Zb > > > > > > > > Service Committer (may be embedded into Orchestrator): > > > > - takes all the updated keys and versions for `otx` = x > > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > > - in a single transaction checks value versions for all the old > values > > > > and replaces them with calculated new ones > > > > - does cleanup of temporary keys and values > > > > - in case of version mismatch or TX timeout just rollbacks and > signals > > > > to Orchestrator to restart the job with new `otx` > > > > > > > > PROFIT!! > > > > > > > > This approach even allows you to run independent parts of the graph > in > > > > parallel (with TX transfer you will always run only one at a time). > > Also > > > it > > > > does not require inventing any special fault tolerance technics > because > > > > Ignite caches are already fault tolerant and all the intermediate > > results > > > > are virtually invisible and stored with TTL, thus in case of any > crash > > > you > > > > will not have inconsistent state or garbage. > > > > > > > > Sergi > > > > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > > >: > > > > > > > > > Okay, we are open for proposals on business task. I mean, we can > make > > > use > > > > > of some other thing, not distributed transaction. Not transaction > > yet. > > > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <[hidden email] > >: > > > > > > > > > > > IMO the use case makes sense. However, as Sergi already > mentioned, > > > the > > > > > > problem is far more complex, than simply passing TX state over a > > > wire. > > > > > Most > > > > > > probably a kind of coordinator will be required still to manage > all > > > > kinds > > > > > > of failures. This task should be started with clean design > proposal > > > > > > explaining how we handle all these concurrent events. And only > > then, > > > > when > > > > > > we understand all implications, we should move to development > > stage. > > > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > Right > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > Good! Basically your orchestrator just takes some predefined > > > graph > > > > of > > > > > > > > distributed services to be invoked, calls them by some kind > of > > > RPC > > > > > and > > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible for > > managing > > > > > > business > > > > > > > > > scenarios flows. Many nodes are involved in scenarios. They > > > > > exchange > > > > > > > data > > > > > > > > > and folow one another. If you acquinted with BPMN > framework, > > so > > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from > Microsoft > > or > > > > > your > > > > > > > > custom > > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which > fulfills > > > > > custom > > > > > > > > logic. > > > > > > > > > > > This servers compound oriented graph (BPMN process) > which > > > > > > > controlled > > > > > > > > by > > > > > > > > > > > Orchestrator. > > > > > > > > > > > For instance, *server1 *creates *variable A *with > value > > 1, > > > > > > > persists > > > > > > > > it > > > > > > > > > > to > > > > > > > > > > > IGNITE cache and creates *variable B *and sends it to* > > > > server2. > > > > > > > *The > > > > > > > > > > > latests receives *variable B*, do some logic with it > and > > > > stores > > > > > > to > > > > > > > > > > IGNITE. > > > > > > > > > > > All the work made by both servers must be fulfilled in > > > *one* > > > > > > > > > transaction. > > > > > > > > > > > Because we need all information done, or > > > nothing(rollbacked). > > > > > The > > > > > > > > > > scenario > > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong > > solution > > > > for > > > > > > it. > > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts transaction > in > > > one > > > > > > node, > > > > > > > > and > > > > > > > > > > > commit > > > > > > > > > > > > > this transaction in another jvm node(or rollback it > > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some > > simplistic > > > > > > > scenario, > > > > > > > > > get > > > > > > > > > > > > ready > > > > > > > > > > > > > to > > > > > > > > > > > > > > write many fault tolerance tests and make sure > that > > > you > > > > > TXs > > > > > > > > work > > > > > > > > > > > > > gracefully > > > > > > > > > > > > > > in all modes in case of crashes. Also make sure > > that > > > we > > > > > do > > > > > > > not > > > > > > > > > have > > > > > > > > > > > any > > > > > > > > > > > > > > performance drops after all your changes in > > existing > > > > > > > > benchmarks. > > > > > > > > > > All > > > > > > > > > > > in > > > > > > > > > > > > > all > > > > > > > > > > > > > > I don't believe these conditions will be met and > > your > > > > > > > > > contribution > > > > > > > > > > > will > > > > > > > > > > > > > be > > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to > > > another > > > > > > node? > > > > > > > > The > > > > > > > > > > > > problem > > > > > > > > > > > > > > statement itself is already wrong. What business > > case > > > > you > > > > > > are > > > > > > > > > > trying > > > > > > > > > > > to > > > > > > > > > > > > > > solve? I'm sure everything you need can be done > in > > a > > > > much > > > > > > > more > > > > > > > > > > simple > > > > > > > > > > > > and > > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and deserializing > it > > > on > > > > > > > another > > > > > > > > > node > > > > > > > > > > > is > > > > > > > > > > > > > > > > meaningless, because other nodes > participating > > in > > > > the > > > > > > TX > > > > > > > > have > > > > > > > > > > to > > > > > > > > > > > > know > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > the new coordinator. This will require > protocol > > > > > > changes, > > > > > > > we > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > have fault tolerance and performance issues. > > IMO > > > > the > > > > > > > whole > > > > > > > > > idea > > > > > > > > > > > is > > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation > > contains > > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > > which > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy > > Setrakyan > > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we > are > > > > > passing > > > > > > > > > > > transaction > > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > around. Such object may contain all sorts > > of > > > > > Ignite > > > > > > > > > > context. > > > > > > > > > > > If > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > needs to be passed across, we should > > create a > > > > > > special > > > > > > > > > > > transfer > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY > > > > > KUZNETSOV > > > > > > < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues > preventing > > > > > > > transaction > > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > > At first, After transaction > serialization > > > and > > > > > > > > > > > deserialization > > > > > > > > > > > > > on > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > > server, there is no txState. So im > going > > to > > > > put > > > > > > it > > > > > > > in > > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized > transaction > > > > lacks > > > > > of > > > > > > > > > shared > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > > field at TransactionProxyImpl. Perhaps, > > it > > > > must > > > > > > be > > > > > > > > > > injected > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY > > > > KUZNETSOV > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing > > transaction > > > > in > > > > > > > > > different > > > > > > > > > > > jvms > > > > > > > > > > > > > in > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > serialization exception in > > > > writeExternalMeta > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > > writeExternal(ObjectOutput > > > > > > > > out) > > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what you > > > want, > > > > > > but I > > > > > > > > > have > > > > > > > > > > a > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > > - What is the API for the proposed > > > change? > > > > > In > > > > > > > your > > > > > > > > > > test, > > > > > > > > > > > > you > > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > instance of transaction created on > > > > ignite(0) > > > > > to > > > > > > > the > > > > > > > > > > > ignite > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > ignite(1). This is obviously not > > possible > > > > in > > > > > a > > > > > > > > truly > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > > - How will you synchronize cache > update > > > > > actions > > > > > > > and > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > > Say, you have one node that decided > to > > > > > commit, > > > > > > > but > > > > > > > > > > > another > > > > > > > > > > > > > node > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > writing within this transaction. How > do > > > you > > > > > > make > > > > > > > > sure > > > > > > > > > > > that > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > not call commit() and rollback() > > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > > - How do you make sure that either > > > > commit() > > > > > or > > > > > > > > > > > rollback() > > > > > > > > > > > > is > > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий > > Рябов > > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial > > > > > > understanding > > > > > > > > was > > > > > > > > > > > that > > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > ownership from one node to another > > will > > > > be > > > > > > > > happened > > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 ALEKSEY > > > > > KUZNETSOV > > > > > > < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction on > > > > multiple > > > > > > > > > threads, > > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > every node is able to rollback, > or > > > > commit > > > > > > > > common > > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > > need to transfer tx between nodes > > in > > > > > order > > > > > > to > > > > > > > > > > commit > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > different node(in the same jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, > Alexey > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a > > concept > > > > of > > > > > > > > > > transferring > > > > > > > > > > > > of > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > one node to another? My initial > > > > > > > understanding > > > > > > > > > was > > > > > > > > > > > > that > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > > to update keys in a transaction > > > from > > > > > > > multiple > > > > > > > > > > > threads > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 > > ALEKSEY > > > > > > > KUZNETSOV > > > > > > > > < > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction > > > started > > > > in > > > > > > one > > > > > > > > > node, > > > > > > > > > > > and > > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > > The following test describes > my > > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > transactions = > > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, Integer> > > > cache > > > > = > > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > > transactions.txStart( > > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > > fut = > > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* > we > > > just > > > > > > > rebind > > > > > > > > > *tx* > > > > > > > > > > > to > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void > txStart(Transaction > > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we alter > > > > > > *threadMap* > > > > > > > > so > > > > > > > > > > that > > > > > > > > > > > > it > > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, > > Denis > > > > > > Magda < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational > > > behind > > > > > this > > > > > > > and > > > > > > > > > the > > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 > AM, > > > > > ALEKSEY > > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing > > > > distributed > > > > > > > > > > transaction > > > > > > > > > > > > > which > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > > node, and continued at > > other > > > > one. > > > > > > Has > > > > > > > > > > anybody > > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Best Regards,* > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > -- > > > > > > *Best Regards,* > > > > > > *Kuznetsov Aleksey* > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > |
|
|
hmm, If we have millions of entities, and millions of operations, would not
this approache lead to memory overflow and perfomance degradation чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin <[hidden email]>: > 1. Actually you have to check versions on all the values you have read > during the tx. > > For example if we have [k1 => v1, k2 => v2] and do: > > put(k1, get(k2) + 5) > > We have to remember the version for k2. This logic can be relatively easily > encapsulated in a framework atop of Ignite. You need to implement one to > make all this stuff usable. > > 2. I suggest to avoid any locking here, because you easily will end up with > deadlocks. If you do not have too frequent updates for your keys, > optimistic approach will work just fine. > > Theoretically in the Committer Service you can start a thread for the > lifetime of the whole distributed transaction, take a lock on the key using > IgniteCache.lock(K key) before executing any Services, wait for all the > services to complete, execute optimistic commit in the same thread while > keeping this lock and then release it. Notice that all the Ignite > transactions inside of all Services must be optimistic here to be able to > read this locked key. > > But again I do not recommend you using this approach until you have a > reliable deadlock avoidance scheme. > > Sergi > > > > > > > > 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > Yeah, now i got it. > > There are some doubts on this approach > > 1) During optimistic commit phase, when you assure no one altered the > > original values, you must check versions of other dependent keys. How > could > > we obtain those keys(in an automative manner, of course) ? > > 2) How could we lock a key before some Service A introduce changes? So no > > other service is allowed to change this key-value?(sort of pessimistic > > blocking) > > May be you know some implementations of such approach ? > > > > ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > Thank you very much for help. I will answer later. > > > > > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin <[hidden email] > >: > > > > > > All the services do not update key in place, but only generate new keys > > > augmented by otx and store the updated value in the same cache + > remember > > > the keys and versions participating in the transaction in some separate > > > atomic cache. > > > > > > Follow this sequence of changes applied to cache contents by each > > Service: > > > > > > Initial cache contents: > > > [k1 => v1] > > > [k2 => v2] > > > [k3 => v3] > > > > > > Cache contents after Service A: > > > [k1 => v1] > > > [k2 => v2] > > > [k3 => v3] > > > [k1x => v1a] > > > [k2x => v2a] > > > > > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate atomic > cache > > > > > > Cache contents after Service B: > > > [k1 => v1] > > > [k2 => v2] > > > [k3 => v3] > > > [k1x => v1a] > > > [k2x => v2ab] > > > [k3x => v3b] > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate > > > atomic cache > > > > > > Finally the Committer Service takes this map of updated keys and their > > > versions from some separate atomic cache, starts Ignite transaction and > > > replaces all the values for k* keys to values taken from k*x keys. The > > > successful result must be the following: > > > > > > [k1 => v1a] > > > [k2 => v2ab] > > > [k3 => v3b] > > > [k1x => v1a] > > > [k2x => v2ab] > > > [k3x => v3b] > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some separate > > > atomic cache > > > > > > But Committer Service also has to check that no one updated the > original > > > values before us, because otherwise we can not give any serializability > > > guarantee for these distributed transactions. Here we may need to check > > not > > > only versions of the updated keys, but also versions of any other keys > > end > > > result depends on. > > > > > > After that Committer Service has to do a cleanup (may be outside of the > > > committing tx) to come to the following final state: > > > > > > [k1 => v1a] > > > [k2 => v2ab] > > > [k3 => v3b] > > > > > > Makes sense? > > > > > > Sergi > > > > > > > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > > > - what do u mean by saying " > > > > *in a single transaction checks value versions for all the old values > > > > and replaces them with calculated new ones *"? Every time you > > change > > > > value(in some service), you store it to *some special atomic > cache* > > , > > > so > > > > when all services ceased working, Service commiter got a values > with > > > the > > > > last versions. > > > > - After "*does cleanup of temporary keys and values*" Service > > commiter > > > > persists them into permanent store, isn't it ? > > > > - I cant grasp your though, you say "*in case of version mismatch > or > > > TX > > > > timeout just rollbacks*". But what versions would it match? > > > > > > > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > [hidden email] > > >: > > > > > > > > > Ok, here is what you actually need to implement at the application > > > level. > > > > > > > > > > Lets say we have to call 2 services in the following order: > > > > > - Service A: wants to update keys [k1 => v1, k2 => v2] to [k1 > => > > > > v1a, > > > > > k2 => v2a] > > > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 > => > > > > v2ab, > > > > > k3 => v3b] > > > > > > > > > > The change > > > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > > > must happen in a single transaction. > > > > > > > > > > > > > > > Optimistic protocol to solve this: > > > > > > > > > > Each cache key must have a field `otx`, which is a unique > > orchestrator > > > TX > > > > > identifier - it must be a parameter passed to all the services. If > > > `otx` > > > > is > > > > > set to some value it means that it is an intermediate key and is > > > visible > > > > > only inside of some transaction, for the finalized key `otx` must > be > > > > null - > > > > > it means the key is committed and visible for everyone. > > > > > > > > > > Each cache value must have a field `ver` which is a version of that > > > > value. > > > > > > > > > > For both fields (`otx` and `ver`) the safest way is to use UUID. > > > > > > > > > > Workflow is the following: > > > > > > > > > > Orchestrator starts the distributed transaction with `otx` = x and > > > passes > > > > > this parameter to all the services. > > > > > > > > > > Service A: > > > > > - does some computations > > > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > > > where > > > > > Za - left time from max Orchestrator TX duration after > > > Service > > > > A > > > > > end > > > > > k1x, k2x - new temporary keys with field `otx` = x > > > > > v2a has updated version `ver` > > > > > - returns a set of updated keys and all the old versions to the > > > > > orchestrator > > > > > or just stores it in some special atomic cache like > > > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > > > > > Service B: > > > > > - retrieves the updated value k2x => v2a because it knows `otx` = > x > > > > > - does computations > > > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > > > - updates the set of updated keys like [x => (k1 -> ver1, k2 -> > > ver2, > > > k3 > > > > > -> ver3)] TTL = Zb > > > > > > > > > > Service Committer (may be embedded into Orchestrator): > > > > > - takes all the updated keys and versions for `otx` = x > > > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > > > - in a single transaction checks value versions for all the old > > values > > > > > and replaces them with calculated new ones > > > > > - does cleanup of temporary keys and values > > > > > - in case of version mismatch or TX timeout just rollbacks and > > signals > > > > > to Orchestrator to restart the job with new `otx` > > > > > > > > > > PROFIT!! > > > > > > > > > > This approach even allows you to run independent parts of the graph > > in > > > > > parallel (with TX transfer you will always run only one at a time). > > > Also > > > > it > > > > > does not require inventing any special fault tolerance technics > > because > > > > > Ignite caches are already fault tolerant and all the intermediate > > > results > > > > > are virtually invisible and stored with TTL, thus in case of any > > crash > > > > you > > > > > will not have inconsistent state or garbage. > > > > > > > > > > Sergi > > > > > > > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > > [hidden email] > > > >: > > > > > > > > > > > Okay, we are open for proposals on business task. I mean, we can > > make > > > > use > > > > > > of some other thing, not distributed transaction. Not transaction > > > yet. > > > > > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > [hidden email] > > >: > > > > > > > > > > > > > IMO the use case makes sense. However, as Sergi already > > mentioned, > > > > the > > > > > > > problem is far more complex, than simply passing TX state over > a > > > > wire. > > > > > > Most > > > > > > > probably a kind of coordinator will be required still to manage > > all > > > > > kinds > > > > > > > of failures. This task should be started with clean design > > proposal > > > > > > > explaining how we handle all these concurrent events. And only > > > then, > > > > > when > > > > > > > we understand all implications, we should move to development > > > stage. > > > > > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > Right > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > Good! Basically your orchestrator just takes some > predefined > > > > graph > > > > > of > > > > > > > > > distributed services to be invoked, calls them by some kind > > of > > > > RPC > > > > > > and > > > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible for > > > managing > > > > > > > business > > > > > > > > > > scenarios flows. Many nodes are involved in scenarios. > They > > > > > > exchange > > > > > > > > data > > > > > > > > > > and folow one another. If you acquinted with BPMN > > framework, > > > so > > > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from > > Microsoft > > > or > > > > > > your > > > > > > > > > custom > > > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which > > fulfills > > > > > > custom > > > > > > > > > logic. > > > > > > > > > > > > This servers compound oriented graph (BPMN process) > > which > > > > > > > > controlled > > > > > > > > > by > > > > > > > > > > > > Orchestrator. > > > > > > > > > > > > For instance, *server1 *creates *variable A *with > > value > > > 1, > > > > > > > > persists > > > > > > > > > it > > > > > > > > > > > to > > > > > > > > > > > > IGNITE cache and creates *variable B *and sends it > to* > > > > > server2. > > > > > > > > *The > > > > > > > > > > > > latests receives *variable B*, do some logic with it > > and > > > > > stores > > > > > > > to > > > > > > > > > > > IGNITE. > > > > > > > > > > > > All the work made by both servers must be fulfilled > in > > > > *one* > > > > > > > > > > transaction. > > > > > > > > > > > > Because we need all information done, or > > > > nothing(rollbacked). > > > > > > The > > > > > > > > > > > scenario > > > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong > > > solution > > > > > for > > > > > > > it. > > > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts transaction > > in > > > > one > > > > > > > node, > > > > > > > > > and > > > > > > > > > > > > commit > > > > > > > > > > > > > > this transaction in another jvm node(or rollback > it > > > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some > > > simplistic > > > > > > > > scenario, > > > > > > > > > > get > > > > > > > > > > > > > ready > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > write many fault tolerance tests and make sure > > that > > > > you > > > > > > TXs > > > > > > > > > work > > > > > > > > > > > > > > gracefully > > > > > > > > > > > > > > > in all modes in case of crashes. Also make sure > > > that > > > > we > > > > > > do > > > > > > > > not > > > > > > > > > > have > > > > > > > > > > > > any > > > > > > > > > > > > > > > performance drops after all your changes in > > > existing > > > > > > > > > benchmarks. > > > > > > > > > > > All > > > > > > > > > > > > in > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > I don't believe these conditions will be met > and > > > your > > > > > > > > > > contribution > > > > > > > > > > > > will > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX to > > > > another > > > > > > > node? > > > > > > > > > The > > > > > > > > > > > > > problem > > > > > > > > > > > > > > > statement itself is already wrong. What > business > > > case > > > > > you > > > > > > > are > > > > > > > > > > > trying > > > > > > > > > > > > to > > > > > > > > > > > > > > > solve? I'm sure everything you need can be done > > in > > > a > > > > > much > > > > > > > > more > > > > > > > > > > > simple > > > > > > > > > > > > > and > > > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and > deserializing > > it > > > > on > > > > > > > > another > > > > > > > > > > node > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > meaningless, because other nodes > > participating > > > in > > > > > the > > > > > > > TX > > > > > > > > > have > > > > > > > > > > > to > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > the new coordinator. This will require > > protocol > > > > > > > changes, > > > > > > > > we > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > have fault tolerance and performance > issues. > > > IMO > > > > > the > > > > > > > > whole > > > > > > > > > > idea > > > > > > > > > > > > is > > > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > > > and it makes no sense to waste time on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY > KUZNETSOV > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation > > > contains > > > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy > > > Setrakyan > > > > < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that we > > are > > > > > > passing > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > > around. Such object may contain all > sorts > > > of > > > > > > Ignite > > > > > > > > > > > context. > > > > > > > > > > > > If > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > needs to be passed across, we should > > > create a > > > > > > > special > > > > > > > > > > > > transfer > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, > ALEKSEY > > > > > > KUZNETSOV > > > > > > > < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues > > preventing > > > > > > > > transaction > > > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > > > At first, After transaction > > serialization > > > > and > > > > > > > > > > > > deserialization > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > > > server, there is no txState. So im > > going > > > to > > > > > put > > > > > > > it > > > > > > > > in > > > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized > > transaction > > > > > lacks > > > > > > of > > > > > > > > > > shared > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > > > field at TransactionProxyImpl. > Perhaps, > > > it > > > > > must > > > > > > > be > > > > > > > > > > > injected > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, ALEKSEY > > > > > KUZNETSOV > > > > > > < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing > > > transaction > > > > > in > > > > > > > > > > different > > > > > > > > > > > > jvms > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > serialization exception in > > > > > writeExternalMeta > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > > > writeExternal(ObjectOutput > > > > > > > > > out) > > > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be serialized. > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, Alexey > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what > you > > > > want, > > > > > > > but I > > > > > > > > > > have > > > > > > > > > > > a > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > > > - What is the API for the proposed > > > > change? > > > > > > In > > > > > > > > your > > > > > > > > > > > test, > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > instance of transaction created on > > > > > ignite(0) > > > > > > to > > > > > > > > the > > > > > > > > > > > > ignite > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > ignite(1). This is obviously not > > > possible > > > > > in > > > > > > a > > > > > > > > > truly > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > > > - How will you synchronize cache > > update > > > > > > actions > > > > > > > > and > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > > > Say, you have one node that decided > > to > > > > > > commit, > > > > > > > > but > > > > > > > > > > > > another > > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > writing within this transaction. > How > > do > > > > you > > > > > > > make > > > > > > > > > sure > > > > > > > > > > > > that > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > not call commit() and rollback() > > > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > > > - How do you make sure that either > > > > > commit() > > > > > > or > > > > > > > > > > > > rollback() > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий > > > Рябов > > > > < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my initial > > > > > > > understanding > > > > > > > > > was > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > ownership from one node to > another > > > will > > > > > be > > > > > > > > > happened > > > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 > ALEKSEY > > > > > > KUZNETSOV > > > > > > > < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction > on > > > > > multiple > > > > > > > > > > threads, > > > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > > every node is able to rollback, > > or > > > > > commit > > > > > > > > > common > > > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > > > need to transfer tx between > nodes > > > in > > > > > > order > > > > > > > to > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > different node(in the same > jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, > > Alexey > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a > > > concept > > > > > of > > > > > > > > > > > transferring > > > > > > > > > > > > > of > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > one node to another? My > initial > > > > > > > > understanding > > > > > > > > > > was > > > > > > > > > > > > > that > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > > > to update keys in a > transaction > > > > from > > > > > > > > multiple > > > > > > > > > > > > threads > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 > > > ALEKSEY > > > > > > > > KUZNETSOV > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider transaction > > > > started > > > > > in > > > > > > > one > > > > > > > > > > node, > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > > > The following test > describes > > my > > > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > transactions = > > > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, > Integer> > > > > cache > > > > > = > > > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > > > transactions.txStart( > > > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > > > fut = > > > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions ts = > > > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *ts.txStart(...)* > > we > > > > just > > > > > > > > rebind > > > > > > > > > > *tx* > > > > > > > > > > > > to > > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void > > txStart(Transaction > > > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we > alter > > > > > > > *threadMap* > > > > > > > > > so > > > > > > > > > > > that > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в 22:38, > > > Denis > > > > > > > Magda < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the rational > > > > behind > > > > > > this > > > > > > > > and > > > > > > > > > > the > > > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at 3:19 > > AM, > > > > > > ALEKSEY > > > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing > > > > > distributed > > > > > > > > > > > transaction > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > > > node, and continued at > > > other > > > > > one. > > > > > > > Has > > > > > > > > > > > anybody > > > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > -- > > > > > > *Best Regards,* > > > > > > *Kuznetsov Aleksey* > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > *Best Regards,* *Kuznetsov Aleksey* |
|
|
The problem "How to run millions of entities, and millions of operations on
a single Pentium3" is out of scope here. Do the math, plan capacity reasonably. Sergi 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > hmm, If we have millions of entities, and millions of operations, would not > this approache lead to memory overflow and perfomance degradation > > чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin <[hidden email]>: > > > 1. Actually you have to check versions on all the values you have read > > during the tx. > > > > For example if we have [k1 => v1, k2 => v2] and do: > > > > put(k1, get(k2) + 5) > > > > We have to remember the version for k2. This logic can be relatively > easily > > encapsulated in a framework atop of Ignite. You need to implement one to > > make all this stuff usable. > > > > 2. I suggest to avoid any locking here, because you easily will end up > with > > deadlocks. If you do not have too frequent updates for your keys, > > optimistic approach will work just fine. > > > > Theoretically in the Committer Service you can start a thread for the > > lifetime of the whole distributed transaction, take a lock on the key > using > > IgniteCache.lock(K key) before executing any Services, wait for all the > > services to complete, execute optimistic commit in the same thread while > > keeping this lock and then release it. Notice that all the Ignite > > transactions inside of all Services must be optimistic here to be able to > > read this locked key. > > > > But again I do not recommend you using this approach until you have a > > reliable deadlock avoidance scheme. > > > > Sergi > > > > > > > > > > > > > > > > 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > > > Yeah, now i got it. > > > There are some doubts on this approach > > > 1) During optimistic commit phase, when you assure no one altered the > > > original values, you must check versions of other dependent keys. How > > could > > > we obtain those keys(in an automative manner, of course) ? > > > 2) How could we lock a key before some Service A introduce changes? So > no > > > other service is allowed to change this key-value?(sort of pessimistic > > > blocking) > > > May be you know some implementations of such approach ? > > > > > > ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > [hidden email] > > >: > > > > > > > Thank you very much for help. I will answer later. > > > > > > > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > [hidden email] > > >: > > > > > > > > All the services do not update key in place, but only generate new > keys > > > > augmented by otx and store the updated value in the same cache + > > remember > > > > the keys and versions participating in the transaction in some > separate > > > > atomic cache. > > > > > > > > Follow this sequence of changes applied to cache contents by each > > > Service: > > > > > > > > Initial cache contents: > > > > [k1 => v1] > > > > [k2 => v2] > > > > [k3 => v3] > > > > > > > > Cache contents after Service A: > > > > [k1 => v1] > > > > [k2 => v2] > > > > [k3 => v3] > > > > [k1x => v1a] > > > > [k2x => v2a] > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate atomic > > cache > > > > > > > > Cache contents after Service B: > > > > [k1 => v1] > > > > [k2 => v2] > > > > [k3 => v3] > > > > [k1x => v1a] > > > > [k2x => v2ab] > > > > [k3x => v3b] > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > separate > > > > atomic cache > > > > > > > > Finally the Committer Service takes this map of updated keys and > their > > > > versions from some separate atomic cache, starts Ignite transaction > and > > > > replaces all the values for k* keys to values taken from k*x keys. > The > > > > successful result must be the following: > > > > > > > > [k1 => v1a] > > > > [k2 => v2ab] > > > > [k3 => v3b] > > > > [k1x => v1a] > > > > [k2x => v2ab] > > > > [k3x => v3b] > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > separate > > > > atomic cache > > > > > > > > But Committer Service also has to check that no one updated the > > original > > > > values before us, because otherwise we can not give any > serializability > > > > guarantee for these distributed transactions. Here we may need to > check > > > not > > > > only versions of the updated keys, but also versions of any other > keys > > > end > > > > result depends on. > > > > > > > > After that Committer Service has to do a cleanup (may be outside of > the > > > > committing tx) to come to the following final state: > > > > > > > > [k1 => v1a] > > > > [k2 => v2ab] > > > > [k3 => v3b] > > > > > > > > Makes sense? > > > > > > > > Sergi > > > > > > > > > > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > > >: > > > > > > > > > - what do u mean by saying " > > > > > *in a single transaction checks value versions for all the old > values > > > > > and replaces them with calculated new ones *"? Every time you > > > change > > > > > value(in some service), you store it to *some special atomic > > cache* > > > , > > > > so > > > > > when all services ceased working, Service commiter got a values > > with > > > > the > > > > > last versions. > > > > > - After "*does cleanup of temporary keys and values*" Service > > > commiter > > > > > persists them into permanent store, isn't it ? > > > > > - I cant grasp your though, you say "*in case of version > mismatch > > or > > > > TX > > > > > timeout just rollbacks*". But what versions would it match? > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > > [hidden email] > > > >: > > > > > > > > > > > Ok, here is what you actually need to implement at the > application > > > > level. > > > > > > > > > > > > Lets say we have to call 2 services in the following order: > > > > > > - Service A: wants to update keys [k1 => v1, k2 => v2] to > [k1 > > => > > > > > v1a, > > > > > > k2 => v2a] > > > > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to [k2 > > => > > > > > v2ab, > > > > > > k3 => v3b] > > > > > > > > > > > > The change > > > > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > > > > must happen in a single transaction. > > > > > > > > > > > > > > > > > > Optimistic protocol to solve this: > > > > > > > > > > > > Each cache key must have a field `otx`, which is a unique > > > orchestrator > > > > TX > > > > > > identifier - it must be a parameter passed to all the services. > If > > > > `otx` > > > > > is > > > > > > set to some value it means that it is an intermediate key and is > > > > visible > > > > > > only inside of some transaction, for the finalized key `otx` must > > be > > > > > null - > > > > > > it means the key is committed and visible for everyone. > > > > > > > > > > > > Each cache value must have a field `ver` which is a version of > that > > > > > value. > > > > > > > > > > > > For both fields (`otx` and `ver`) the safest way is to use UUID. > > > > > > > > > > > > Workflow is the following: > > > > > > > > > > > > Orchestrator starts the distributed transaction with `otx` = x > and > > > > passes > > > > > > this parameter to all the services. > > > > > > > > > > > > Service A: > > > > > > - does some computations > > > > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > > > > where > > > > > > Za - left time from max Orchestrator TX duration after > > > > Service > > > > > A > > > > > > end > > > > > > k1x, k2x - new temporary keys with field `otx` = x > > > > > > v2a has updated version `ver` > > > > > > - returns a set of updated keys and all the old versions to the > > > > > > orchestrator > > > > > > or just stores it in some special atomic cache like > > > > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > > > > > > > Service B: > > > > > > - retrieves the updated value k2x => v2a because it knows `otx` > = > > x > > > > > > - does computations > > > > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > > > > - updates the set of updated keys like [x => (k1 -> ver1, k2 -> > > > ver2, > > > > k3 > > > > > > -> ver3)] TTL = Zb > > > > > > > > > > > > Service Committer (may be embedded into Orchestrator): > > > > > > - takes all the updated keys and versions for `otx` = x > > > > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > > > > - in a single transaction checks value versions for all the old > > > values > > > > > > and replaces them with calculated new ones > > > > > > - does cleanup of temporary keys and values > > > > > > - in case of version mismatch or TX timeout just rollbacks and > > > signals > > > > > > to Orchestrator to restart the job with new `otx` > > > > > > > > > > > > PROFIT!! > > > > > > > > > > > > This approach even allows you to run independent parts of the > graph > > > in > > > > > > parallel (with TX transfer you will always run only one at a > time). > > > > Also > > > > > it > > > > > > does not require inventing any special fault tolerance technics > > > because > > > > > > Ignite caches are already fault tolerant and all the intermediate > > > > results > > > > > > are virtually invisible and stored with TTL, thus in case of any > > > crash > > > > > you > > > > > > will not have inconsistent state or garbage. > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Okay, we are open for proposals on business task. I mean, we > can > > > make > > > > > use > > > > > > > of some other thing, not distributed transaction. Not > transaction > > > > yet. > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > > [hidden email] > > > >: > > > > > > > > > > > > > > > IMO the use case makes sense. However, as Sergi already > > > mentioned, > > > > > the > > > > > > > > problem is far more complex, than simply passing TX state > over > > a > > > > > wire. > > > > > > > Most > > > > > > > > probably a kind of coordinator will be required still to > manage > > > all > > > > > > kinds > > > > > > > > of failures. This task should be started with clean design > > > proposal > > > > > > > > explaining how we handle all these concurrent events. And > only > > > > then, > > > > > > when > > > > > > > > we understand all implications, we should move to development > > > > stage. > > > > > > > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > Right > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > Good! Basically your orchestrator just takes some > > predefined > > > > > graph > > > > > > of > > > > > > > > > > distributed services to be invoked, calls them by some > kind > > > of > > > > > RPC > > > > > > > and > > > > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible for > > > > managing > > > > > > > > business > > > > > > > > > > > scenarios flows. Many nodes are involved in scenarios. > > They > > > > > > > exchange > > > > > > > > > data > > > > > > > > > > > and folow one another. If you acquinted with BPMN > > > framework, > > > > so > > > > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from > > > Microsoft > > > > or > > > > > > > your > > > > > > > > > > custom > > > > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which > > > fulfills > > > > > > > custom > > > > > > > > > > logic. > > > > > > > > > > > > > This servers compound oriented graph (BPMN process) > > > which > > > > > > > > > controlled > > > > > > > > > > by > > > > > > > > > > > > > Orchestrator. > > > > > > > > > > > > > For instance, *server1 *creates *variable A *with > > > value > > > > 1, > > > > > > > > > persists > > > > > > > > > > it > > > > > > > > > > > > to > > > > > > > > > > > > > IGNITE cache and creates *variable B *and sends it > > to* > > > > > > server2. > > > > > > > > > *The > > > > > > > > > > > > > latests receives *variable B*, do some logic with > it > > > and > > > > > > stores > > > > > > > > to > > > > > > > > > > > > IGNITE. > > > > > > > > > > > > > All the work made by both servers must be fulfilled > > in > > > > > *one* > > > > > > > > > > > transaction. > > > > > > > > > > > > > Because we need all information done, or > > > > > nothing(rollbacked). > > > > > > > The > > > > > > > > > > > > scenario > > > > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong > > > > solution > > > > > > for > > > > > > > > it. > > > > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts > transaction > > > in > > > > > one > > > > > > > > node, > > > > > > > > > > and > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > this transaction in another jvm node(or > rollback > > it > > > > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some > > > > simplistic > > > > > > > > > scenario, > > > > > > > > > > > get > > > > > > > > > > > > > > ready > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > write many fault tolerance tests and make > sure > > > that > > > > > you > > > > > > > TXs > > > > > > > > > > work > > > > > > > > > > > > > > > gracefully > > > > > > > > > > > > > > > > in all modes in case of crashes. Also make > sure > > > > that > > > > > we > > > > > > > do > > > > > > > > > not > > > > > > > > > > > have > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > performance drops after all your changes in > > > > existing > > > > > > > > > > benchmarks. > > > > > > > > > > > > All > > > > > > > > > > > > > in > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > I don't believe these conditions will be met > > and > > > > your > > > > > > > > > > > contribution > > > > > > > > > > > > > will > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX > to > > > > > another > > > > > > > > node? > > > > > > > > > > The > > > > > > > > > > > > > > problem > > > > > > > > > > > > > > > > statement itself is already wrong. What > > business > > > > case > > > > > > you > > > > > > > > are > > > > > > > > > > > > trying > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > solve? I'm sure everything you need can be > done > > > in > > > > a > > > > > > much > > > > > > > > > more > > > > > > > > > > > > simple > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi > Vladykin < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and > > deserializing > > > it > > > > > on > > > > > > > > > another > > > > > > > > > > > node > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > meaningless, because other nodes > > > participating > > > > in > > > > > > the > > > > > > > > TX > > > > > > > > > > have > > > > > > > > > > > > to > > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > the new coordinator. This will require > > > protocol > > > > > > > > changes, > > > > > > > > > we > > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > have fault tolerance and performance > > issues. > > > > IMO > > > > > > the > > > > > > > > > whole > > > > > > > > > > > idea > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > > > > and it makes no sense to waste time on > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY > > KUZNETSOV > > > < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState implememntation > > > > contains > > > > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy > > > > Setrakyan > > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that > we > > > are > > > > > > > passing > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > > > around. Such object may contain all > > sorts > > > > of > > > > > > > Ignite > > > > > > > > > > > > context. > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > > needs to be passed across, we should > > > > create a > > > > > > > > special > > > > > > > > > > > > > transfer > > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, > > ALEKSEY > > > > > > > KUZNETSOV > > > > > > > > < > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues > > > preventing > > > > > > > > > transaction > > > > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > > > > At first, After transaction > > > serialization > > > > > and > > > > > > > > > > > > > deserialization > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > > > > server, there is no txState. So im > > > going > > > > to > > > > > > put > > > > > > > > it > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized > > > transaction > > > > > > lacks > > > > > > > of > > > > > > > > > > > shared > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > > > > field at TransactionProxyImpl. > > Perhaps, > > > > it > > > > > > must > > > > > > > > be > > > > > > > > > > > > injected > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, > ALEKSEY > > > > > > KUZNETSOV > > > > > > > < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing > > > > transaction > > > > > > in > > > > > > > > > > > different > > > > > > > > > > > > > jvms > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > > serialization exception in > > > > > > writeExternalMeta > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > > > > writeExternal(ObjectOutput > > > > > > > > > > out) > > > > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be > serialized. > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, > Alexey > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get what > > you > > > > > want, > > > > > > > > but I > > > > > > > > > > > have > > > > > > > > > > > > a > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > > > > - What is the API for the > proposed > > > > > change? > > > > > > > In > > > > > > > > > your > > > > > > > > > > > > test, > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > instance of transaction created > on > > > > > > ignite(0) > > > > > > > to > > > > > > > > > the > > > > > > > > > > > > > ignite > > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > > ignite(1). This is obviously not > > > > possible > > > > > > in > > > > > > > a > > > > > > > > > > truly > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > > > > - How will you synchronize cache > > > update > > > > > > > actions > > > > > > > > > and > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > > > > Say, you have one node that > decided > > > to > > > > > > > commit, > > > > > > > > > but > > > > > > > > > > > > > another > > > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > writing within this transaction. > > How > > > do > > > > > you > > > > > > > > make > > > > > > > > > > sure > > > > > > > > > > > > > that > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > not call commit() and rollback() > > > > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > > > > - How do you make sure that > either > > > > > > commit() > > > > > > > or > > > > > > > > > > > > > rollback() > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 > Дмитрий > > > > Рябов > > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my > initial > > > > > > > > understanding > > > > > > > > > > was > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > > ownership from one node to > > another > > > > will > > > > > > be > > > > > > > > > > happened > > > > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > > > originating node is gone down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 > > ALEKSEY > > > > > > > KUZNETSOV > > > > > > > > < > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span transaction > > on > > > > > > multiple > > > > > > > > > > > threads, > > > > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > > > every node is able to > rollback, > > > or > > > > > > commit > > > > > > > > > > common > > > > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > > > > need to transfer tx between > > nodes > > > > in > > > > > > > order > > > > > > > > to > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > different node(in the same > > jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 15:20, > > > Alexey > > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you want a > > > > concept > > > > > > of > > > > > > > > > > > > transferring > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > > one node to another? My > > initial > > > > > > > > > understanding > > > > > > > > > > > was > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > > > > to update keys in a > > transaction > > > > > from > > > > > > > > > multiple > > > > > > > > > > > > > threads > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 GMT+03:00 > > > > ALEKSEY > > > > > > > > > KUZNETSOV > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider > transaction > > > > > started > > > > > > in > > > > > > > > one > > > > > > > > > > > node, > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > > > > The following test > > describes > > > my > > > > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = > ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > > transactions = > > > > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, > > Integer> > > > > > cache > > > > > > = > > > > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > > > > transactions.txStart( > > > > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > > > > fut = > > > > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > ts = > > > > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method > *ts.txStart(...)* > > > we > > > > > just > > > > > > > > > rebind > > > > > > > > > > > *tx* > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void > > > txStart(Transaction > > > > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > > > > cctx.tm().reopenTx( > > > > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we > > alter > > > > > > > > *threadMap* > > > > > > > > > > so > > > > > > > > > > > > that > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в > 22:38, > > > > Denis > > > > > > > > Magda < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the > rational > > > > > behind > > > > > > > this > > > > > > > > > and > > > > > > > > > > > the > > > > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at > 3:19 > > > AM, > > > > > > > ALEKSEY > > > > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im designing > > > > > > distributed > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > > > > node, and continued > at > > > > other > > > > > > one. > > > > > > > > Has > > > > > > > > > > > > anybody > > > > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Best Regards,* > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > -- > > > > > > *Best Regards,* > > > > > > *Kuznetsov Aleksey* > > > > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > |
|
|
we've discovered several problems regarding your "accumulation"
approach.These are 1. perfomance issues when transfering data from temporary cache to permanent one. Keep in mind big deal of concurent transactions in Service commiter 2. extreme memory load when keeping temporary cache in memory 3. As long as user is not acquainted with ignite, working with cache must be transparent for him. Keep this in mind. User's node can evaluate logic with no transaction at all, so we should deal with both types of execution flow : transactional and non-transactional.Another one problem is transaction id support at the user node. We would have handled all this issues and many more. 4. we cannot pessimistically lock entity. As a result, we decided to move on building distributed transaction. We put aside your "accumulation" approach until we realize how to solve difficulties above . чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin <[hidden email]>: > The problem "How to run millions of entities, and millions of operations on > a single Pentium3" is out of scope here. Do the math, plan capacity > reasonably. > > Sergi > > 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > hmm, If we have millions of entities, and millions of operations, would > not > > this approache lead to memory overflow and perfomance degradation > > > > чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin <[hidden email]>: > > > > > 1. Actually you have to check versions on all the values you have read > > > during the tx. > > > > > > For example if we have [k1 => v1, k2 => v2] and do: > > > > > > put(k1, get(k2) + 5) > > > > > > We have to remember the version for k2. This logic can be relatively > > easily > > > encapsulated in a framework atop of Ignite. You need to implement one > to > > > make all this stuff usable. > > > > > > 2. I suggest to avoid any locking here, because you easily will end up > > with > > > deadlocks. If you do not have too frequent updates for your keys, > > > optimistic approach will work just fine. > > > > > > Theoretically in the Committer Service you can start a thread for the > > > lifetime of the whole distributed transaction, take a lock on the key > > using > > > IgniteCache.lock(K key) before executing any Services, wait for all the > > > services to complete, execute optimistic commit in the same thread > while > > > keeping this lock and then release it. Notice that all the Ignite > > > transactions inside of all Services must be optimistic here to be able > to > > > read this locked key. > > > > > > But again I do not recommend you using this approach until you have a > > > reliable deadlock avoidance scheme. > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > > > Yeah, now i got it. > > > > There are some doubts on this approach > > > > 1) During optimistic commit phase, when you assure no one altered the > > > > original values, you must check versions of other dependent keys. How > > > could > > > > we obtain those keys(in an automative manner, of course) ? > > > > 2) How could we lock a key before some Service A introduce changes? > So > > no > > > > other service is allowed to change this key-value?(sort of > pessimistic > > > > blocking) > > > > May be you know some implementations of such approach ? > > > > > > > > ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > > [hidden email] > > > >: > > > > > > > > > Thank you very much for help. I will answer later. > > > > > > > > > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > > [hidden email] > > > >: > > > > > > > > > > All the services do not update key in place, but only generate new > > keys > > > > > augmented by otx and store the updated value in the same cache + > > > remember > > > > > the keys and versions participating in the transaction in some > > separate > > > > > atomic cache. > > > > > > > > > > Follow this sequence of changes applied to cache contents by each > > > > Service: > > > > > > > > > > Initial cache contents: > > > > > [k1 => v1] > > > > > [k2 => v2] > > > > > [k3 => v3] > > > > > > > > > > Cache contents after Service A: > > > > > [k1 => v1] > > > > > [k2 => v2] > > > > > [k3 => v3] > > > > > [k1x => v1a] > > > > > [k2x => v2a] > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate atomic > > > cache > > > > > > > > > > Cache contents after Service B: > > > > > [k1 => v1] > > > > > [k2 => v2] > > > > > [k3 => v3] > > > > > [k1x => v1a] > > > > > [k2x => v2ab] > > > > > [k3x => v3b] > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > separate > > > > > atomic cache > > > > > > > > > > Finally the Committer Service takes this map of updated keys and > > their > > > > > versions from some separate atomic cache, starts Ignite transaction > > and > > > > > replaces all the values for k* keys to values taken from k*x keys. > > The > > > > > successful result must be the following: > > > > > > > > > > [k1 => v1a] > > > > > [k2 => v2ab] > > > > > [k3 => v3b] > > > > > [k1x => v1a] > > > > > [k2x => v2ab] > > > > > [k3x => v3b] > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > separate > > > > > atomic cache > > > > > > > > > > But Committer Service also has to check that no one updated the > > > original > > > > > values before us, because otherwise we can not give any > > serializability > > > > > guarantee for these distributed transactions. Here we may need to > > check > > > > not > > > > > only versions of the updated keys, but also versions of any other > > keys > > > > end > > > > > result depends on. > > > > > > > > > > After that Committer Service has to do a cleanup (may be outside of > > the > > > > > committing tx) to come to the following final state: > > > > > > > > > > [k1 => v1a] > > > > > [k2 => v2ab] > > > > > [k3 => v3b] > > > > > > > > > > Makes sense? > > > > > > > > > > Sergi > > > > > > > > > > > > > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > > [hidden email] > > > >: > > > > > > > > > > > - what do u mean by saying " > > > > > > *in a single transaction checks value versions for all the old > > values > > > > > > and replaces them with calculated new ones *"? Every time you > > > > change > > > > > > value(in some service), you store it to *some special atomic > > > cache* > > > > , > > > > > so > > > > > > when all services ceased working, Service commiter got a > values > > > with > > > > > the > > > > > > last versions. > > > > > > - After "*does cleanup of temporary keys and values*" Service > > > > commiter > > > > > > persists them into permanent store, isn't it ? > > > > > > - I cant grasp your though, you say "*in case of version > > mismatch > > > or > > > > > TX > > > > > > timeout just rollbacks*". But what versions would it match? > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > > Ok, here is what you actually need to implement at the > > application > > > > > level. > > > > > > > > > > > > > > Lets say we have to call 2 services in the following order: > > > > > > > - Service A: wants to update keys [k1 => v1, k2 => v2] to > > [k1 > > > => > > > > > > v1a, > > > > > > > k2 => v2a] > > > > > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to > [k2 > > > => > > > > > > v2ab, > > > > > > > k3 => v3b] > > > > > > > > > > > > > > The change > > > > > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > > > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > > > > > must happen in a single transaction. > > > > > > > > > > > > > > > > > > > > > Optimistic protocol to solve this: > > > > > > > > > > > > > > Each cache key must have a field `otx`, which is a unique > > > > orchestrator > > > > > TX > > > > > > > identifier - it must be a parameter passed to all the services. > > If > > > > > `otx` > > > > > > is > > > > > > > set to some value it means that it is an intermediate key and > is > > > > > visible > > > > > > > only inside of some transaction, for the finalized key `otx` > must > > > be > > > > > > null - > > > > > > > it means the key is committed and visible for everyone. > > > > > > > > > > > > > > Each cache value must have a field `ver` which is a version of > > that > > > > > > value. > > > > > > > > > > > > > > For both fields (`otx` and `ver`) the safest way is to use > UUID. > > > > > > > > > > > > > > Workflow is the following: > > > > > > > > > > > > > > Orchestrator starts the distributed transaction with `otx` = x > > and > > > > > passes > > > > > > > this parameter to all the services. > > > > > > > > > > > > > > Service A: > > > > > > > - does some computations > > > > > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > > > > > where > > > > > > > Za - left time from max Orchestrator TX duration > after > > > > > Service > > > > > > A > > > > > > > end > > > > > > > k1x, k2x - new temporary keys with field `otx` = x > > > > > > > v2a has updated version `ver` > > > > > > > - returns a set of updated keys and all the old versions to > the > > > > > > > orchestrator > > > > > > > or just stores it in some special atomic cache like > > > > > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > > > > > > > > > Service B: > > > > > > > - retrieves the updated value k2x => v2a because it knows > `otx` > > = > > > x > > > > > > > - does computations > > > > > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > > > > > - updates the set of updated keys like [x => (k1 -> ver1, k2 > -> > > > > ver2, > > > > > k3 > > > > > > > -> ver3)] TTL = Zb > > > > > > > > > > > > > > Service Committer (may be embedded into Orchestrator): > > > > > > > - takes all the updated keys and versions for `otx` = x > > > > > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > > > > > - in a single transaction checks value versions for all the > old > > > > values > > > > > > > and replaces them with calculated new ones > > > > > > > - does cleanup of temporary keys and values > > > > > > > - in case of version mismatch or TX timeout just rollbacks and > > > > signals > > > > > > > to Orchestrator to restart the job with new `otx` > > > > > > > > > > > > > > PROFIT!! > > > > > > > > > > > > > > This approach even allows you to run independent parts of the > > graph > > > > in > > > > > > > parallel (with TX transfer you will always run only one at a > > time). > > > > > Also > > > > > > it > > > > > > > does not require inventing any special fault tolerance technics > > > > because > > > > > > > Ignite caches are already fault tolerant and all the > intermediate > > > > > results > > > > > > > are virtually invisible and stored with TTL, thus in case of > any > > > > crash > > > > > > you > > > > > > > will not have inconsistent state or garbage. > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > Okay, we are open for proposals on business task. I mean, we > > can > > > > make > > > > > > use > > > > > > > > of some other thing, not distributed transaction. Not > > transaction > > > > > yet. > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > > > [hidden email] > > > > >: > > > > > > > > > > > > > > > > > IMO the use case makes sense. However, as Sergi already > > > > mentioned, > > > > > > the > > > > > > > > > problem is far more complex, than simply passing TX state > > over > > > a > > > > > > wire. > > > > > > > > Most > > > > > > > > > probably a kind of coordinator will be required still to > > manage > > > > all > > > > > > > kinds > > > > > > > > > of failures. This task should be started with clean design > > > > proposal > > > > > > > > > explaining how we handle all these concurrent events. And > > only > > > > > then, > > > > > > > when > > > > > > > > > we understand all implications, we should move to > development > > > > > stage. > > > > > > > > > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > > > Right > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Good! Basically your orchestrator just takes some > > > predefined > > > > > > graph > > > > > > > of > > > > > > > > > > > distributed services to be invoked, calls them by some > > kind > > > > of > > > > > > RPC > > > > > > > > and > > > > > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible for > > > > > managing > > > > > > > > > business > > > > > > > > > > > > scenarios flows. Many nodes are involved in > scenarios. > > > They > > > > > > > > exchange > > > > > > > > > > data > > > > > > > > > > > > and folow one another. If you acquinted with BPMN > > > > framework, > > > > > so > > > > > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > > > > > [hidden email] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from > > > > Microsoft > > > > > or > > > > > > > > your > > > > > > > > > > > custom > > > > > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers which > > > > fulfills > > > > > > > > custom > > > > > > > > > > > logic. > > > > > > > > > > > > > > This servers compound oriented graph (BPMN > process) > > > > which > > > > > > > > > > controlled > > > > > > > > > > > by > > > > > > > > > > > > > > Orchestrator. > > > > > > > > > > > > > > For instance, *server1 *creates *variable A > *with > > > > value > > > > > 1, > > > > > > > > > > persists > > > > > > > > > > > it > > > > > > > > > > > > > to > > > > > > > > > > > > > > IGNITE cache and creates *variable B *and sends > it > > > to* > > > > > > > server2. > > > > > > > > > > *The > > > > > > > > > > > > > > latests receives *variable B*, do some logic with > > it > > > > and > > > > > > > stores > > > > > > > > > to > > > > > > > > > > > > > IGNITE. > > > > > > > > > > > > > > All the work made by both servers must be > fulfilled > > > in > > > > > > *one* > > > > > > > > > > > > transaction. > > > > > > > > > > > > > > Because we need all information done, or > > > > > > nothing(rollbacked). > > > > > > > > The > > > > > > > > > > > > > scenario > > > > > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your wrong > > > > > solution > > > > > > > for > > > > > > > > > it. > > > > > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts > > transaction > > > > in > > > > > > one > > > > > > > > > node, > > > > > > > > > > > and > > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > this transaction in another jvm node(or > > rollback > > > it > > > > > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some > > > > > simplistic > > > > > > > > > > scenario, > > > > > > > > > > > > get > > > > > > > > > > > > > > > ready > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > write many fault tolerance tests and make > > sure > > > > that > > > > > > you > > > > > > > > TXs > > > > > > > > > > > work > > > > > > > > > > > > > > > > gracefully > > > > > > > > > > > > > > > > > in all modes in case of crashes. Also make > > sure > > > > > that > > > > > > we > > > > > > > > do > > > > > > > > > > not > > > > > > > > > > > > have > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > performance drops after all your changes in > > > > > existing > > > > > > > > > > > benchmarks. > > > > > > > > > > > > > All > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > I don't believe these conditions will be > met > > > and > > > > > your > > > > > > > > > > > > contribution > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending TX > > to > > > > > > another > > > > > > > > > node? > > > > > > > > > > > The > > > > > > > > > > > > > > > problem > > > > > > > > > > > > > > > > > statement itself is already wrong. What > > > business > > > > > case > > > > > > > you > > > > > > > > > are > > > > > > > > > > > > > trying > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > solve? I'm sure everything you need can be > > done > > > > in > > > > > a > > > > > > > much > > > > > > > > > > more > > > > > > > > > > > > > simple > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY > KUZNETSOV > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi > > Vladykin < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and > > > deserializing > > > > it > > > > > > on > > > > > > > > > > another > > > > > > > > > > > > node > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > meaningless, because other nodes > > > > participating > > > > > in > > > > > > > the > > > > > > > > > TX > > > > > > > > > > > have > > > > > > > > > > > > > to > > > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > the new coordinator. This will require > > > > protocol > > > > > > > > > changes, > > > > > > > > > > we > > > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > have fault tolerance and performance > > > issues. > > > > > IMO > > > > > > > the > > > > > > > > > > whole > > > > > > > > > > > > idea > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > > > > > and it makes no sense to waste time on > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY > > > KUZNETSOV > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState > implememntation > > > > > contains > > > > > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy > > > > > Setrakyan > > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me that > > we > > > > are > > > > > > > > passing > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > > > > around. Such object may contain all > > > sorts > > > > > of > > > > > > > > Ignite > > > > > > > > > > > > > context. > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > > > needs to be passed across, we > should > > > > > create a > > > > > > > > > special > > > > > > > > > > > > > > transfer > > > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, > > > ALEKSEY > > > > > > > > KUZNETSOV > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues > > > > preventing > > > > > > > > > > transaction > > > > > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > > > > > At first, After transaction > > > > serialization > > > > > > and > > > > > > > > > > > > > > deserialization > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > > > > > server, there is no txState. So > im > > > > going > > > > > to > > > > > > > put > > > > > > > > > it > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized > > > > transaction > > > > > > > lacks > > > > > > > > of > > > > > > > > > > > > shared > > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > > > > > field at TransactionProxyImpl. > > > Perhaps, > > > > > it > > > > > > > must > > > > > > > > > be > > > > > > > > > > > > > injected > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, > > ALEKSEY > > > > > > > KUZNETSOV > > > > > > > > < > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing > > > > > transaction > > > > > > > in > > > > > > > > > > > > different > > > > > > > > > > > > > > jvms > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > > > serialization exception in > > > > > > > writeExternalMeta > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > > > > > writeExternal(ObjectOutput > > > > > > > > > > > out) > > > > > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be > > serialized. > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, > > Alexey > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get > what > > > you > > > > > > want, > > > > > > > > > but I > > > > > > > > > > > > have > > > > > > > > > > > > > a > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > > > > > - What is the API for the > > proposed > > > > > > change? > > > > > > > > In > > > > > > > > > > your > > > > > > > > > > > > > test, > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > instance of transaction created > > on > > > > > > > ignite(0) > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > > > > ignite > > > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > > > ignite(1). This is obviously > not > > > > > possible > > > > > > > in > > > > > > > > a > > > > > > > > > > > truly > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > > > > > - How will you synchronize > cache > > > > update > > > > > > > > actions > > > > > > > > > > and > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > > > > > Say, you have one node that > > decided > > > > to > > > > > > > > commit, > > > > > > > > > > but > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > > writing within this > transaction. > > > How > > > > do > > > > > > you > > > > > > > > > make > > > > > > > > > > > sure > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > not call commit() and > rollback() > > > > > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > > > > > - How do you make sure that > > either > > > > > > > commit() > > > > > > > > or > > > > > > > > > > > > > > rollback() > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 > > Дмитрий > > > > > Рябов > > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my > > initial > > > > > > > > > understanding > > > > > > > > > > > was > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > > > ownership from one node to > > > another > > > > > will > > > > > > > be > > > > > > > > > > > happened > > > > > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > > > > originating node is gone > down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 > > > ALEKSEY > > > > > > > > KUZNETSOV > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span > transaction > > > on > > > > > > > multiple > > > > > > > > > > > > threads, > > > > > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > > > > every node is able to > > rollback, > > > > or > > > > > > > commit > > > > > > > > > > > common > > > > > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > > > > > need to transfer tx between > > > nodes > > > > > in > > > > > > > > order > > > > > > > > > to > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > different node(in the same > > > jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в > 15:20, > > > > Alexey > > > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you > want a > > > > > concept > > > > > > > of > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > > > one node to another? My > > > initial > > > > > > > > > > understanding > > > > > > > > > > > > was > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > > > > > to update keys in a > > > transaction > > > > > > from > > > > > > > > > > multiple > > > > > > > > > > > > > > threads > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 > GMT+03:00 > > > > > ALEKSEY > > > > > > > > > > KUZNETSOV > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider > > transaction > > > > > > started > > > > > > > in > > > > > > > > > one > > > > > > > > > > > > node, > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > > > > > The following test > > > describes > > > > my > > > > > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = > > ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > > > transactions = > > > > > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, > > > Integer> > > > > > > cache > > > > > > > = > > > > > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > > > > > transactions.txStart( > > > > > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > > > > > fut = > > > > > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > ts = > > > > > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", > 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method > > *ts.txStart(...)* > > > > we > > > > > > just > > > > > > > > > > rebind > > > > > > > > > > > > *tx* > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void > > > > txStart(Transaction > > > > > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > > > > > cctx.tm > ().reopenTx( > > > > > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* we > > > alter > > > > > > > > > *threadMap* > > > > > > > > > > > so > > > > > > > > > > > > > that > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about > it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в > > 22:38, > > > > > Denis > > > > > > > > > Magda < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the > > rational > > > > > > behind > > > > > > > > this > > > > > > > > > > and > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, at > > 3:19 > > > > AM, > > > > > > > > ALEKSEY > > > > > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im > designing > > > > > > > distributed > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > > > > > node, and continued > > at > > > > > other > > > > > > > one. > > > > > > > > > Has > > > > > > > > > > > > > anybody > > > > > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Best Regards,* > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > *Best Regards,* *Kuznetsov Aleksey* |
|
|
You know better, go ahead! :)
Sergi 2017-03-17 16:16 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > we've discovered several problems regarding your "accumulation" > approach.These are > > 1. perfomance issues when transfering data from temporary cache to > permanent one. Keep in mind big deal of concurent transactions in > Service > commiter > 2. extreme memory load when keeping temporary cache in memory > 3. As long as user is not acquainted with ignite, working with cache > must be transparent for him. Keep this in mind. User's node can evaluate > logic with no transaction at all, so we should deal with both types of > execution flow : transactional and non-transactional.Another one > problem is > transaction id support at the user node. We would have handled all this > issues and many more. > 4. we cannot pessimistically lock entity. > > As a result, we decided to move on building distributed transaction. We put > aside your "accumulation" approach until we realize how to solve > difficulties above . > > чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin <[hidden email]>: > > > The problem "How to run millions of entities, and millions of operations > on > > a single Pentium3" is out of scope here. Do the math, plan capacity > > reasonably. > > > > Sergi > > > > 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > > > hmm, If we have millions of entities, and millions of operations, would > > not > > > this approache lead to memory overflow and perfomance degradation > > > > > > чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin <[hidden email] > >: > > > > > > > 1. Actually you have to check versions on all the values you have > read > > > > during the tx. > > > > > > > > For example if we have [k1 => v1, k2 => v2] and do: > > > > > > > > put(k1, get(k2) + 5) > > > > > > > > We have to remember the version for k2. This logic can be relatively > > > easily > > > > encapsulated in a framework atop of Ignite. You need to implement one > > to > > > > make all this stuff usable. > > > > > > > > 2. I suggest to avoid any locking here, because you easily will end > up > > > with > > > > deadlocks. If you do not have too frequent updates for your keys, > > > > optimistic approach will work just fine. > > > > > > > > Theoretically in the Committer Service you can start a thread for the > > > > lifetime of the whole distributed transaction, take a lock on the key > > > using > > > > IgniteCache.lock(K key) before executing any Services, wait for all > the > > > > services to complete, execute optimistic commit in the same thread > > while > > > > keeping this lock and then release it. Notice that all the Ignite > > > > transactions inside of all Services must be optimistic here to be > able > > to > > > > read this locked key. > > > > > > > > But again I do not recommend you using this approach until you have a > > > > reliable deadlock avoidance scheme. > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > > >: > > > > > > > > > Yeah, now i got it. > > > > > There are some doubts on this approach > > > > > 1) During optimistic commit phase, when you assure no one altered > the > > > > > original values, you must check versions of other dependent keys. > How > > > > could > > > > > we obtain those keys(in an automative manner, of course) ? > > > > > 2) How could we lock a key before some Service A introduce changes? > > So > > > no > > > > > other service is allowed to change this key-value?(sort of > > pessimistic > > > > > blocking) > > > > > May be you know some implementations of such approach ? > > > > > > > > > > ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > > > [hidden email] > > > > >: > > > > > > > > > > > Thank you very much for help. I will answer later. > > > > > > > > > > > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > > > [hidden email] > > > > >: > > > > > > > > > > > > All the services do not update key in place, but only generate > new > > > keys > > > > > > augmented by otx and store the updated value in the same cache + > > > > remember > > > > > > the keys and versions participating in the transaction in some > > > separate > > > > > > atomic cache. > > > > > > > > > > > > Follow this sequence of changes applied to cache contents by each > > > > > Service: > > > > > > > > > > > > Initial cache contents: > > > > > > [k1 => v1] > > > > > > [k2 => v2] > > > > > > [k3 => v3] > > > > > > > > > > > > Cache contents after Service A: > > > > > > [k1 => v1] > > > > > > [k2 => v2] > > > > > > [k3 => v3] > > > > > > [k1x => v1a] > > > > > > [k2x => v2a] > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate > atomic > > > > cache > > > > > > > > > > > > Cache contents after Service B: > > > > > > [k1 => v1] > > > > > > [k2 => v2] > > > > > > [k3 => v3] > > > > > > [k1x => v1a] > > > > > > [k2x => v2ab] > > > > > > [k3x => v3b] > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > > separate > > > > > > atomic cache > > > > > > > > > > > > Finally the Committer Service takes this map of updated keys and > > > their > > > > > > versions from some separate atomic cache, starts Ignite > transaction > > > and > > > > > > replaces all the values for k* keys to values taken from k*x > keys. > > > The > > > > > > successful result must be the following: > > > > > > > > > > > > [k1 => v1a] > > > > > > [k2 => v2ab] > > > > > > [k3 => v3b] > > > > > > [k1x => v1a] > > > > > > [k2x => v2ab] > > > > > > [k3x => v3b] > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > > separate > > > > > > atomic cache > > > > > > > > > > > > But Committer Service also has to check that no one updated the > > > > original > > > > > > values before us, because otherwise we can not give any > > > serializability > > > > > > guarantee for these distributed transactions. Here we may need to > > > check > > > > > not > > > > > > only versions of the updated keys, but also versions of any other > > > keys > > > > > end > > > > > > result depends on. > > > > > > > > > > > > After that Committer Service has to do a cleanup (may be outside > of > > > the > > > > > > committing tx) to come to the following final state: > > > > > > > > > > > > [k1 => v1a] > > > > > > [k2 => v2ab] > > > > > > [k3 => v3b] > > > > > > > > > > > > Makes sense? > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > > > [hidden email] > > > > >: > > > > > > > > > > > > > - what do u mean by saying " > > > > > > > *in a single transaction checks value versions for all the old > > > values > > > > > > > and replaces them with calculated new ones *"? Every time > you > > > > > change > > > > > > > value(in some service), you store it to *some special atomic > > > > cache* > > > > > , > > > > > > so > > > > > > > when all services ceased working, Service commiter got a > > values > > > > with > > > > > > the > > > > > > > last versions. > > > > > > > - After "*does cleanup of temporary keys and values*" > Service > > > > > commiter > > > > > > > persists them into permanent store, isn't it ? > > > > > > > - I cant grasp your though, you say "*in case of version > > > mismatch > > > > or > > > > > > TX > > > > > > > timeout just rollbacks*". But what versions would it match? > > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > Ok, here is what you actually need to implement at the > > > application > > > > > > level. > > > > > > > > > > > > > > > > Lets say we have to call 2 services in the following order: > > > > > > > > - Service A: wants to update keys [k1 => v1, k2 => v2] to > > > [k1 > > > > => > > > > > > > v1a, > > > > > > > > k2 => v2a] > > > > > > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] to > > [k2 > > > > => > > > > > > > v2ab, > > > > > > > > k3 => v3b] > > > > > > > > > > > > > > > > The change > > > > > > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > > > > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > > > > > > must happen in a single transaction. > > > > > > > > > > > > > > > > > > > > > > > > Optimistic protocol to solve this: > > > > > > > > > > > > > > > > Each cache key must have a field `otx`, which is a unique > > > > > orchestrator > > > > > > TX > > > > > > > > identifier - it must be a parameter passed to all the > services. > > > If > > > > > > `otx` > > > > > > > is > > > > > > > > set to some value it means that it is an intermediate key and > > is > > > > > > visible > > > > > > > > only inside of some transaction, for the finalized key `otx` > > must > > > > be > > > > > > > null - > > > > > > > > it means the key is committed and visible for everyone. > > > > > > > > > > > > > > > > Each cache value must have a field `ver` which is a version > of > > > that > > > > > > > value. > > > > > > > > > > > > > > > > For both fields (`otx` and `ver`) the safest way is to use > > UUID. > > > > > > > > > > > > > > > > Workflow is the following: > > > > > > > > > > > > > > > > Orchestrator starts the distributed transaction with `otx` = > x > > > and > > > > > > passes > > > > > > > > this parameter to all the services. > > > > > > > > > > > > > > > > Service A: > > > > > > > > - does some computations > > > > > > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > > > > > > where > > > > > > > > Za - left time from max Orchestrator TX duration > > after > > > > > > Service > > > > > > > A > > > > > > > > end > > > > > > > > k1x, k2x - new temporary keys with field `otx` = x > > > > > > > > v2a has updated version `ver` > > > > > > > > - returns a set of updated keys and all the old versions to > > the > > > > > > > > orchestrator > > > > > > > > or just stores it in some special atomic cache like > > > > > > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > > > > > > > > > > > Service B: > > > > > > > > - retrieves the updated value k2x => v2a because it knows > > `otx` > > > = > > > > x > > > > > > > > - does computations > > > > > > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > > > > > > - updates the set of updated keys like [x => (k1 -> ver1, k2 > > -> > > > > > ver2, > > > > > > k3 > > > > > > > > -> ver3)] TTL = Zb > > > > > > > > > > > > > > > > Service Committer (may be embedded into Orchestrator): > > > > > > > > - takes all the updated keys and versions for `otx` = x > > > > > > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > > > > > > - in a single transaction checks value versions for all the > > old > > > > > values > > > > > > > > and replaces them with calculated new ones > > > > > > > > - does cleanup of temporary keys and values > > > > > > > > - in case of version mismatch or TX timeout just rollbacks > and > > > > > signals > > > > > > > > to Orchestrator to restart the job with new `otx` > > > > > > > > > > > > > > > > PROFIT!! > > > > > > > > > > > > > > > > This approach even allows you to run independent parts of the > > > graph > > > > > in > > > > > > > > parallel (with TX transfer you will always run only one at a > > > time). > > > > > > Also > > > > > > > it > > > > > > > > does not require inventing any special fault tolerance > technics > > > > > because > > > > > > > > Ignite caches are already fault tolerant and all the > > intermediate > > > > > > results > > > > > > > > are virtually invisible and stored with TTL, thus in case of > > any > > > > > crash > > > > > > > you > > > > > > > > will not have inconsistent state or garbage. > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > Okay, we are open for proposals on business task. I mean, > we > > > can > > > > > make > > > > > > > use > > > > > > > > > of some other thing, not distributed transaction. Not > > > transaction > > > > > > yet. > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > > > > > IMO the use case makes sense. However, as Sergi already > > > > > mentioned, > > > > > > > the > > > > > > > > > > problem is far more complex, than simply passing TX state > > > over > > > > a > > > > > > > wire. > > > > > > > > > Most > > > > > > > > > > probably a kind of coordinator will be required still to > > > manage > > > > > all > > > > > > > > kinds > > > > > > > > > > of failures. This task should be started with clean > design > > > > > proposal > > > > > > > > > > explaining how we handle all these concurrent events. And > > > only > > > > > > then, > > > > > > > > when > > > > > > > > > > we understand all implications, we should move to > > development > > > > > > stage. > > > > > > > > > > > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > > > > > Right > > > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > Good! Basically your orchestrator just takes some > > > > predefined > > > > > > > graph > > > > > > > > of > > > > > > > > > > > > distributed services to be invoked, calls them by > some > > > kind > > > > > of > > > > > > > RPC > > > > > > > > > and > > > > > > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible > for > > > > > > managing > > > > > > > > > > business > > > > > > > > > > > > > scenarios flows. Many nodes are involved in > > scenarios. > > > > They > > > > > > > > > exchange > > > > > > > > > > > data > > > > > > > > > > > > > and folow one another. If you acquinted with BPMN > > > > > framework, > > > > > > so > > > > > > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > > > > > > [hidden email] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing from > > > > > Microsoft > > > > > > or > > > > > > > > > your > > > > > > > > > > > > custom > > > > > > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers > which > > > > > fulfills > > > > > > > > > custom > > > > > > > > > > > > logic. > > > > > > > > > > > > > > > This servers compound oriented graph (BPMN > > process) > > > > > which > > > > > > > > > > > controlled > > > > > > > > > > > > by > > > > > > > > > > > > > > > Orchestrator. > > > > > > > > > > > > > > > For instance, *server1 *creates *variable A > > *with > > > > > value > > > > > > 1, > > > > > > > > > > > persists > > > > > > > > > > > > it > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > IGNITE cache and creates *variable B *and sends > > it > > > > to* > > > > > > > > server2. > > > > > > > > > > > *The > > > > > > > > > > > > > > > latests receives *variable B*, do some logic > with > > > it > > > > > and > > > > > > > > stores > > > > > > > > > > to > > > > > > > > > > > > > > IGNITE. > > > > > > > > > > > > > > > All the work made by both servers must be > > fulfilled > > > > in > > > > > > > *one* > > > > > > > > > > > > > transaction. > > > > > > > > > > > > > > > Because we need all information done, or > > > > > > > nothing(rollbacked). > > > > > > > > > The > > > > > > > > > > > > > > scenario > > > > > > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your > wrong > > > > > > solution > > > > > > > > for > > > > > > > > > > it. > > > > > > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV > < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts > > > transaction > > > > > in > > > > > > > one > > > > > > > > > > node, > > > > > > > > > > > > and > > > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > this transaction in another jvm node(or > > > rollback > > > > it > > > > > > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi > Vladykin < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for some > > > > > > simplistic > > > > > > > > > > > scenario, > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > ready > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > write many fault tolerance tests and make > > > sure > > > > > that > > > > > > > you > > > > > > > > > TXs > > > > > > > > > > > > work > > > > > > > > > > > > > > > > > gracefully > > > > > > > > > > > > > > > > > > in all modes in case of crashes. Also > make > > > sure > > > > > > that > > > > > > > we > > > > > > > > > do > > > > > > > > > > > not > > > > > > > > > > > > > have > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > performance drops after all your changes > in > > > > > > existing > > > > > > > > > > > > benchmarks. > > > > > > > > > > > > > > All > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > I don't believe these conditions will be > > met > > > > and > > > > > > your > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? Sending > TX > > > to > > > > > > > another > > > > > > > > > > node? > > > > > > > > > > > > The > > > > > > > > > > > > > > > > problem > > > > > > > > > > > > > > > > > > statement itself is already wrong. What > > > > business > > > > > > case > > > > > > > > you > > > > > > > > > > are > > > > > > > > > > > > > > trying > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > solve? I'm sure everything you need can > be > > > done > > > > > in > > > > > > a > > > > > > > > much > > > > > > > > > > > more > > > > > > > > > > > > > > simple > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY > > KUZNETSOV > > > < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better > solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi > > > Vladykin < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and > > > > deserializing > > > > > it > > > > > > > on > > > > > > > > > > > another > > > > > > > > > > > > > node > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > meaningless, because other nodes > > > > > participating > > > > > > in > > > > > > > > the > > > > > > > > > > TX > > > > > > > > > > > > have > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > the new coordinator. This will > require > > > > > protocol > > > > > > > > > > changes, > > > > > > > > > > > we > > > > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > have fault tolerance and performance > > > > issues. > > > > > > IMO > > > > > > > > the > > > > > > > > > > > whole > > > > > > > > > > > > > idea > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > > > > > > and it makes no sense to waste time > on > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY > > > > KUZNETSOV > > > > > < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState > > implememntation > > > > > > contains > > > > > > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, > Dmitriy > > > > > > Setrakyan > > > > > > > < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me > that > > > we > > > > > are > > > > > > > > > passing > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > > > > > around. Such object may contain > all > > > > sorts > > > > > > of > > > > > > > > > Ignite > > > > > > > > > > > > > > context. > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > > > > needs to be passed across, we > > should > > > > > > create a > > > > > > > > > > special > > > > > > > > > > > > > > > transfer > > > > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, > > > > ALEKSEY > > > > > > > > > KUZNETSOV > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of issues > > > > > preventing > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > > > > > > At first, After transaction > > > > > serialization > > > > > > > and > > > > > > > > > > > > > > > deserialization > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > > > > > > server, there is no txState. So > > im > > > > > going > > > > > > to > > > > > > > > put > > > > > > > > > > it > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized > > > > > transaction > > > > > > > > lacks > > > > > > > > > of > > > > > > > > > > > > > shared > > > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > > > > > > field at TransactionProxyImpl. > > > > Perhaps, > > > > > > it > > > > > > > > must > > > > > > > > > > be > > > > > > > > > > > > > > injected > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, > > > ALEKSEY > > > > > > > > KUZNETSOV > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and continuing > > > > > > transaction > > > > > > > > in > > > > > > > > > > > > > different > > > > > > > > > > > > > > > jvms > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > > > > serialization exception in > > > > > > > > writeExternalMeta > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > > > > > > writeExternal(ObjectOutput > > > > > > > > > > > > out) > > > > > > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be > > > serialized. > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25, > > > Alexey > > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to get > > what > > > > you > > > > > > > want, > > > > > > > > > > but I > > > > > > > > > > > > > have > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > > > > > > - What is the API for the > > > proposed > > > > > > > change? > > > > > > > > > In > > > > > > > > > > > your > > > > > > > > > > > > > > test, > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > > instance of transaction > created > > > on > > > > > > > > ignite(0) > > > > > > > > > to > > > > > > > > > > > the > > > > > > > > > > > > > > > ignite > > > > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > > > > ignite(1). This is obviously > > not > > > > > > possible > > > > > > > > in > > > > > > > > > a > > > > > > > > > > > > truly > > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > > > > > > - How will you synchronize > > cache > > > > > update > > > > > > > > > actions > > > > > > > > > > > and > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > > > > > > Say, you have one node that > > > decided > > > > > to > > > > > > > > > commit, > > > > > > > > > > > but > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > > > writing within this > > transaction. > > > > How > > > > > do > > > > > > > you > > > > > > > > > > make > > > > > > > > > > > > sure > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > > not call commit() and > > rollback() > > > > > > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > > > > > > - How do you make sure that > > > either > > > > > > > > commit() > > > > > > > > > or > > > > > > > > > > > > > > > rollback() > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 > > > Дмитрий > > > > > > Рябов > > > > > > > < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my > > > initial > > > > > > > > > > understanding > > > > > > > > > > > > was > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > > > > ownership from one node to > > > > another > > > > > > will > > > > > > > > be > > > > > > > > > > > > happened > > > > > > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > > > > > originating node is gone > > down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00 > > > > ALEKSEY > > > > > > > > > KUZNETSOV > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span > > transaction > > > > on > > > > > > > > multiple > > > > > > > > > > > > > threads, > > > > > > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > > > > > every node is able to > > > rollback, > > > > > or > > > > > > > > commit > > > > > > > > > > > > common > > > > > > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > > > > > > need to transfer tx > between > > > > nodes > > > > > > in > > > > > > > > > order > > > > > > > > > > to > > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > different node(in the > same > > > > jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в > > 15:20, > > > > > Alexey > > > > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you > > want a > > > > > > concept > > > > > > > > of > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > > > > one node to another? My > > > > initial > > > > > > > > > > > understanding > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > > > > > > to update keys in a > > > > transaction > > > > > > > from > > > > > > > > > > > multiple > > > > > > > > > > > > > > > threads > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 > > GMT+03:00 > > > > > > ALEKSEY > > > > > > > > > > > KUZNETSOV > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider > > > transaction > > > > > > > started > > > > > > > > in > > > > > > > > > > one > > > > > > > > > > > > > node, > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The following test > > > > describes > > > > > my > > > > > > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = > > > ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > > > > transactions = > > > > > > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, > > > > Integer> > > > > > > > cache > > > > > > > > = > > > > > > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > > > > > > transactions.txStart( > > > > > > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > > > > > > fut = > > > > > > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > ts = > > > > > > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", > > 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method > > > *ts.txStart(...)* > > > > > we > > > > > > > just > > > > > > > > > > > rebind > > > > > > > > > > > > > *tx* > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void > > > > > txStart(Transaction > > > > > > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > > > > > > cctx.tm > > ().reopenTx( > > > > > > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method *reopenTx* > we > > > > alter > > > > > > > > > > *threadMap* > > > > > > > > > > > > so > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think about > > it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. в > > > 22:38, > > > > > > Denis > > > > > > > > > > Magda < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the > > > rational > > > > > > > behind > > > > > > > > > this > > > > > > > > > > > and > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, > at > > > 3:19 > > > > > AM, > > > > > > > > > ALEKSEY > > > > > > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im > > designing > > > > > > > > distributed > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > node, and > continued > > > at > > > > > > other > > > > > > > > one. > > > > > > > > > > Has > > > > > > > > > > > > > > anybody > > > > > > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov > Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > -- > > > > > > > > > > *Best Regards,* > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > -- > > > > > > *Best Regards,* > > > > > > *Kuznetsov Aleksey* > > > > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > |
|
|
Hi, Igniters! I 've made implementation of transactions of non-single
coordinator. Here you can start transaction in one thread and commit it in another thread. Take a look on it. Give your thoughts on it. https://github.com/voipp/ignite/pull/10/commits/3a3d90aa6ac84f125e4c3ce4ced4f269a695ef45 пт, 17 мар. 2017 г. в 19:26, Sergi Vladykin <[hidden email]>: > You know better, go ahead! :) > > Sergi > > 2017-03-17 16:16 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > > > we've discovered several problems regarding your "accumulation" > > approach.These are > > > > 1. perfomance issues when transfering data from temporary cache to > > permanent one. Keep in mind big deal of concurent transactions in > > Service > > commiter > > 2. extreme memory load when keeping temporary cache in memory > > 3. As long as user is not acquainted with ignite, working with cache > > must be transparent for him. Keep this in mind. User's node can > evaluate > > logic with no transaction at all, so we should deal with both types of > > execution flow : transactional and non-transactional.Another one > > problem is > > transaction id support at the user node. We would have handled all > this > > issues and many more. > > 4. we cannot pessimistically lock entity. > > > > As a result, we decided to move on building distributed transaction. We > put > > aside your "accumulation" approach until we realize how to solve > > difficulties above . > > > > чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin <[hidden email]>: > > > > > The problem "How to run millions of entities, and millions of > operations > > on > > > a single Pentium3" is out of scope here. Do the math, plan capacity > > > reasonably. > > > > > > Sergi > > > > > > 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > > > > > > > hmm, If we have millions of entities, and millions of operations, > would > > > not > > > > this approache lead to memory overflow and perfomance degradation > > > > > > > > чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin < > [hidden email] > > >: > > > > > > > > > 1. Actually you have to check versions on all the values you have > > read > > > > > during the tx. > > > > > > > > > > For example if we have [k1 => v1, k2 => v2] and do: > > > > > > > > > > put(k1, get(k2) + 5) > > > > > > > > > > We have to remember the version for k2. This logic can be > relatively > > > > easily > > > > > encapsulated in a framework atop of Ignite. You need to implement > one > > > to > > > > > make all this stuff usable. > > > > > > > > > > 2. I suggest to avoid any locking here, because you easily will end > > up > > > > with > > > > > deadlocks. If you do not have too frequent updates for your keys, > > > > > optimistic approach will work just fine. > > > > > > > > > > Theoretically in the Committer Service you can start a thread for > the > > > > > lifetime of the whole distributed transaction, take a lock on the > key > > > > using > > > > > IgniteCache.lock(K key) before executing any Services, wait for all > > the > > > > > services to complete, execute optimistic commit in the same thread > > > while > > > > > keeping this lock and then release it. Notice that all the Ignite > > > > > transactions inside of all Services must be optimistic here to be > > able > > > to > > > > > read this locked key. > > > > > > > > > > But again I do not recommend you using this approach until you > have a > > > > > reliable deadlock avoidance scheme. > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV < > > [hidden email] > > > >: > > > > > > > > > > > Yeah, now i got it. > > > > > > There are some doubts on this approach > > > > > > 1) During optimistic commit phase, when you assure no one altered > > the > > > > > > original values, you must check versions of other dependent keys. > > How > > > > > could > > > > > > we obtain those keys(in an automative manner, of course) ? > > > > > > 2) How could we lock a key before some Service A introduce > changes? > > > So > > > > no > > > > > > other service is allowed to change this key-value?(sort of > > > pessimistic > > > > > > blocking) > > > > > > May be you know some implementations of such approach ? > > > > > > > > > > > > ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > Thank you very much for help. I will answer later. > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > All the services do not update key in place, but only generate > > new > > > > keys > > > > > > > augmented by otx and store the updated value in the same cache > + > > > > > remember > > > > > > > the keys and versions participating in the transaction in some > > > > separate > > > > > > > atomic cache. > > > > > > > > > > > > > > Follow this sequence of changes applied to cache contents by > each > > > > > > Service: > > > > > > > > > > > > > > Initial cache contents: > > > > > > > [k1 => v1] > > > > > > > [k2 => v2] > > > > > > > [k3 => v3] > > > > > > > > > > > > > > Cache contents after Service A: > > > > > > > [k1 => v1] > > > > > > > [k2 => v2] > > > > > > > [k3 => v3] > > > > > > > [k1x => v1a] > > > > > > > [k2x => v2a] > > > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2)] in some separate > > atomic > > > > > cache > > > > > > > > > > > > > > Cache contents after Service B: > > > > > > > [k1 => v1] > > > > > > > [k2 => v2] > > > > > > > [k3 => v3] > > > > > > > [k1x => v1a] > > > > > > > [k2x => v2ab] > > > > > > > [k3x => v3b] > > > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > > > separate > > > > > > > atomic cache > > > > > > > > > > > > > > Finally the Committer Service takes this map of updated keys > and > > > > their > > > > > > > versions from some separate atomic cache, starts Ignite > > transaction > > > > and > > > > > > > replaces all the values for k* keys to values taken from k*x > > keys. > > > > The > > > > > > > successful result must be the following: > > > > > > > > > > > > > > [k1 => v1a] > > > > > > > [k2 => v2ab] > > > > > > > [k3 => v3b] > > > > > > > [k1x => v1a] > > > > > > > [k2x => v2ab] > > > > > > > [k3x => v3b] > > > > > > > > > > > > > > + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > > > separate > > > > > > > atomic cache > > > > > > > > > > > > > > But Committer Service also has to check that no one updated the > > > > > original > > > > > > > values before us, because otherwise we can not give any > > > > serializability > > > > > > > guarantee for these distributed transactions. Here we may need > to > > > > check > > > > > > not > > > > > > > only versions of the updated keys, but also versions of any > other > > > > keys > > > > > > end > > > > > > > result depends on. > > > > > > > > > > > > > > After that Committer Service has to do a cleanup (may be > outside > > of > > > > the > > > > > > > committing tx) to come to the following final state: > > > > > > > > > > > > > > [k1 => v1a] > > > > > > > [k2 => v2ab] > > > > > > > [k3 => v3b] > > > > > > > > > > > > > > Makes sense? > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > > > > [hidden email] > > > > > >: > > > > > > > > > > > > > > > - what do u mean by saying " > > > > > > > > *in a single transaction checks value versions for all the > old > > > > values > > > > > > > > and replaces them with calculated new ones *"? Every time > > you > > > > > > change > > > > > > > > value(in some service), you store it to *some special > atomic > > > > > cache* > > > > > > , > > > > > > > so > > > > > > > > when all services ceased working, Service commiter got a > > > values > > > > > with > > > > > > > the > > > > > > > > last versions. > > > > > > > > - After "*does cleanup of temporary keys and values*" > > Service > > > > > > commiter > > > > > > > > persists them into permanent store, isn't it ? > > > > > > > > - I cant grasp your though, you say "*in case of version > > > > mismatch > > > > > or > > > > > > > TX > > > > > > > > timeout just rollbacks*". But what versions would it > match? > > > > > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > Ok, here is what you actually need to implement at the > > > > application > > > > > > > level. > > > > > > > > > > > > > > > > > > Lets say we have to call 2 services in the following order: > > > > > > > > > - Service A: wants to update keys [k1 => v1, k2 => v2] > to > > > > [k1 > > > > > => > > > > > > > > v1a, > > > > > > > > > k2 => v2a] > > > > > > > > > - Service B: wants to update keys [k2 => v2a, k3 => v3] > to > > > [k2 > > > > > => > > > > > > > > v2ab, > > > > > > > > > k3 => v3b] > > > > > > > > > > > > > > > > > > The change > > > > > > > > > from [ k1 => v1, k2 => v2, k3 => v3 ] > > > > > > > > > to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > > > > > > > > must happen in a single transaction. > > > > > > > > > > > > > > > > > > > > > > > > > > > Optimistic protocol to solve this: > > > > > > > > > > > > > > > > > > Each cache key must have a field `otx`, which is a unique > > > > > > orchestrator > > > > > > > TX > > > > > > > > > identifier - it must be a parameter passed to all the > > services. > > > > If > > > > > > > `otx` > > > > > > > > is > > > > > > > > > set to some value it means that it is an intermediate key > and > > > is > > > > > > > visible > > > > > > > > > only inside of some transaction, for the finalized key > `otx` > > > must > > > > > be > > > > > > > > null - > > > > > > > > > it means the key is committed and visible for everyone. > > > > > > > > > > > > > > > > > > Each cache value must have a field `ver` which is a version > > of > > > > that > > > > > > > > value. > > > > > > > > > > > > > > > > > > For both fields (`otx` and `ver`) the safest way is to use > > > UUID. > > > > > > > > > > > > > > > > > > Workflow is the following: > > > > > > > > > > > > > > > > > > Orchestrator starts the distributed transaction with `otx` > = > > x > > > > and > > > > > > > passes > > > > > > > > > this parameter to all the services. > > > > > > > > > > > > > > > > > > Service A: > > > > > > > > > - does some computations > > > > > > > > > - stores [k1x => v1a, k2x => v2a] with TTL = Za > > > > > > > > > where > > > > > > > > > Za - left time from max Orchestrator TX duration > > > after > > > > > > > Service > > > > > > > > A > > > > > > > > > end > > > > > > > > > k1x, k2x - new temporary keys with field `otx` = > x > > > > > > > > > v2a has updated version `ver` > > > > > > > > > - returns a set of updated keys and all the old versions > to > > > the > > > > > > > > > orchestrator > > > > > > > > > or just stores it in some special atomic cache like > > > > > > > > > [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > > > > > > > > > > > > > > > > > Service B: > > > > > > > > > - retrieves the updated value k2x => v2a because it knows > > > `otx` > > > > = > > > > > x > > > > > > > > > - does computations > > > > > > > > > - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > > > > > > > > - updates the set of updated keys like [x => (k1 -> ver1, > k2 > > > -> > > > > > > ver2, > > > > > > > k3 > > > > > > > > > -> ver3)] TTL = Zb > > > > > > > > > > > > > > > > > > Service Committer (may be embedded into Orchestrator): > > > > > > > > > - takes all the updated keys and versions for `otx` = x > > > > > > > > > [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > > > > > > > > - in a single transaction checks value versions for all > the > > > old > > > > > > values > > > > > > > > > and replaces them with calculated new ones > > > > > > > > > - does cleanup of temporary keys and values > > > > > > > > > - in case of version mismatch or TX timeout just rollbacks > > and > > > > > > signals > > > > > > > > > to Orchestrator to restart the job with new `otx` > > > > > > > > > > > > > > > > > > PROFIT!! > > > > > > > > > > > > > > > > > > This approach even allows you to run independent parts of > the > > > > graph > > > > > > in > > > > > > > > > parallel (with TX transfer you will always run only one at > a > > > > time). > > > > > > > Also > > > > > > > > it > > > > > > > > > does not require inventing any special fault tolerance > > technics > > > > > > because > > > > > > > > > Ignite caches are already fault tolerant and all the > > > intermediate > > > > > > > results > > > > > > > > > are virtually invisible and stored with TTL, thus in case > of > > > any > > > > > > crash > > > > > > > > you > > > > > > > > > will not have inconsistent state or garbage. > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > [hidden email] > > > > > > > >: > > > > > > > > > > > > > > > > > > > Okay, we are open for proposals on business task. I mean, > > we > > > > can > > > > > > make > > > > > > > > use > > > > > > > > > > of some other thing, not distributed transaction. Not > > > > transaction > > > > > > > yet. > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > > > > > [hidden email] > > > > > > >: > > > > > > > > > > > > > > > > > > > > > IMO the use case makes sense. However, as Sergi already > > > > > > mentioned, > > > > > > > > the > > > > > > > > > > > problem is far more complex, than simply passing TX > state > > > > over > > > > > a > > > > > > > > wire. > > > > > > > > > > Most > > > > > > > > > > > probably a kind of coordinator will be required still > to > > > > manage > > > > > > all > > > > > > > > > kinds > > > > > > > > > > > of failures. This task should be started with clean > > design > > > > > > proposal > > > > > > > > > > > explaining how we handle all these concurrent events. > And > > > > only > > > > > > > then, > > > > > > > > > when > > > > > > > > > > > we understand all implications, we should move to > > > development > > > > > > > stage. > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > > > > > > > > > > [hidden email]> wrote: > > > > > > > > > > > > > > > > > > > > > > > Right > > > > > > > > > > > > > > > > > > > > > > > > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Good! Basically your orchestrator just takes some > > > > > predefined > > > > > > > > graph > > > > > > > > > of > > > > > > > > > > > > > distributed services to be invoked, calls them by > > some > > > > kind > > > > > > of > > > > > > > > RPC > > > > > > > > > > and > > > > > > > > > > > > > passes the needed parameters between them, right? > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > [hidden email] > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > orchestrator is a custom thing. He is responsible > > for > > > > > > > managing > > > > > > > > > > > business > > > > > > > > > > > > > > scenarios flows. Many nodes are involved in > > > scenarios. > > > > > They > > > > > > > > > > exchange > > > > > > > > > > > > data > > > > > > > > > > > > > > and folow one another. If you acquinted with BPMN > > > > > > framework, > > > > > > > so > > > > > > > > > > > > > > orchestrator is like bpmn engine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > > > > > > > > [hidden email] > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What is Orchestrator for you? Is it a thing > from > > > > > > Microsoft > > > > > > > or > > > > > > > > > > your > > > > > > > > > > > > > custom > > > > > > > > > > > > > > > in-house software? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Fine. Let's say we've got multiple servers > > which > > > > > > fulfills > > > > > > > > > > custom > > > > > > > > > > > > > logic. > > > > > > > > > > > > > > > > This servers compound oriented graph (BPMN > > > process) > > > > > > which > > > > > > > > > > > > controlled > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > Orchestrator. > > > > > > > > > > > > > > > > For instance, *server1 *creates *variable A > > > *with > > > > > > value > > > > > > > 1, > > > > > > > > > > > > persists > > > > > > > > > > > > > it > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > IGNITE cache and creates *variable B *and > sends > > > it > > > > > to* > > > > > > > > > server2. > > > > > > > > > > > > *The > > > > > > > > > > > > > > > > latests receives *variable B*, do some logic > > with > > > > it > > > > > > and > > > > > > > > > stores > > > > > > > > > > > to > > > > > > > > > > > > > > > IGNITE. > > > > > > > > > > > > > > > > All the work made by both servers must be > > > fulfilled > > > > > in > > > > > > > > *one* > > > > > > > > > > > > > > transaction. > > > > > > > > > > > > > > > > Because we need all information done, or > > > > > > > > nothing(rollbacked). > > > > > > > > > > The > > > > > > > > > > > > > > > scenario > > > > > > > > > > > > > > > > is managed by orchestrator. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, it is not a business case, it is your > > wrong > > > > > > > solution > > > > > > > > > for > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > Lets try again, what is the business case? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY > KUZNETSOV > > < > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The case is the following, One starts > > > > transaction > > > > > > in > > > > > > > > one > > > > > > > > > > > node, > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > > this transaction in another jvm node(or > > > > rollback > > > > > it > > > > > > > > > > > remotely). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi > > Vladykin < > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because even if you make it work for > some > > > > > > > simplistic > > > > > > > > > > > > scenario, > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > > ready > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > write many fault tolerance tests and > make > > > > sure > > > > > > that > > > > > > > > you > > > > > > > > > > TXs > > > > > > > > > > > > > work > > > > > > > > > > > > > > > > > > gracefully > > > > > > > > > > > > > > > > > > > in all modes in case of crashes. Also > > make > > > > sure > > > > > > > that > > > > > > > > we > > > > > > > > > > do > > > > > > > > > > > > not > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > > performance drops after all your > changes > > in > > > > > > > existing > > > > > > > > > > > > > benchmarks. > > > > > > > > > > > > > > > All > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > I don't believe these conditions will > be > > > met > > > > > and > > > > > > > your > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Better solution to what problem? > Sending > > TX > > > > to > > > > > > > > another > > > > > > > > > > > node? > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > problem > > > > > > > > > > > > > > > > > > > statement itself is already wrong. What > > > > > business > > > > > > > case > > > > > > > > > you > > > > > > > > > > > are > > > > > > > > > > > > > > > trying > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > solve? I'm sure everything you need can > > be > > > > done > > > > > > in > > > > > > > a > > > > > > > > > much > > > > > > > > > > > > more > > > > > > > > > > > > > > > simple > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > efficient way at the application level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY > > > KUZNETSOV > > > > < > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why wrong ? You know the better > > solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi > > > > Vladykin < > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just serializing TX object and > > > > > deserializing > > > > > > it > > > > > > > > on > > > > > > > > > > > > another > > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > meaningless, because other nodes > > > > > > participating > > > > > > > in > > > > > > > > > the > > > > > > > > > > > TX > > > > > > > > > > > > > have > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > > the new coordinator. This will > > require > > > > > > protocol > > > > > > > > > > > changes, > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > have fault tolerance and > performance > > > > > issues. > > > > > > > IMO > > > > > > > > > the > > > > > > > > > > > > whole > > > > > > > > > > > > > > idea > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > wrong > > > > > > > > > > > > > > > > > > > > > and it makes no sense to waste time > > on > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY > > > > > KUZNETSOV > > > > > > < > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactionState > > > implememntation > > > > > > > contains > > > > > > > > > > > > > > > IgniteTxEntry's > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > supposed to be transferable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32, > > Dmitriy > > > > > > > Setrakyan > > > > > > > > < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It sounds a little scary to me > > that > > > > we > > > > > > are > > > > > > > > > > passing > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > > > > > > around. Such object may contain > > all > > > > > sorts > > > > > > > of > > > > > > > > > > Ignite > > > > > > > > > > > > > > > context. > > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > > > > > needs to be passed across, we > > > should > > > > > > > create a > > > > > > > > > > > special > > > > > > > > > > > > > > > > transfer > > > > > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > this case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 > AM, > > > > > ALEKSEY > > > > > > > > > > KUZNETSOV > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > well, there a couple of > issues > > > > > > preventing > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > proceeding. > > > > > > > > > > > > > > > > > > > > > > > > At first, After transaction > > > > > > serialization > > > > > > > > and > > > > > > > > > > > > > > > > deserialization > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > remote > > > > > > > > > > > > > > > > > > > > > > > > server, there is no txState. > So > > > im > > > > > > going > > > > > > > to > > > > > > > > > put > > > > > > > > > > > it > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > writeExternal()\readExternal() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The last one is Deserialized > > > > > > transaction > > > > > > > > > lacks > > > > > > > > > > of > > > > > > > > > > > > > > shared > > > > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > > > > > > > field at > TransactionProxyImpl. > > > > > Perhaps, > > > > > > > it > > > > > > > > > must > > > > > > > > > > > be > > > > > > > > > > > > > > > injected > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > > GridResourceProcessor ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27, > > > > ALEKSEY > > > > > > > > > KUZNETSOV > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > while starting and > continuing > > > > > > > transaction > > > > > > > > > in > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > jvms > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > > > > > serialization exception in > > > > > > > > > writeExternalMeta > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Override public void > > > > > > > > > > > writeExternal(ObjectOutput > > > > > > > > > > > > > out) > > > > > > > > > > > > > > > > > throws > > > > > > > > > > > > > > > > > > > > > > > IOException > > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > > writeExternalMeta(out); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some meta is cannot be > > > > serialized. > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в > 17:25, > > > > Alexey > > > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think I am starting to > get > > > what > > > > > you > > > > > > > > want, > > > > > > > > > > > but I > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > > > concerns: > > > > > > > > > > > > > > > > > > > > > > > > > - What is the API for the > > > > proposed > > > > > > > > change? > > > > > > > > > > In > > > > > > > > > > > > your > > > > > > > > > > > > > > > test, > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > pass > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > > > instance of transaction > > created > > > > on > > > > > > > > > ignite(0) > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > > > ignite > > > > > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > > > > > ignite(1). This is > obviously > > > not > > > > > > > possible > > > > > > > > > in > > > > > > > > > > a > > > > > > > > > > > > > truly > > > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > > > (multi-jvm) environment. > > > > > > > > > > > > > > > > > > > > > > > > > - How will you synchronize > > > cache > > > > > > update > > > > > > > > > > actions > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > commit? > > > > > > > > > > > > > > > > > > > > > > > > > Say, you have one node that > > > > decided > > > > > > to > > > > > > > > > > commit, > > > > > > > > > > > > but > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > > > > writing within this > > > transaction. > > > > > How > > > > > > do > > > > > > > > you > > > > > > > > > > > make > > > > > > > > > > > > > sure > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > > > not call commit() and > > > rollback() > > > > > > > > > > > simultaneously? > > > > > > > > > > > > > > > > > > > > > > > > > - How do you make sure > that > > > > either > > > > > > > > > commit() > > > > > > > > > > or > > > > > > > > > > > > > > > > rollback() > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > called > > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > > > originator failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 > > > > Дмитрий > > > > > > > Рябов > > > > > > > > < > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, heh, my > > > > initial > > > > > > > > > > > understanding > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > > > > > ownership from one node > to > > > > > another > > > > > > > will > > > > > > > > > be > > > > > > > > > > > > > happened > > > > > > > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > > > > > > originating node is gone > > > down. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:36 > GMT+03:00 > > > > > ALEKSEY > > > > > > > > > > KUZNETSOV > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Im aiming to span > > > transaction > > > > > on > > > > > > > > > multiple > > > > > > > > > > > > > > threads, > > > > > > > > > > > > > > > > > nodes, > > > > > > > > > > > > > > > > > > > > > > > jvms(soon). > > > > > > > > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > > > > > > every node is able to > > > > rollback, > > > > > > or > > > > > > > > > commit > > > > > > > > > > > > > common > > > > > > > > > > > > > > > > > > > > transaction.It > > > > > > > > > > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > > > > > > > > > > > > up i > > > > > > > > > > > > > > > > > > > > > > > > > > > need to transfer tx > > between > > > > > nodes > > > > > > > in > > > > > > > > > > order > > > > > > > > > > > to > > > > > > > > > > > > > > > commit > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > different node(in the > > same > > > > > jvm). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 10 мар. 2017 г. в > > > 15:20, > > > > > > Alexey > > > > > > > > > > > > Goncharuk < > > > > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Aleksey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you mean that you > > > want a > > > > > > > concept > > > > > > > > > of > > > > > > > > > > > > > > > transferring > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > tx > > > > > > > > > > > > > > > > > > > > > > > ownership > > > > > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > > > > > one node to another? > My > > > > > initial > > > > > > > > > > > > understanding > > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > > > > > > > to update keys in a > > > > > transaction > > > > > > > > from > > > > > > > > > > > > multiple > > > > > > > > > > > > > > > > threads > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --AG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-03-10 15:01 > > > GMT+03:00 > > > > > > > ALEKSEY > > > > > > > > > > > > KUZNETSOV > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well. Consider > > > > transaction > > > > > > > > started > > > > > > > > > in > > > > > > > > > > > one > > > > > > > > > > > > > > node, > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > continued > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The following test > > > > > describes > > > > > > my > > > > > > > > > idea: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ignite ignite1 = > > > > ignite(0); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > > > > > transactions = > > > > > > > > > > > > > > > > > > > ignite1.transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteCache<String, > > > > > Integer> > > > > > > > > cache > > > > > > > > > = > > > > > > > > > > > > > > > > > > > > > > ignite1.getOrCreateCache(" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > testCache"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Transaction tx = > > > > > > > > > > transactions.txStart( > > > > > > > > > > > > > > > > concurrency, > > > > > > > > > > > > > > > > > > > > > > isolation); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key1", > 1); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key2", > 2); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.stop(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean> > > > > > > > > fut = > > > > > > > > > > > > > > > > > > > > > GridTestUtils.runAsync(() > > > > > > > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IgniteTransactions > > > > ts = > > > > > > > > > > > > > > > > > ignite(1).transactions(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertNull(ts.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > > > > > TransactionState.STOPPED, > > > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ts.txStart(tx); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals(TransactionState.ACTIVE, > > > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cache.put("key3", > > > 3); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertTrue(cache.remove("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tx.commit(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > return true; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fut.get(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals( > > > > > > > > > > > > > > TransactionState.COMMITTED, > > > > > > > > > > > > > > > > > > > > > tx.state()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)1, > > > > > > > > > > > > > > > > > > (long)cache.get("key1")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertEquals((long)3, > > > > > > > > > > > > > > > > > > (long)cache.get("key3")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Assert.assertFalse(cache. > > > > > > > > > > > > > > containsKey("key2")); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method > > > > *ts.txStart(...)* > > > > > > we > > > > > > > > just > > > > > > > > > > > > rebind > > > > > > > > > > > > > > *tx* > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > > > > > > thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public void > > > > > > txStart(Transaction > > > > > > > > > tx) { > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TransactionProxyImpl > > > > > > > > > > > > transactionProxy = > > > > > > > > > > > > > > > > > > > > > > > > > (TransactionProxyImpl)tx; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cctx.tm > > > ().reopenTx( > > > > > > > > > > > > > > transactionProxy.tx()); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > transactionProxy. > > > > > > > > > > > > bindToCurrentThread(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In method > *reopenTx* > > we > > > > > alter > > > > > > > > > > > *threadMap* > > > > > > > > > > > > > so > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > binds > > > > > > > > > > > > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to current thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How do u think > about > > > it ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 7 мар. 2017 г. > в > > > > 22:38, > > > > > > > Denis > > > > > > > > > > > Magda < > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please share the > > > > rational > > > > > > > > behind > > > > > > > > > > this > > > > > > > > > > > > and > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > thoughts, > > > > > > > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > > > > > > > ideas > > > > > > > > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > have in mind. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > — > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 7, 2017, > > at > > > > 3:19 > > > > > > AM, > > > > > > > > > > ALEKSEY > > > > > > > > > > > > > > > > KUZNETSOV < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! Im > > > designing > > > > > > > > > distributed > > > > > > > > > > > > > > > transaction > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > node, and > > continued > > > > at > > > > > > > other > > > > > > > > > one. > > > > > > > > > > > Has > > > > > > > > > > > > > > > anybody > > > > > > > > > > > > > > > > > > > > thoughts > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov > > Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best Regards,* > > > > > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > > > > > > > -- > > > > > > > > *Best Regards,* > > > > > > > > *Kuznetsov Aleksey* > > > > > > > > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > > *Best Regards,* *Kuznetsov Aleksey* |
|
|
Aleksey,
Do you have a ticket for this? Could you briefly list what exactly was done and how the things work. — Denis > On Mar 28, 2017, at 8:32 AM, ALEKSEY KUZNETSOV <[hidden email]> wrote: > > Hi, Igniters! I 've made implementation of transactions of non-single > coordinator. Here you can start transaction in one thread and commit it in > another thread. > Take a look on it. Give your thoughts on it. > > https://github.com/voipp/ignite/pull/10/commits/3a3d90aa6ac84f125e4c3ce4ced4f269a695ef45 > > пт, 17 мар. 2017 г. в 19:26, Sergi Vladykin <[hidden email]>: > >> You know better, go ahead! :) >> >> Sergi >> >> 2017-03-17 16:16 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: >> >>> we've discovered several problems regarding your "accumulation" >>> approach.These are >>> >>> 1. perfomance issues when transfering data from temporary cache to >>> permanent one. Keep in mind big deal of concurent transactions in >>> Service >>> commiter >>> 2. extreme memory load when keeping temporary cache in memory >>> 3. As long as user is not acquainted with ignite, working with cache >>> must be transparent for him. Keep this in mind. User's node can >> evaluate >>> logic with no transaction at all, so we should deal with both types of >>> execution flow : transactional and non-transactional.Another one >>> problem is >>> transaction id support at the user node. We would have handled all >> this >>> issues and many more. >>> 4. we cannot pessimistically lock entity. >>> >>> As a result, we decided to move on building distributed transaction. We >> put >>> aside your "accumulation" approach until we realize how to solve >>> difficulties above . >>> >>> чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin <[hidden email]>: >>> >>>> The problem "How to run millions of entities, and millions of >> operations >>> on >>>> a single Pentium3" is out of scope here. Do the math, plan capacity >>>> reasonably. >>>> >>>> Sergi >>>> >>>> 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] >>> : >>>> >>>>> hmm, If we have millions of entities, and millions of operations, >> would >>>> not >>>>> this approache lead to memory overflow and perfomance degradation >>>>> >>>>> чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin < >> [hidden email] >>>> : >>>>> >>>>>> 1. Actually you have to check versions on all the values you have >>> read >>>>>> during the tx. >>>>>> >>>>>> For example if we have [k1 => v1, k2 => v2] and do: >>>>>> >>>>>> put(k1, get(k2) + 5) >>>>>> >>>>>> We have to remember the version for k2. This logic can be >> relatively >>>>> easily >>>>>> encapsulated in a framework atop of Ignite. You need to implement >> one >>>> to >>>>>> make all this stuff usable. >>>>>> >>>>>> 2. I suggest to avoid any locking here, because you easily will end >>> up >>>>> with >>>>>> deadlocks. If you do not have too frequent updates for your keys, >>>>>> optimistic approach will work just fine. >>>>>> >>>>>> Theoretically in the Committer Service you can start a thread for >> the >>>>>> lifetime of the whole distributed transaction, take a lock on the >> key >>>>> using >>>>>> IgniteCache.lock(K key) before executing any Services, wait for all >>> the >>>>>> services to complete, execute optimistic commit in the same thread >>>> while >>>>>> keeping this lock and then release it. Notice that all the Ignite >>>>>> transactions inside of all Services must be optimistic here to be >>> able >>>> to >>>>>> read this locked key. >>>>>> >>>>>> But again I do not recommend you using this approach until you >> have a >>>>>> reliable deadlock avoidance scheme. >>>>>> >>>>>> Sergi >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV < >>> [hidden email] >>>>> : >>>>>> >>>>>>> Yeah, now i got it. >>>>>>> There are some doubts on this approach >>>>>>> 1) During optimistic commit phase, when you assure no one altered >>> the >>>>>>> original values, you must check versions of other dependent keys. >>> How >>>>>> could >>>>>>> we obtain those keys(in an automative manner, of course) ? >>>>>>> 2) How could we lock a key before some Service A introduce >> changes? >>>> So >>>>> no >>>>>>> other service is allowed to change this key-value?(sort of >>>> pessimistic >>>>>>> blocking) >>>>>>> May be you know some implementations of such approach ? >>>>>>> >>>>>>> ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < >>>>> [hidden email] >>>>>>> : >>>>>>> >>>>>>>> Thank you very much for help. I will answer later. >>>>>>>> >>>>>>>> ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < >>>>> [hidden email] >>>>>>> : >>>>>>>> >>>>>>>> All the services do not update key in place, but only generate >>> new >>>>> keys >>>>>>>> augmented by otx and store the updated value in the same cache >> + >>>>>> remember >>>>>>>> the keys and versions participating in the transaction in some >>>>> separate >>>>>>>> atomic cache. >>>>>>>> >>>>>>>> Follow this sequence of changes applied to cache contents by >> each >>>>>>> Service: >>>>>>>> >>>>>>>> Initial cache contents: >>>>>>>> [k1 => v1] >>>>>>>> [k2 => v2] >>>>>>>> [k3 => v3] >>>>>>>> >>>>>>>> Cache contents after Service A: >>>>>>>> [k1 => v1] >>>>>>>> [k2 => v2] >>>>>>>> [k3 => v3] >>>>>>>> [k1x => v1a] >>>>>>>> [k2x => v2a] >>>>>>>> >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2)] in some separate >>> atomic >>>>>> cache >>>>>>>> >>>>>>>> Cache contents after Service B: >>>>>>>> [k1 => v1] >>>>>>>> [k2 => v2] >>>>>>>> [k3 => v3] >>>>>>>> [k1x => v1a] >>>>>>>> [k2x => v2ab] >>>>>>>> [k3x => v3b] >>>>>>>> >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some >>>>> separate >>>>>>>> atomic cache >>>>>>>> >>>>>>>> Finally the Committer Service takes this map of updated keys >> and >>>>> their >>>>>>>> versions from some separate atomic cache, starts Ignite >>> transaction >>>>> and >>>>>>>> replaces all the values for k* keys to values taken from k*x >>> keys. >>>>> The >>>>>>>> successful result must be the following: >>>>>>>> >>>>>>>> [k1 => v1a] >>>>>>>> [k2 => v2ab] >>>>>>>> [k3 => v3b] >>>>>>>> [k1x => v1a] >>>>>>>> [k2x => v2ab] >>>>>>>> [k3x => v3b] >>>>>>>> >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some >>>>> separate >>>>>>>> atomic cache >>>>>>>> >>>>>>>> But Committer Service also has to check that no one updated the >>>>>> original >>>>>>>> values before us, because otherwise we can not give any >>>>> serializability >>>>>>>> guarantee for these distributed transactions. Here we may need >> to >>>>> check >>>>>>> not >>>>>>>> only versions of the updated keys, but also versions of any >> other >>>>> keys >>>>>>> end >>>>>>>> result depends on. >>>>>>>> >>>>>>>> After that Committer Service has to do a cleanup (may be >> outside >>> of >>>>> the >>>>>>>> committing tx) to come to the following final state: >>>>>>>> >>>>>>>> [k1 => v1a] >>>>>>>> [k2 => v2ab] >>>>>>>> [k3 => v3b] >>>>>>>> >>>>>>>> Makes sense? >>>>>>>> >>>>>>>> Sergi >>>>>>>> >>>>>>>> >>>>>>>> 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < >>>>> [hidden email] >>>>>>> : >>>>>>>> >>>>>>>>> - what do u mean by saying " >>>>>>>>> *in a single transaction checks value versions for all the >> old >>>>> values >>>>>>>>> and replaces them with calculated new ones *"? Every time >>> you >>>>>>> change >>>>>>>>> value(in some service), you store it to *some special >> atomic >>>>>> cache* >>>>>>> , >>>>>>>> so >>>>>>>>> when all services ceased working, Service commiter got a >>>> values >>>>>> with >>>>>>>> the >>>>>>>>> last versions. >>>>>>>>> - After "*does cleanup of temporary keys and values*" >>> Service >>>>>>> commiter >>>>>>>>> persists them into permanent store, isn't it ? >>>>>>>>> - I cant grasp your though, you say "*in case of version >>>>> mismatch >>>>>> or >>>>>>>> TX >>>>>>>>> timeout just rollbacks*". But what versions would it >> match? >>>>>>>>> >>>>>>>>> >>>>>>>>> ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < >>>>>> [hidden email] >>>>>>>> : >>>>>>>>> >>>>>>>>>> Ok, here is what you actually need to implement at the >>>>> application >>>>>>>> level. >>>>>>>>>> >>>>>>>>>> Lets say we have to call 2 services in the following order: >>>>>>>>>> - Service A: wants to update keys [k1 => v1, k2 => v2] >> to >>>>> [k1 >>>>>> => >>>>>>>>> v1a, >>>>>>>>>> k2 => v2a] >>>>>>>>>> - Service B: wants to update keys [k2 => v2a, k3 => v3] >> to >>>> [k2 >>>>>> => >>>>>>>>> v2ab, >>>>>>>>>> k3 => v3b] >>>>>>>>>> >>>>>>>>>> The change >>>>>>>>>> from [ k1 => v1, k2 => v2, k3 => v3 ] >>>>>>>>>> to [ k1 => v1a, k2 => v2ab, k3 => v3b ] >>>>>>>>>> must happen in a single transaction. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Optimistic protocol to solve this: >>>>>>>>>> >>>>>>>>>> Each cache key must have a field `otx`, which is a unique >>>>>>> orchestrator >>>>>>>> TX >>>>>>>>>> identifier - it must be a parameter passed to all the >>> services. >>>>> If >>>>>>>> `otx` >>>>>>>>> is >>>>>>>>>> set to some value it means that it is an intermediate key >> and >>>> is >>>>>>>> visible >>>>>>>>>> only inside of some transaction, for the finalized key >> `otx` >>>> must >>>>>> be >>>>>>>>> null - >>>>>>>>>> it means the key is committed and visible for everyone. >>>>>>>>>> >>>>>>>>>> Each cache value must have a field `ver` which is a version >>> of >>>>> that >>>>>>>>> value. >>>>>>>>>> >>>>>>>>>> For both fields (`otx` and `ver`) the safest way is to use >>>> UUID. >>>>>>>>>> >>>>>>>>>> Workflow is the following: >>>>>>>>>> >>>>>>>>>> Orchestrator starts the distributed transaction with `otx` >> = >>> x >>>>> and >>>>>>>> passes >>>>>>>>>> this parameter to all the services. >>>>>>>>>> >>>>>>>>>> Service A: >>>>>>>>>> - does some computations >>>>>>>>>> - stores [k1x => v1a, k2x => v2a] with TTL = Za >>>>>>>>>> where >>>>>>>>>> Za - left time from max Orchestrator TX duration >>>> after >>>>>>>> Service >>>>>>>>> A >>>>>>>>>> end >>>>>>>>>> k1x, k2x - new temporary keys with field `otx` = >> x >>>>>>>>>> v2a has updated version `ver` >>>>>>>>>> - returns a set of updated keys and all the old versions >> to >>>> the >>>>>>>>>> orchestrator >>>>>>>>>> or just stores it in some special atomic cache like >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2)] TTL = Za >>>>>>>>>> >>>>>>>>>> Service B: >>>>>>>>>> - retrieves the updated value k2x => v2a because it knows >>>> `otx` >>>>> = >>>>>> x >>>>>>>>>> - does computations >>>>>>>>>> - stores [k2x => v2ab, k3x => v3b] TTL = Zb >>>>>>>>>> - updates the set of updated keys like [x => (k1 -> ver1, >> k2 >>>> -> >>>>>>> ver2, >>>>>>>> k3 >>>>>>>>>> -> ver3)] TTL = Zb >>>>>>>>>> >>>>>>>>>> Service Committer (may be embedded into Orchestrator): >>>>>>>>>> - takes all the updated keys and versions for `otx` = x >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] >>>>>>>>>> - in a single transaction checks value versions for all >> the >>>> old >>>>>>> values >>>>>>>>>> and replaces them with calculated new ones >>>>>>>>>> - does cleanup of temporary keys and values >>>>>>>>>> - in case of version mismatch or TX timeout just rollbacks >>> and >>>>>>> signals >>>>>>>>>> to Orchestrator to restart the job with new `otx` >>>>>>>>>> >>>>>>>>>> PROFIT!! >>>>>>>>>> >>>>>>>>>> This approach even allows you to run independent parts of >> the >>>>> graph >>>>>>> in >>>>>>>>>> parallel (with TX transfer you will always run only one at >> a >>>>> time). >>>>>>>> Also >>>>>>>>> it >>>>>>>>>> does not require inventing any special fault tolerance >>> technics >>>>>>> because >>>>>>>>>> Ignite caches are already fault tolerant and all the >>>> intermediate >>>>>>>> results >>>>>>>>>> are virtually invisible and stored with TTL, thus in case >> of >>>> any >>>>>>> crash >>>>>>>>> you >>>>>>>>>> will not have inconsistent state or garbage. >>>>>>>>>> >>>>>>>>>> Sergi >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < >>>>>>> [hidden email] >>>>>>>>> : >>>>>>>>>> >>>>>>>>>>> Okay, we are open for proposals on business task. I mean, >>> we >>>>> can >>>>>>> make >>>>>>>>> use >>>>>>>>>>> of some other thing, not distributed transaction. Not >>>>> transaction >>>>>>>> yet. >>>>>>>>>>> >>>>>>>>>>> ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < >>>>>> [hidden email] >>>>>>>> : >>>>>>>>>>> >>>>>>>>>>>> IMO the use case makes sense. However, as Sergi already >>>>>>> mentioned, >>>>>>>>> the >>>>>>>>>>>> problem is far more complex, than simply passing TX >> state >>>>> over >>>>>> a >>>>>>>>> wire. >>>>>>>>>>> Most >>>>>>>>>>>> probably a kind of coordinator will be required still >> to >>>>> manage >>>>>>> all >>>>>>>>>> kinds >>>>>>>>>>>> of failures. This task should be started with clean >>> design >>>>>>> proposal >>>>>>>>>>>> explaining how we handle all these concurrent events. >> And >>>>> only >>>>>>>> then, >>>>>>>>>> when >>>>>>>>>>>> we understand all implications, we should move to >>>> development >>>>>>>> stage. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < >>>>>>>>>>>> [hidden email]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Right >>>>>>>>>>>>> >>>>>>>>>>>>> ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < >>>>>>>>>> [hidden email] >>>>>>>>>>>> : >>>>>>>>>>>>> >>>>>>>>>>>>>> Good! Basically your orchestrator just takes some >>>>>> predefined >>>>>>>>> graph >>>>>>>>>> of >>>>>>>>>>>>>> distributed services to be invoked, calls them by >>> some >>>>> kind >>>>>>> of >>>>>>>>> RPC >>>>>>>>>>> and >>>>>>>>>>>>>> passes the needed parameters between them, right? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sergi >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < >>>>>>>>>>> [hidden email] >>>>>>>>>>>>> : >>>>>>>>>>>>>> >>>>>>>>>>>>>>> orchestrator is a custom thing. He is responsible >>> for >>>>>>>> managing >>>>>>>>>>>> business >>>>>>>>>>>>>>> scenarios flows. Many nodes are involved in >>>> scenarios. >>>>>> They >>>>>>>>>>> exchange >>>>>>>>>>>>> data >>>>>>>>>>>>>>> and folow one another. If you acquinted with BPMN >>>>>>> framework, >>>>>>>> so >>>>>>>>>>>>>>> orchestrator is like bpmn engine. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < >>>>>>>>>> [hidden email] >>>>>>>>>>>> : >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What is Orchestrator for you? Is it a thing >> from >>>>>>> Microsoft >>>>>>>> or >>>>>>>>>>> your >>>>>>>>>>>>>> custom >>>>>>>>>>>>>>>> in-house software? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sergi >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < >>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Fine. Let's say we've got multiple servers >>> which >>>>>>> fulfills >>>>>>>>>>> custom >>>>>>>>>>>>>> logic. >>>>>>>>>>>>>>>>> This servers compound oriented graph (BPMN >>>> process) >>>>>>> which >>>>>>>>>>>>> controlled >>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>> Orchestrator. >>>>>>>>>>>>>>>>> For instance, *server1 *creates *variable A >>>> *with >>>>>>> value >>>>>>>> 1, >>>>>>>>>>>>> persists >>>>>>>>>>>>>> it >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> IGNITE cache and creates *variable B *and >> sends >>>> it >>>>>> to* >>>>>>>>>> server2. >>>>>>>>>>>>> *The >>>>>>>>>>>>>>>>> latests receives *variable B*, do some logic >>> with >>>>> it >>>>>>> and >>>>>>>>>> stores >>>>>>>>>>>> to >>>>>>>>>>>>>>>> IGNITE. >>>>>>>>>>>>>>>>> All the work made by both servers must be >>>> fulfilled >>>>>> in >>>>>>>>> *one* >>>>>>>>>>>>>>> transaction. >>>>>>>>>>>>>>>>> Because we need all information done, or >>>>>>>>> nothing(rollbacked). >>>>>>>>>>> The >>>>>>>>>>>>>>>> scenario >>>>>>>>>>>>>>>>> is managed by orchestrator. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < >>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Ok, it is not a business case, it is your >>> wrong >>>>>>>> solution >>>>>>>>>> for >>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>> Lets try again, what is the business case? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Sergi >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2017-03-14 16:42 GMT+03:00 ALEKSEY >> KUZNETSOV >>> < >>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The case is the following, One starts >>>>> transaction >>>>>>> in >>>>>>>>> one >>>>>>>>>>>> node, >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> commit >>>>>>>>>>>>>>>>>>> this transaction in another jvm node(or >>>>> rollback >>>>>> it >>>>>>>>>>>> remotely). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 16:30, Sergi >>> Vladykin < >>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Because even if you make it work for >> some >>>>>>>> simplistic >>>>>>>>>>>>> scenario, >>>>>>>>>>>>>>> get >>>>>>>>>>>>>>>>>> ready >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> write many fault tolerance tests and >> make >>>>> sure >>>>>>> that >>>>>>>>> you >>>>>>>>>>> TXs >>>>>>>>>>>>>> work >>>>>>>>>>>>>>>>>>> gracefully >>>>>>>>>>>>>>>>>>>> in all modes in case of crashes. Also >>> make >>>>> sure >>>>>>>> that >>>>>>>>> we >>>>>>>>>>> do >>>>>>>>>>>>> not >>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>> any >>>>>>>>>>>>>>>>>>>> performance drops after all your >> changes >>> in >>>>>>>> existing >>>>>>>>>>>>>> benchmarks. >>>>>>>>>>>>>>>> All >>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>> I don't believe these conditions will >> be >>>> met >>>>>> and >>>>>>>> your >>>>>>>>>>>>>>> contribution >>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>> accepted. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Better solution to what problem? >> Sending >>> TX >>>>> to >>>>>>>>> another >>>>>>>>>>>> node? >>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>>>>> statement itself is already wrong. What >>>>>> business >>>>>>>> case >>>>>>>>>> you >>>>>>>>>>>> are >>>>>>>>>>>>>>>> trying >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> solve? I'm sure everything you need can >>> be >>>>> done >>>>>>> in >>>>>>>> a >>>>>>>>>> much >>>>>>>>>>>>> more >>>>>>>>>>>>>>>> simple >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> efficient way at the application level. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Sergi >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 2017-03-14 16:03 GMT+03:00 ALEKSEY >>>> KUZNETSOV >>>>> < >>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Why wrong ? You know the better >>> solution? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 15:46, Sergi >>>>> Vladykin < >>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Just serializing TX object and >>>>>> deserializing >>>>>>> it >>>>>>>>> on >>>>>>>>>>>>> another >>>>>>>>>>>>>>> node >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> meaningless, because other nodes >>>>>>> participating >>>>>>>> in >>>>>>>>>> the >>>>>>>>>>>> TX >>>>>>>>>>>>>> have >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> know >>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>> the new coordinator. This will >>> require >>>>>>> protocol >>>>>>>>>>>> changes, >>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>> definitely >>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>> have fault tolerance and >> performance >>>>>> issues. >>>>>>>> IMO >>>>>>>>>> the >>>>>>>>>>>>> whole >>>>>>>>>>>>>>> idea >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>> wrong >>>>>>>>>>>>>>>>>>>>>> and it makes no sense to waste time >>> on >>>>> it. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Sergi >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2017-03-14 10:57 GMT+03:00 ALEKSEY >>>>>> KUZNETSOV >>>>>>> < >>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> IgniteTransactionState >>>> implememntation >>>>>>>> contains >>>>>>>>>>>>>>>> IgniteTxEntry's >>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> supposed to be transferable >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 19:32, >>> Dmitriy >>>>>>>> Setrakyan >>>>>>>>> < >>>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> It sounds a little scary to me >>> that >>>>> we >>>>>>> are >>>>>>>>>>> passing >>>>>>>>>>>>>>>>> transaction >>>>>>>>>>>>>>>>>>>>> objects >>>>>>>>>>>>>>>>>>>>>>>> around. Such object may contain >>> all >>>>>> sorts >>>>>>>> of >>>>>>>>>>> Ignite >>>>>>>>>>>>>>>> context. >>>>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>> data >>>>>>>>>>>>>>>>>>>>>>>> needs to be passed across, we >>>> should >>>>>>>> create a >>>>>>>>>>>> special >>>>>>>>>>>>>>>>> transfer >>>>>>>>>>>>>>>>>>>> object >>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>> this case. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> D. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 13, 2017 at 9:10 >> AM, >>>>>> ALEKSEY >>>>>>>>>>> KUZNETSOV >>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> well, there a couple of >> issues >>>>>>> preventing >>>>>>>>>>>>> transaction >>>>>>>>>>>>>>>>>>> proceeding. >>>>>>>>>>>>>>>>>>>>>>>>> At first, After transaction >>>>>>> serialization >>>>>>>>> and >>>>>>>>>>>>>>>>> deserialization >>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> remote >>>>>>>>>>>>>>>>>>>>>>>>> server, there is no txState. >> So >>>> im >>>>>>> going >>>>>>>> to >>>>>>>>>> put >>>>>>>>>>>> it >>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>> >> writeExternal()\readExternal() >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> The last one is Deserialized >>>>>>> transaction >>>>>>>>>> lacks >>>>>>>>>>> of >>>>>>>>>>>>>>> shared >>>>>>>>>>>>>>>>>> cache >>>>>>>>>>>>>>>>>>>>>> context >>>>>>>>>>>>>>>>>>>>>>>>> field at >> TransactionProxyImpl. >>>>>> Perhaps, >>>>>>>> it >>>>>>>>>> must >>>>>>>>>>>> be >>>>>>>>>>>>>>>> injected >>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>> GridResourceProcessor ? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 17:27, >>>>> ALEKSEY >>>>>>>>>> KUZNETSOV >>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> while starting and >> continuing >>>>>>>> transaction >>>>>>>>>> in >>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>> jvms >>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>> run >>>>>>>>>>>>>>>>>>>>>>> into >>>>>>>>>>>>>>>>>>>>>>>>>> serialization exception in >>>>>>>>>> writeExternalMeta >>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> @Override public void >>>>>>>>>>>> writeExternal(ObjectOutput >>>>>>>>>>>>>> out) >>>>>>>>>>>>>>>>>> throws >>>>>>>>>>>>>>>>>>>>>>>> IOException >>>>>>>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>>>>>>> writeExternalMeta(out); >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> some meta is cannot be >>>>> serialized. >>>>>>>>>>>>>>>>>>>>>>>>>> пт, 10 мар. 2017 г. в >> 17:25, >>>>> Alexey >>>>>>>>>>> Goncharuk < >>>>>>>>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Aleksey, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I think I am starting to >> get >>>> what >>>>>> you >>>>>>>>> want, >>>>>>>>>>>> but I >>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>> few >>>>>>>>>>>>>>>>>>>>>>> concerns: >>>>>>>>>>>>>>>>>>>>>>>>>> - What is the API for the >>>>> proposed >>>>>>>>> change? >>>>>>>>>>> In >>>>>>>>>>>>> your >>>>>>>>>>>>>>>> test, >>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>>> pass >>>>>>>>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>>> instance of transaction >>> created >>>>> on >>>>>>>>>> ignite(0) >>>>>>>>>>> to >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> ignite >>>>>>>>>>>>>>>>>>>>> instance >>>>>>>>>>>>>>>>>>>>>>>>>> ignite(1). This is >> obviously >>>> not >>>>>>>> possible >>>>>>>>>> in >>>>>>>>>>> a >>>>>>>>>>>>>> truly >>>>>>>>>>>>>>>>>>>> distributed >>>>>>>>>>>>>>>>>>>>>>>>>> (multi-jvm) environment. >>>>>>>>>>>>>>>>>>>>>>>>>> - How will you synchronize >>>> cache >>>>>>> update >>>>>>>>>>> actions >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> transaction >>>>>>>>>>>>>>>>>>>>>>> commit? >>>>>>>>>>>>>>>>>>>>>>>>>> Say, you have one node that >>>>> decided >>>>>>> to >>>>>>>>>>> commit, >>>>>>>>>>>>> but >>>>>>>>>>>>>>>>> another >>>>>>>>>>>>>>>>>>> node >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>>>>> writing within this >>>> transaction. >>>>>> How >>>>>>> do >>>>>>>>> you >>>>>>>>>>>> make >>>>>>>>>>>>>> sure >>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>> nodes >>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>> not call commit() and >>>> rollback() >>>>>>>>>>>> simultaneously? >>>>>>>>>>>>>>>>>>>>>>>>>> - How do you make sure >> that >>>>> either >>>>>>>>>> commit() >>>>>>>>>>> or >>>>>>>>>>>>>>>>> rollback() >>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> called >>>>>>>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>>> originator failed? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:38 GMT+03:00 >>>>> Дмитрий >>>>>>>> Рябов >>>>>>>>> < >>>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Alexey Goncharuk, heh, my >>>>> initial >>>>>>>>>>>> understanding >>>>>>>>>>>>>> was >>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>> transferring >>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>> tx >>>>>>>>>>>>>>>>>>>>>>>>>>> ownership from one node >> to >>>>>> another >>>>>>>> will >>>>>>>>>> be >>>>>>>>>>>>>> happened >>>>>>>>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>>>>>>>>>>>> originating node is gone >>>> down. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:36 >> GMT+03:00 >>>>>> ALEKSEY >>>>>>>>>>> KUZNETSOV >>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Im aiming to span >>>> transaction >>>>>> on >>>>>>>>>> multiple >>>>>>>>>>>>>>> threads, >>>>>>>>>>>>>>>>>> nodes, >>>>>>>>>>>>>>>>>>>>>>>> jvms(soon). >>>>>>>>>>>>>>>>>>>>>>>>>> So >>>>>>>>>>>>>>>>>>>>>>>>>>>> every node is able to >>>>> rollback, >>>>>>> or >>>>>>>>>> commit >>>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>> transaction.It >>>>>>>>>>>>>>>>>>>>>>>>> turned >>>>>>>>>>>>>>>>>>>>>>>>>>> up i >>>>>>>>>>>>>>>>>>>>>>>>>>>> need to transfer tx >>> between >>>>>> nodes >>>>>>>> in >>>>>>>>>>> order >>>>>>>>>>>> to >>>>>>>>>>>>>>>> commit >>>>>>>>>>>>>>>>>>>>>> transaction >>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>> different node(in the >>> same >>>>>> jvm). >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> пт, 10 мар. 2017 г. в >>>> 15:20, >>>>>>> Alexey >>>>>>>>>>>>> Goncharuk < >>>>>>>>>>>>>>>>>>>>>>>>>>> >> [hidden email] >>>>>>>>>>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aleksey, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do you mean that you >>>> want a >>>>>>>> concept >>>>>>>>>> of >>>>>>>>>>>>>>>> transferring >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>> tx >>>>>>>>>>>>>>>>>>>>>>>> ownership >>>>>>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>>>>> one node to another? >> My >>>>>> initial >>>>>>>>>>>>> understanding >>>>>>>>>>>>>>> was >>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>>>> want >>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to update keys in a >>>>>> transaction >>>>>>>>> from >>>>>>>>>>>>> multiple >>>>>>>>>>>>>>>>> threads >>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>> parallel. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --AG >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:01 >>>> GMT+03:00 >>>>>>>> ALEKSEY >>>>>>>>>>>>> KUZNETSOV >>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Well. Consider >>>>> transaction >>>>>>>>> started >>>>>>>>>> in >>>>>>>>>>>> one >>>>>>>>>>>>>>> node, >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> continued >>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>> another >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following test >>>>>> describes >>>>>>> my >>>>>>>>>> idea: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ignite ignite1 = >>>>> ignite(0); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteTransactions >>>>>>>> transactions = >>>>>>>>>>>>>>>>>>>> ignite1.transactions(); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteCache<String, >>>>>> Integer> >>>>>>>>> cache >>>>>>>>>> = >>>>>>>>>>>>>>>>>>>>>>> ignite1.getOrCreateCache(" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> testCache"); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Transaction tx = >>>>>>>>>>> transactions.txStart( >>>>>>>>>>>>>>>>> concurrency, >>>>>>>>>>>>>>>>>>>>>>> isolation); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cache.put("key1", >> 1); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cache.put("key2", >> 2); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tx.stop(); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> IgniteInternalFuture<Boolean> >>>>>>>>> fut = >>>>>>>>>>>>>>>>>>>>>> GridTestUtils.runAsync(() >>>>>>>>>>>>>>>>>>>>>>>> -> >>>>>>>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> IgniteTransactions >>>>> ts = >>>>>>>>>>>>>>>>>> ignite(1).transactions(); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Assert.assertNull(ts.tx()); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Assert.assertEquals( >>>>>>>>>>>>>>>>> TransactionState.STOPPED, >>>>>>>>>>>>>>>>>>>>>>> tx.state()); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ts.txStart(tx); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Assert.assertEquals(TransactionState.ACTIVE, >>>>>>>>>>>>>>>>>>>>>>> tx.state()); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> cache.put("key3", >>>> 3); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Assert.assertTrue(cache.remove("key2")); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tx.commit(); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> return true; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> }); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fut.get(); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Assert.assertEquals( >>>>>>>>>>>>>>> TransactionState.COMMITTED, >>>>>>>>>>>>>>>>>>>>>> tx.state()); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Assert.assertEquals((long)1, >>>>>>>>>>>>>>>>>>> (long)cache.get("key1")); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Assert.assertEquals((long)3, >>>>>>>>>>>>>>>>>>> (long)cache.get("key3")); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Assert.assertFalse(cache. >>>>>>>>>>>>>>> containsKey("key2")); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In method >>>>> *ts.txStart(...)* >>>>>>> we >>>>>>>>> just >>>>>>>>>>>>> rebind >>>>>>>>>>>>>>> *tx* >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> current >>>>>>>>>>>>>>>>>>>>>>>>> thread: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> public void >>>>>>> txStart(Transaction >>>>>>>>>> tx) { >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> TransactionProxyImpl >>>>>>>>>>>>> transactionProxy = >>>>>>>>>>>>>>>>>>>>>>>>>> (TransactionProxyImpl)tx; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cctx.tm >>>> ().reopenTx( >>>>>>>>>>>>>>> transactionProxy.tx()); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> transactionProxy. >>>>>>>>>>>>> bindToCurrentThread(); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In method >> *reopenTx* >>> we >>>>>> alter >>>>>>>>>>>> *threadMap* >>>>>>>>>>>>>> so >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>> binds >>>>>>>>>>>>>>>>>>>>>>>>>>> transaction >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to current thread. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do u think >> about >>>> it ? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> вт, 7 мар. 2017 г. >> в >>>>> 22:38, >>>>>>>> Denis >>>>>>>>>>>> Magda < >>>>>>>>>>>>>>>>>>>>> [hidden email] >>>>>>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Alexey, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please share the >>>>> rational >>>>>>>>> behind >>>>>>>>>>> this >>>>>>>>>>>>> and >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> thoughts, >>>>>>>>>>>>>>>>>>>>>>>> design >>>>>>>>>>>>>>>>>>>>>>>>>>> ideas >>>>>>>>>>>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have in mind. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> — >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Denis >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mar 7, 2017, >>> at >>>>> 3:19 >>>>>>> AM, >>>>>>>>>>> ALEKSEY >>>>>>>>>>>>>>>>> KUZNETSOV < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [hidden email] >>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all! Im >>>> designing >>>>>>>>>> distributed >>>>>>>>>>>>>>>> transaction >>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>> started >>>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node, and >>> continued >>>>> at >>>>>>>> other >>>>>>>>>> one. >>>>>>>>>>>> Has >>>>>>>>>>>>>>>> anybody >>>>>>>>>>>>>>>>>>>>> thoughts >>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>> ? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov >>> Aleksey* >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> *Best Regards,* >>>>>>>>>>>>> >>>>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> *Best Regards,* >>>>>>>>>>> >>>>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *Best Regards,* >>>>>>>>> >>>>>>>>> *Kuznetsov Aleksey* >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *Best Regards,* >>>>>>>> >>>>>>>> *Kuznetsov Aleksey* >>>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *Best Regards,* >>>>>>> >>>>>>> *Kuznetsov Aleksey* >>>>>>> >>>>>> >>>>> -- >>>>> >>>>> *Best Regards,* >>>>> >>>>> *Kuznetsov Aleksey* >>>>> >>>> >>> -- >>> >>> *Best Regards,* >>> >>> *Kuznetsov Aleksey* >>> >> > -- > > *Best Regards,* > > *Kuznetsov Aleksey* |
|
|
Hi! No, i dont have ticket for this.
In the ticket i have implemented methods that change transaction status to STOP, thus letting it to commit transaction in another thread. In another thread you r going to restart transaction in order to commit it. The mechanism behind it is obvious : we change thread id to newer one in ThreadMap, and make use of serialization of txState, transactions itself to transfer them into another thread. вт, 28 мар. 2017 г. в 20:15, Denis Magda <[hidden email]>: > Aleksey, > > Do you have a ticket for this? Could you briefly list what exactly was > done and how the things work. > > — > Denis > > > On Mar 28, 2017, at 8:32 AM, ALEKSEY KUZNETSOV <[hidden email]> > wrote: > > > > Hi, Igniters! I 've made implementation of transactions of non-single > > coordinator. Here you can start transaction in one thread and commit it > in > > another thread. > > Take a look on it. Give your thoughts on it. > > > > > https://github.com/voipp/ignite/pull/10/commits/3a3d90aa6ac84f125e4c3ce4ced4f269a695ef45 > > > > пт, 17 мар. 2017 г. в 19:26, Sergi Vladykin <[hidden email]>: > > > >> You know better, go ahead! :) > >> > >> Sergi > >> > >> 2017-03-17 16:16 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > >> > >>> we've discovered several problems regarding your "accumulation" > >>> approach.These are > >>> > >>> 1. perfomance issues when transfering data from temporary cache to > >>> permanent one. Keep in mind big deal of concurent transactions in > >>> Service > >>> commiter > >>> 2. extreme memory load when keeping temporary cache in memory > >>> 3. As long as user is not acquainted with ignite, working with cache > >>> must be transparent for him. Keep this in mind. User's node can > >> evaluate > >>> logic with no transaction at all, so we should deal with both types > of > >>> execution flow : transactional and non-transactional.Another one > >>> problem is > >>> transaction id support at the user node. We would have handled all > >> this > >>> issues and many more. > >>> 4. we cannot pessimistically lock entity. > >>> > >>> As a result, we decided to move on building distributed transaction. We > >> put > >>> aside your "accumulation" approach until we realize how to solve > >>> difficulties above . > >>> > >>> чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin <[hidden email] > >: > >>> > >>>> The problem "How to run millions of entities, and millions of > >> operations > >>> on > >>>> a single Pentium3" is out of scope here. Do the math, plan capacity > >>>> reasonably. > >>>> > >>>> Sergi > >>>> > >>>> 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > >>> : > >>>> > >>>>> hmm, If we have millions of entities, and millions of operations, > >> would > >>>> not > >>>>> this approache lead to memory overflow and perfomance degradation > >>>>> > >>>>> чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin < > >> [hidden email] > >>>> : > >>>>> > >>>>>> 1. Actually you have to check versions on all the values you have > >>> read > >>>>>> during the tx. > >>>>>> > >>>>>> For example if we have [k1 => v1, k2 => v2] and do: > >>>>>> > >>>>>> put(k1, get(k2) + 5) > >>>>>> > >>>>>> We have to remember the version for k2. This logic can be > >> relatively > >>>>> easily > >>>>>> encapsulated in a framework atop of Ignite. You need to implement > >> one > >>>> to > >>>>>> make all this stuff usable. > >>>>>> > >>>>>> 2. I suggest to avoid any locking here, because you easily will end > >>> up > >>>>> with > >>>>>> deadlocks. If you do not have too frequent updates for your keys, > >>>>>> optimistic approach will work just fine. > >>>>>> > >>>>>> Theoretically in the Committer Service you can start a thread for > >> the > >>>>>> lifetime of the whole distributed transaction, take a lock on the > >> key > >>>>> using > >>>>>> IgniteCache.lock(K key) before executing any Services, wait for all > >>> the > >>>>>> services to complete, execute optimistic commit in the same thread > >>>> while > >>>>>> keeping this lock and then release it. Notice that all the Ignite > >>>>>> transactions inside of all Services must be optimistic here to be > >>> able > >>>> to > >>>>>> read this locked key. > >>>>>> > >>>>>> But again I do not recommend you using this approach until you > >> have a > >>>>>> reliable deadlock avoidance scheme. > >>>>>> > >>>>>> Sergi > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV < > >>> [hidden email] > >>>>> : > >>>>>> > >>>>>>> Yeah, now i got it. > >>>>>>> There are some doubts on this approach > >>>>>>> 1) During optimistic commit phase, when you assure no one altered > >>> the > >>>>>>> original values, you must check versions of other dependent keys. > >>> How > >>>>>> could > >>>>>>> we obtain those keys(in an automative manner, of course) ? > >>>>>>> 2) How could we lock a key before some Service A introduce > >> changes? > >>>> So > >>>>> no > >>>>>>> other service is allowed to change this key-value?(sort of > >>>> pessimistic > >>>>>>> blocking) > >>>>>>> May be you know some implementations of such approach ? > >>>>>>> > >>>>>>> ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > >>>>> [hidden email] > >>>>>>> : > >>>>>>> > >>>>>>>> Thank you very much for help. I will answer later. > >>>>>>>> > >>>>>>>> ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > >>>>> [hidden email] > >>>>>>> : > >>>>>>>> > >>>>>>>> All the services do not update key in place, but only generate > >>> new > >>>>> keys > >>>>>>>> augmented by otx and store the updated value in the same cache > >> + > >>>>>> remember > >>>>>>>> the keys and versions participating in the transaction in some > >>>>> separate > >>>>>>>> atomic cache. > >>>>>>>> > >>>>>>>> Follow this sequence of changes applied to cache contents by > >> each > >>>>>>> Service: > >>>>>>>> > >>>>>>>> Initial cache contents: > >>>>>>>> [k1 => v1] > >>>>>>>> [k2 => v2] > >>>>>>>> [k3 => v3] > >>>>>>>> > >>>>>>>> Cache contents after Service A: > >>>>>>>> [k1 => v1] > >>>>>>>> [k2 => v2] > >>>>>>>> [k3 => v3] > >>>>>>>> [k1x => v1a] > >>>>>>>> [k2x => v2a] > >>>>>>>> > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2)] in some separate > >>> atomic > >>>>>> cache > >>>>>>>> > >>>>>>>> Cache contents after Service B: > >>>>>>>> [k1 => v1] > >>>>>>>> [k2 => v2] > >>>>>>>> [k3 => v3] > >>>>>>>> [k1x => v1a] > >>>>>>>> [k2x => v2ab] > >>>>>>>> [k3x => v3b] > >>>>>>>> > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > >>>>> separate > >>>>>>>> atomic cache > >>>>>>>> > >>>>>>>> Finally the Committer Service takes this map of updated keys > >> and > >>>>> their > >>>>>>>> versions from some separate atomic cache, starts Ignite > >>> transaction > >>>>> and > >>>>>>>> replaces all the values for k* keys to values taken from k*x > >>> keys. > >>>>> The > >>>>>>>> successful result must be the following: > >>>>>>>> > >>>>>>>> [k1 => v1a] > >>>>>>>> [k2 => v2ab] > >>>>>>>> [k3 => v3b] > >>>>>>>> [k1x => v1a] > >>>>>>>> [k2x => v2ab] > >>>>>>>> [k3x => v3b] > >>>>>>>> > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > >>>>> separate > >>>>>>>> atomic cache > >>>>>>>> > >>>>>>>> But Committer Service also has to check that no one updated the > >>>>>> original > >>>>>>>> values before us, because otherwise we can not give any > >>>>> serializability > >>>>>>>> guarantee for these distributed transactions. Here we may need > >> to > >>>>> check > >>>>>>> not > >>>>>>>> only versions of the updated keys, but also versions of any > >> other > >>>>> keys > >>>>>>> end > >>>>>>>> result depends on. > >>>>>>>> > >>>>>>>> After that Committer Service has to do a cleanup (may be > >> outside > >>> of > >>>>> the > >>>>>>>> committing tx) to come to the following final state: > >>>>>>>> > >>>>>>>> [k1 => v1a] > >>>>>>>> [k2 => v2ab] > >>>>>>>> [k3 => v3b] > >>>>>>>> > >>>>>>>> Makes sense? > >>>>>>>> > >>>>>>>> Sergi > >>>>>>>> > >>>>>>>> > >>>>>>>> 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > >>>>> [hidden email] > >>>>>>> : > >>>>>>>> > >>>>>>>>> - what do u mean by saying " > >>>>>>>>> *in a single transaction checks value versions for all the > >> old > >>>>> values > >>>>>>>>> and replaces them with calculated new ones *"? Every time > >>> you > >>>>>>> change > >>>>>>>>> value(in some service), you store it to *some special > >> atomic > >>>>>> cache* > >>>>>>> , > >>>>>>>> so > >>>>>>>>> when all services ceased working, Service commiter got a > >>>> values > >>>>>> with > >>>>>>>> the > >>>>>>>>> last versions. > >>>>>>>>> - After "*does cleanup of temporary keys and values*" > >>> Service > >>>>>>> commiter > >>>>>>>>> persists them into permanent store, isn't it ? > >>>>>>>>> - I cant grasp your though, you say "*in case of version > >>>>> mismatch > >>>>>> or > >>>>>>>> TX > >>>>>>>>> timeout just rollbacks*". But what versions would it > >> match? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > >>>>>> [hidden email] > >>>>>>>> : > >>>>>>>>> > >>>>>>>>>> Ok, here is what you actually need to implement at the > >>>>> application > >>>>>>>> level. > >>>>>>>>>> > >>>>>>>>>> Lets say we have to call 2 services in the following order: > >>>>>>>>>> - Service A: wants to update keys [k1 => v1, k2 => v2] > >> to > >>>>> [k1 > >>>>>> => > >>>>>>>>> v1a, > >>>>>>>>>> k2 => v2a] > >>>>>>>>>> - Service B: wants to update keys [k2 => v2a, k3 => v3] > >> to > >>>> [k2 > >>>>>> => > >>>>>>>>> v2ab, > >>>>>>>>>> k3 => v3b] > >>>>>>>>>> > >>>>>>>>>> The change > >>>>>>>>>> from [ k1 => v1, k2 => v2, k3 => v3 ] > >>>>>>>>>> to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > >>>>>>>>>> must happen in a single transaction. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Optimistic protocol to solve this: > >>>>>>>>>> > >>>>>>>>>> Each cache key must have a field `otx`, which is a unique > >>>>>>> orchestrator > >>>>>>>> TX > >>>>>>>>>> identifier - it must be a parameter passed to all the > >>> services. > >>>>> If > >>>>>>>> `otx` > >>>>>>>>> is > >>>>>>>>>> set to some value it means that it is an intermediate key > >> and > >>>> is > >>>>>>>> visible > >>>>>>>>>> only inside of some transaction, for the finalized key > >> `otx` > >>>> must > >>>>>> be > >>>>>>>>> null - > >>>>>>>>>> it means the key is committed and visible for everyone. > >>>>>>>>>> > >>>>>>>>>> Each cache value must have a field `ver` which is a version > >>> of > >>>>> that > >>>>>>>>> value. > >>>>>>>>>> > >>>>>>>>>> For both fields (`otx` and `ver`) the safest way is to use > >>>> UUID. > >>>>>>>>>> > >>>>>>>>>> Workflow is the following: > >>>>>>>>>> > >>>>>>>>>> Orchestrator starts the distributed transaction with `otx` > >> = > >>> x > >>>>> and > >>>>>>>> passes > >>>>>>>>>> this parameter to all the services. > >>>>>>>>>> > >>>>>>>>>> Service A: > >>>>>>>>>> - does some computations > >>>>>>>>>> - stores [k1x => v1a, k2x => v2a] with TTL = Za > >>>>>>>>>> where > >>>>>>>>>> Za - left time from max Orchestrator TX duration > >>>> after > >>>>>>>> Service > >>>>>>>>> A > >>>>>>>>>> end > >>>>>>>>>> k1x, k2x - new temporary keys with field `otx` = > >> x > >>>>>>>>>> v2a has updated version `ver` > >>>>>>>>>> - returns a set of updated keys and all the old versions > >> to > >>>> the > >>>>>>>>>> orchestrator > >>>>>>>>>> or just stores it in some special atomic cache like > >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > >>>>>>>>>> > >>>>>>>>>> Service B: > >>>>>>>>>> - retrieves the updated value k2x => v2a because it knows > >>>> `otx` > >>>>> = > >>>>>> x > >>>>>>>>>> - does computations > >>>>>>>>>> - stores [k2x => v2ab, k3x => v3b] TTL = Zb > >>>>>>>>>> - updates the set of updated keys like [x => (k1 -> ver1, > >> k2 > >>>> -> > >>>>>>> ver2, > >>>>>>>> k3 > >>>>>>>>>> -> ver3)] TTL = Zb > >>>>>>>>>> > >>>>>>>>>> Service Committer (may be embedded into Orchestrator): > >>>>>>>>>> - takes all the updated keys and versions for `otx` = x > >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > >>>>>>>>>> - in a single transaction checks value versions for all > >> the > >>>> old > >>>>>>> values > >>>>>>>>>> and replaces them with calculated new ones > >>>>>>>>>> - does cleanup of temporary keys and values > >>>>>>>>>> - in case of version mismatch or TX timeout just rollbacks > >>> and > >>>>>>> signals > >>>>>>>>>> to Orchestrator to restart the job with new `otx` > >>>>>>>>>> > >>>>>>>>>> PROFIT!! > >>>>>>>>>> > >>>>>>>>>> This approach even allows you to run independent parts of > >> the > >>>>> graph > >>>>>>> in > >>>>>>>>>> parallel (with TX transfer you will always run only one at > >> a > >>>>> time). > >>>>>>>> Also > >>>>>>>>> it > >>>>>>>>>> does not require inventing any special fault tolerance > >>> technics > >>>>>>> because > >>>>>>>>>> Ignite caches are already fault tolerant and all the > >>>> intermediate > >>>>>>>> results > >>>>>>>>>> are virtually invisible and stored with TTL, thus in case > >> of > >>>> any > >>>>>>> crash > >>>>>>>>> you > >>>>>>>>>> will not have inconsistent state or garbage. > >>>>>>>>>> > >>>>>>>>>> Sergi > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > >>>>>>> [hidden email] > >>>>>>>>> : > >>>>>>>>>> > >>>>>>>>>>> Okay, we are open for proposals on business task. I mean, > >>> we > >>>>> can > >>>>>>> make > >>>>>>>>> use > >>>>>>>>>>> of some other thing, not distributed transaction. Not > >>>>> transaction > >>>>>>>> yet. > >>>>>>>>>>> > >>>>>>>>>>> ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > >>>>>> [hidden email] > >>>>>>>> : > >>>>>>>>>>> > >>>>>>>>>>>> IMO the use case makes sense. However, as Sergi already > >>>>>>> mentioned, > >>>>>>>>> the > >>>>>>>>>>>> problem is far more complex, than simply passing TX > >> state > >>>>> over > >>>>>> a > >>>>>>>>> wire. > >>>>>>>>>>> Most > >>>>>>>>>>>> probably a kind of coordinator will be required still > >> to > >>>>> manage > >>>>>>> all > >>>>>>>>>> kinds > >>>>>>>>>>>> of failures. This task should be started with clean > >>> design > >>>>>>> proposal > >>>>>>>>>>>> explaining how we handle all these concurrent events. > >> And > >>>>> only > >>>>>>>> then, > >>>>>>>>>> when > >>>>>>>>>>>> we understand all implications, we should move to > >>>> development > >>>>>>>> stage. > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > >>>>>>>>>>>> [hidden email]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Right > >>>>>>>>>>>>> > >>>>>>>>>>>>> ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > >>>>>>>>>> [hidden email] > >>>>>>>>>>>> : > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Good! Basically your orchestrator just takes some > >>>>>> predefined > >>>>>>>>> graph > >>>>>>>>>> of > >>>>>>>>>>>>>> distributed services to be invoked, calls them by > >>> some > >>>>> kind > >>>>>>> of > >>>>>>>>> RPC > >>>>>>>>>>> and > >>>>>>>>>>>>>> passes the needed parameters between them, right? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > >>>>>>>>>>> [hidden email] > >>>>>>>>>>>>> : > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> orchestrator is a custom thing. He is responsible > >>> for > >>>>>>>> managing > >>>>>>>>>>>> business > >>>>>>>>>>>>>>> scenarios flows. Many nodes are involved in > >>>> scenarios. > >>>>>> They > >>>>>>>>>>> exchange > >>>>>>>>>>>>> data > >>>>>>>>>>>>>>> and folow one another. If you acquinted with BPMN > >>>>>>> framework, > >>>>>>>> so > >>>>>>>>>>>>>>> orchestrator is like bpmn engine. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > >>>>>>>>>> [hidden email] > >>>>>>>>>>>> : > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> What is Orchestrator for you? Is it a thing > >> from > >>>>>>> Microsoft > >>>>>>>> or > >>>>>>>>>>> your > >>>>>>>>>>>>>> custom > >>>>>>>>>>>>>>>> in-house software? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > >>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Fine. Let's say we've got multiple servers > >>> which > >>>>>>> fulfills > >>>>>>>>>>> custom > >>>>>>>>>>>>>> logic. > >>>>>>>>>>>>>>>>> This servers compound oriented graph (BPMN > >>>> process) > >>>>>>> which > >>>>>>>>>>>>> controlled > >>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>> Orchestrator. > >>>>>>>>>>>>>>>>> For instance, *server1 *creates *variable A > >>>> *with > >>>>>>> value > >>>>>>>> 1, > >>>>>>>>>>>>> persists > >>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> IGNITE cache and creates *variable B *and > >> sends > >>>> it > >>>>>> to* > >>>>>>>>>> server2. > >>>>>>>>>>>>> *The > >>>>>>>>>>>>>>>>> latests receives *variable B*, do some logic > >>> with > >>>>> it > >>>>>>> and > >>>>>>>>>> stores > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>> IGNITE. > >>>>>>>>>>>>>>>>> All the work made by both servers must be > >>>> fulfilled > >>>>>> in > >>>>>>>>> *one* > >>>>>>>>>>>>>>> transaction. > >>>>>>>>>>>>>>>>> Because we need all information done, or > >>>>>>>>> nothing(rollbacked). > >>>>>>>>>>> The > >>>>>>>>>>>>>>>> scenario > >>>>>>>>>>>>>>>>> is managed by orchestrator. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > >>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Ok, it is not a business case, it is your > >>> wrong > >>>>>>>> solution > >>>>>>>>>> for > >>>>>>>>>>>> it. > >>>>>>>>>>>>>>>>>> Lets try again, what is the business case? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 2017-03-14 16:42 GMT+03:00 ALEKSEY > >> KUZNETSOV > >>> < > >>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> The case is the following, One starts > >>>>> transaction > >>>>>>> in > >>>>>>>>> one > >>>>>>>>>>>> node, > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> commit > >>>>>>>>>>>>>>>>>>> this transaction in another jvm node(or > >>>>> rollback > >>>>>> it > >>>>>>>>>>>> remotely). > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 16:30, Sergi > >>> Vladykin < > >>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Because even if you make it work for > >> some > >>>>>>>> simplistic > >>>>>>>>>>>>> scenario, > >>>>>>>>>>>>>>> get > >>>>>>>>>>>>>>>>>> ready > >>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>> write many fault tolerance tests and > >> make > >>>>> sure > >>>>>>> that > >>>>>>>>> you > >>>>>>>>>>> TXs > >>>>>>>>>>>>>> work > >>>>>>>>>>>>>>>>>>> gracefully > >>>>>>>>>>>>>>>>>>>> in all modes in case of crashes. Also > >>> make > >>>>> sure > >>>>>>>> that > >>>>>>>>> we > >>>>>>>>>>> do > >>>>>>>>>>>>> not > >>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>> any > >>>>>>>>>>>>>>>>>>>> performance drops after all your > >> changes > >>> in > >>>>>>>> existing > >>>>>>>>>>>>>> benchmarks. > >>>>>>>>>>>>>>>> All > >>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>>> I don't believe these conditions will > >> be > >>>> met > >>>>>> and > >>>>>>>> your > >>>>>>>>>>>>>>> contribution > >>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>> accepted. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Better solution to what problem? > >> Sending > >>> TX > >>>>> to > >>>>>>>>> another > >>>>>>>>>>>> node? > >>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>> problem > >>>>>>>>>>>>>>>>>>>> statement itself is already wrong. What > >>>>>> business > >>>>>>>> case > >>>>>>>>>> you > >>>>>>>>>>>> are > >>>>>>>>>>>>>>>> trying > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>> solve? I'm sure everything you need can > >>> be > >>>>> done > >>>>>>> in > >>>>>>>> a > >>>>>>>>>> much > >>>>>>>>>>>>> more > >>>>>>>>>>>>>>>> simple > >>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>> efficient way at the application level. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> 2017-03-14 16:03 GMT+03:00 ALEKSEY > >>>> KUZNETSOV > >>>>> < > >>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Why wrong ? You know the better > >>> solution? > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 15:46, Sergi > >>>>> Vladykin < > >>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Just serializing TX object and > >>>>>> deserializing > >>>>>>> it > >>>>>>>>> on > >>>>>>>>>>>>> another > >>>>>>>>>>>>>>> node > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>> meaningless, because other nodes > >>>>>>> participating > >>>>>>>> in > >>>>>>>>>> the > >>>>>>>>>>>> TX > >>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>> know > >>>>>>>>>>>>>>>>>>>>> about > >>>>>>>>>>>>>>>>>>>>>> the new coordinator. This will > >>> require > >>>>>>> protocol > >>>>>>>>>>>> changes, > >>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>> definitely > >>>>>>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>>>>> have fault tolerance and > >> performance > >>>>>> issues. > >>>>>>>> IMO > >>>>>>>>>> the > >>>>>>>>>>>>> whole > >>>>>>>>>>>>>>> idea > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>> wrong > >>>>>>>>>>>>>>>>>>>>>> and it makes no sense to waste time > >>> on > >>>>> it. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 2017-03-14 10:57 GMT+03:00 ALEKSEY > >>>>>> KUZNETSOV > >>>>>>> < > >>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> IgniteTransactionState > >>>> implememntation > >>>>>>>> contains > >>>>>>>>>>>>>>>> IgniteTxEntry's > >>>>>>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>> supposed to be transferable > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 19:32, > >>> Dmitriy > >>>>>>>> Setrakyan > >>>>>>>>> < > >>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> It sounds a little scary to me > >>> that > >>>>> we > >>>>>>> are > >>>>>>>>>>> passing > >>>>>>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>>>> objects > >>>>>>>>>>>>>>>>>>>>>>>> around. Such object may contain > >>> all > >>>>>> sorts > >>>>>>>> of > >>>>>>>>>>> Ignite > >>>>>>>>>>>>>>>> context. > >>>>>>>>>>>>>>>>> If > >>>>>>>>>>>>>>>>>>>> some > >>>>>>>>>>>>>>>>>>>>>> data > >>>>>>>>>>>>>>>>>>>>>>>> needs to be passed across, we > >>>> should > >>>>>>>> create a > >>>>>>>>>>>> special > >>>>>>>>>>>>>>>>> transfer > >>>>>>>>>>>>>>>>>>>> object > >>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>> this case. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> D. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 13, 2017 at 9:10 > >> AM, > >>>>>> ALEKSEY > >>>>>>>>>>> KUZNETSOV > >>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> well, there a couple of > >> issues > >>>>>>> preventing > >>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>> proceeding. > >>>>>>>>>>>>>>>>>>>>>>>>> At first, After transaction > >>>>>>> serialization > >>>>>>>>> and > >>>>>>>>>>>>>>>>> deserialization > >>>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> remote > >>>>>>>>>>>>>>>>>>>>>>>>> server, there is no txState. > >> So > >>>> im > >>>>>>> going > >>>>>>>> to > >>>>>>>>>> put > >>>>>>>>>>>> it > >>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>> > >> writeExternal()\readExternal() > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> The last one is Deserialized > >>>>>>> transaction > >>>>>>>>>> lacks > >>>>>>>>>>> of > >>>>>>>>>>>>>>> shared > >>>>>>>>>>>>>>>>>> cache > >>>>>>>>>>>>>>>>>>>>>> context > >>>>>>>>>>>>>>>>>>>>>>>>> field at > >> TransactionProxyImpl. > >>>>>> Perhaps, > >>>>>>>> it > >>>>>>>>>> must > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>> injected > >>>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>>>>>>>> GridResourceProcessor ? > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 17:27, > >>>>> ALEKSEY > >>>>>>>>>> KUZNETSOV > >>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> while starting and > >> continuing > >>>>>>>> transaction > >>>>>>>>>> in > >>>>>>>>>>>>>>> different > >>>>>>>>>>>>>>>>> jvms > >>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>> run > >>>>>>>>>>>>>>>>>>>>>>> into > >>>>>>>>>>>>>>>>>>>>>>>>>> serialization exception in > >>>>>>>>>> writeExternalMeta > >>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> @Override public void > >>>>>>>>>>>> writeExternal(ObjectOutput > >>>>>>>>>>>>>> out) > >>>>>>>>>>>>>>>>>> throws > >>>>>>>>>>>>>>>>>>>>>>>> IOException > >>>>>>>>>>>>>>>>>>>>>>>>> { > >>>>>>>>>>>>>>>>>>>>>>>>>> writeExternalMeta(out); > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> some meta is cannot be > >>>>> serialized. > >>>>>>>>>>>>>>>>>>>>>>>>>> пт, 10 мар. 2017 г. в > >> 17:25, > >>>>> Alexey > >>>>>>>>>>> Goncharuk < > >>>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Aleksey, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think I am starting to > >> get > >>>> what > >>>>>> you > >>>>>>>>> want, > >>>>>>>>>>>> but I > >>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>> few > >>>>>>>>>>>>>>>>>>>>>>> concerns: > >>>>>>>>>>>>>>>>>>>>>>>>>> - What is the API for the > >>>>> proposed > >>>>>>>>> change? > >>>>>>>>>>> In > >>>>>>>>>>>>> your > >>>>>>>>>>>>>>>> test, > >>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>> pass > >>>>>>>>>>>>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>>>>>> instance of transaction > >>> created > >>>>> on > >>>>>>>>>> ignite(0) > >>>>>>>>>>> to > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> ignite > >>>>>>>>>>>>>>>>>>>>> instance > >>>>>>>>>>>>>>>>>>>>>>>>>> ignite(1). This is > >> obviously > >>>> not > >>>>>>>> possible > >>>>>>>>>> in > >>>>>>>>>>> a > >>>>>>>>>>>>>> truly > >>>>>>>>>>>>>>>>>>>> distributed > >>>>>>>>>>>>>>>>>>>>>>>>>> (multi-jvm) environment. > >>>>>>>>>>>>>>>>>>>>>>>>>> - How will you synchronize > >>>> cache > >>>>>>> update > >>>>>>>>>>> actions > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>>>>>> commit? > >>>>>>>>>>>>>>>>>>>>>>>>>> Say, you have one node that > >>>>> decided > >>>>>>> to > >>>>>>>>>>> commit, > >>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>> another > >>>>>>>>>>>>>>>>>>> node > >>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>>>>>>>>>>> writing within this > >>>> transaction. > >>>>>> How > >>>>>>> do > >>>>>>>>> you > >>>>>>>>>>>> make > >>>>>>>>>>>>>> sure > >>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>> two > >>>>>>>>>>>>>>>>>>>>>> nodes > >>>>>>>>>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>>>>>>>>> not call commit() and > >>>> rollback() > >>>>>>>>>>>> simultaneously? > >>>>>>>>>>>>>>>>>>>>>>>>>> - How do you make sure > >> that > >>>>> either > >>>>>>>>>> commit() > >>>>>>>>>>> or > >>>>>>>>>>>>>>>>> rollback() > >>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>> called > >>>>>>>>>>>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>>>>>> originator failed? > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:38 GMT+03:00 > >>>>> Дмитрий > >>>>>>>> Рябов > >>>>>>>>> < > >>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Alexey Goncharuk, heh, my > >>>>> initial > >>>>>>>>>>>> understanding > >>>>>>>>>>>>>> was > >>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>> transferring > >>>>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>> tx > >>>>>>>>>>>>>>>>>>>>>>>>>>> ownership from one node > >> to > >>>>>> another > >>>>>>>> will > >>>>>>>>>> be > >>>>>>>>>>>>>> happened > >>>>>>>>>>>>>>>>>>>>> automatically > >>>>>>>>>>>>>>>>>>>>>>>> when > >>>>>>>>>>>>>>>>>>>>>>>>>>> originating node is gone > >>>> down. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:36 > >> GMT+03:00 > >>>>>> ALEKSEY > >>>>>>>>>>> KUZNETSOV > >>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Im aiming to span > >>>> transaction > >>>>>> on > >>>>>>>>>> multiple > >>>>>>>>>>>>>>> threads, > >>>>>>>>>>>>>>>>>> nodes, > >>>>>>>>>>>>>>>>>>>>>>>> jvms(soon). > >>>>>>>>>>>>>>>>>>>>>>>>>> So > >>>>>>>>>>>>>>>>>>>>>>>>>>>> every node is able to > >>>>> rollback, > >>>>>>> or > >>>>>>>>>> commit > >>>>>>>>>>>>>> common > >>>>>>>>>>>>>>>>>>>>> transaction.It > >>>>>>>>>>>>>>>>>>>>>>>>> turned > >>>>>>>>>>>>>>>>>>>>>>>>>>> up i > >>>>>>>>>>>>>>>>>>>>>>>>>>>> need to transfer tx > >>> between > >>>>>> nodes > >>>>>>>> in > >>>>>>>>>>> order > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>> commit > >>>>>>>>>>>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>>>> different node(in the > >>> same > >>>>>> jvm). > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> пт, 10 мар. 2017 г. в > >>>> 15:20, > >>>>>>> Alexey > >>>>>>>>>>>>> Goncharuk < > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aleksey, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do you mean that you > >>>> want a > >>>>>>>> concept > >>>>>>>>>> of > >>>>>>>>>>>>>>>> transferring > >>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>> tx > >>>>>>>>>>>>>>>>>>>>>>>> ownership > >>>>>>>>>>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> one node to another? > >> My > >>>>>> initial > >>>>>>>>>>>>> understanding > >>>>>>>>>>>>>>> was > >>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>>> want > >>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>>>> able > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to update keys in a > >>>>>> transaction > >>>>>>>>> from > >>>>>>>>>>>>> multiple > >>>>>>>>>>>>>>>>> threads > >>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>> parallel. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --AG > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:01 > >>>> GMT+03:00 > >>>>>>>> ALEKSEY > >>>>>>>>>>>>> KUZNETSOV > >>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Well. Consider > >>>>> transaction > >>>>>>>>> started > >>>>>>>>>> in > >>>>>>>>>>>> one > >>>>>>>>>>>>>>> node, > >>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> continued > >>>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>>>> another > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following test > >>>>>> describes > >>>>>>> my > >>>>>>>>>> idea: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ignite ignite1 = > >>>>> ignite(0); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteTransactions > >>>>>>>> transactions = > >>>>>>>>>>>>>>>>>>>> ignite1.transactions(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteCache<String, > >>>>>> Integer> > >>>>>>>>> cache > >>>>>>>>>> = > >>>>>>>>>>>>>>>>>>>>>>> ignite1.getOrCreateCache(" > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> testCache"); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Transaction tx = > >>>>>>>>>>> transactions.txStart( > >>>>>>>>>>>>>>>>> concurrency, > >>>>>>>>>>>>>>>>>>>>>>> isolation); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cache.put("key1", > >> 1); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cache.put("key2", > >> 2); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tx.stop(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> IgniteInternalFuture<Boolean> > >>>>>>>>> fut = > >>>>>>>>>>>>>>>>>>>>>> GridTestUtils.runAsync(() > >>>>>>>>>>>>>>>>>>>>>>>> -> > >>>>>>>>>>>>>>>>>>>>>>>>> { > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> IgniteTransactions > >>>>> ts = > >>>>>>>>>>>>>>>>>> ignite(1).transactions(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Assert.assertNull(ts.tx()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Assert.assertEquals( > >>>>>>>>>>>>>>>>> TransactionState.STOPPED, > >>>>>>>>>>>>>>>>>>>>>>> tx.state()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ts.txStart(tx); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Assert.assertEquals(TransactionState.ACTIVE, > >>>>>>>>>>>>>>>>>>>>>>> tx.state()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> cache.put("key3", > >>>> 3); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> Assert.assertTrue(cache.remove("key2")); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tx.commit(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> return true; > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> }); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fut.get(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> Assert.assertEquals( > >>>>>>>>>>>>>>> TransactionState.COMMITTED, > >>>>>>>>>>>>>>>>>>>>>> tx.state()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Assert.assertEquals((long)1, > >>>>>>>>>>>>>>>>>>> (long)cache.get("key1")); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Assert.assertEquals((long)3, > >>>>>>>>>>>>>>>>>>> (long)cache.get("key3")); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Assert.assertFalse(cache. > >>>>>>>>>>>>>>> containsKey("key2")); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In method > >>>>> *ts.txStart(...)* > >>>>>>> we > >>>>>>>>> just > >>>>>>>>>>>>> rebind > >>>>>>>>>>>>>>> *tx* > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> current > >>>>>>>>>>>>>>>>>>>>>>>>> thread: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> public void > >>>>>>> txStart(Transaction > >>>>>>>>>> tx) { > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> TransactionProxyImpl > >>>>>>>>>>>>> transactionProxy = > >>>>>>>>>>>>>>>>>>>>>>>>>> (TransactionProxyImpl)tx; > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cctx.tm > >>>> ().reopenTx( > >>>>>>>>>>>>>>> transactionProxy.tx()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> transactionProxy. > >>>>>>>>>>>>> bindToCurrentThread(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In method > >> *reopenTx* > >>> we > >>>>>> alter > >>>>>>>>>>>> *threadMap* > >>>>>>>>>>>>>> so > >>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>> binds > >>>>>>>>>>>>>>>>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to current thread. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do u think > >> about > >>>> it ? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> вт, 7 мар. 2017 г. > >> в > >>>>> 22:38, > >>>>>>>> Denis > >>>>>>>>>>>> Magda < > >>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Alexey, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please share the > >>>>> rational > >>>>>>>>> behind > >>>>>>>>>>> this > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> thoughts, > >>>>>>>>>>>>>>>>>>>>>>>> design > >>>>>>>>>>>>>>>>>>>>>>>>>>> ideas > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have in mind. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> — > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Denis > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mar 7, 2017, > >>> at > >>>>> 3:19 > >>>>>>> AM, > >>>>>>>>>>> ALEKSEY > >>>>>>>>>>>>>>>>> KUZNETSOV < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> [hidden email] > >>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all! Im > >>>> designing > >>>>>>>>>> distributed > >>>>>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>> started > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node, and > >>> continued > >>>>> at > >>>>>>>> other > >>>>>>>>>> one. > >>>>>>>>>>>> Has > >>>>>>>>>>>>>>>> anybody > >>>>>>>>>>>>>>>>>>>>> thoughts > >>>>>>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>>>> ? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov > >>> Aleksey* > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Best Regards,* > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> > >>>>>>>>>>> *Best Regards,* > >>>>>>>>>>> > >>>>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> > >>>>>>>>> *Best Regards,* > >>>>>>>>> > >>>>>>>>> *Kuznetsov Aleksey* > >>>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> > >>>>>>>> *Best Regards,* > >>>>>>>> > >>>>>>>> *Kuznetsov Aleksey* > >>>>>>>> > >>>>>>> -- > >>>>>>> > >>>>>>> *Best Regards,* > >>>>>>> > >>>>>>> *Kuznetsov Aleksey* > >>>>>>> > >>>>>> > >>>>> -- > >>>>> > >>>>> *Best Regards,* > >>>>> > >>>>> *Kuznetsov Aleksey* > >>>>> > >>>> > >>> -- > >>> > >>> *Best Regards,* > >>> > >>> *Kuznetsov Aleksey* > >>> > >> > > -- > > > > *Best Regards,* > > > > *Kuznetsov Aleksey* > > -- *Best Regards,* *Kuznetsov Aleksey* |
|
|
So, what do u think on my idea ?
ср, 29 мар. 2017 г. в 10:35, ALEKSEY KUZNETSOV <[hidden email]>: > Hi! No, i dont have ticket for this. > In the ticket i have implemented methods that change transaction status to > STOP, thus letting it to commit transaction in another thread. In another > thread you r going to restart transaction in order to commit it. > The mechanism behind it is obvious : we change thread id to newer one in > ThreadMap, and make use of serialization of txState, transactions itself to > transfer them into another thread. > > > вт, 28 мар. 2017 г. в 20:15, Denis Magda <[hidden email]>: > > Aleksey, > > Do you have a ticket for this? Could you briefly list what exactly was > done and how the things work. > > — > Denis > > > On Mar 28, 2017, at 8:32 AM, ALEKSEY KUZNETSOV <[hidden email]> > wrote: > > > > Hi, Igniters! I 've made implementation of transactions of non-single > > coordinator. Here you can start transaction in one thread and commit it > in > > another thread. > > Take a look on it. Give your thoughts on it. > > > > > https://github.com/voipp/ignite/pull/10/commits/3a3d90aa6ac84f125e4c3ce4ced4f269a695ef45 > > > > пт, 17 мар. 2017 г. в 19:26, Sergi Vladykin <[hidden email]>: > > > >> You know better, go ahead! :) > >> > >> Sergi > >> > >> 2017-03-17 16:16 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email] > >: > >> > >>> we've discovered several problems regarding your "accumulation" > >>> approach.These are > >>> > >>> 1. perfomance issues when transfering data from temporary cache to > >>> permanent one. Keep in mind big deal of concurent transactions in > >>> Service > >>> commiter > >>> 2. extreme memory load when keeping temporary cache in memory > >>> 3. As long as user is not acquainted with ignite, working with cache > >>> must be transparent for him. Keep this in mind. User's node can > >> evaluate > >>> logic with no transaction at all, so we should deal with both types > of > >>> execution flow : transactional and non-transactional.Another one > >>> problem is > >>> transaction id support at the user node. We would have handled all > >> this > >>> issues and many more. > >>> 4. we cannot pessimistically lock entity. > >>> > >>> As a result, we decided to move on building distributed transaction. We > >> put > >>> aside your "accumulation" approach until we realize how to solve > >>> difficulties above . > >>> > >>> чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin <[hidden email] > >: > >>> > >>>> The problem "How to run millions of entities, and millions of > >> operations > >>> on > >>>> a single Pentium3" is out of scope here. Do the math, plan capacity > >>>> reasonably. > >>>> > >>>> Sergi > >>>> > >>>> 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > >>> : > >>>> > >>>>> hmm, If we have millions of entities, and millions of operations, > >> would > >>>> not > >>>>> this approache lead to memory overflow and perfomance degradation > >>>>> > >>>>> чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin < > >> [hidden email] > >>>> : > >>>>> > >>>>>> 1. Actually you have to check versions on all the values you have > >>> read > >>>>>> during the tx. > >>>>>> > >>>>>> For example if we have [k1 => v1, k2 => v2] and do: > >>>>>> > >>>>>> put(k1, get(k2) + 5) > >>>>>> > >>>>>> We have to remember the version for k2. This logic can be > >> relatively > >>>>> easily > >>>>>> encapsulated in a framework atop of Ignite. You need to implement > >> one > >>>> to > >>>>>> make all this stuff usable. > >>>>>> > >>>>>> 2. I suggest to avoid any locking here, because you easily will end > >>> up > >>>>> with > >>>>>> deadlocks. If you do not have too frequent updates for your keys, > >>>>>> optimistic approach will work just fine. > >>>>>> > >>>>>> Theoretically in the Committer Service you can start a thread for > >> the > >>>>>> lifetime of the whole distributed transaction, take a lock on the > >> key > >>>>> using > >>>>>> IgniteCache.lock(K key) before executing any Services, wait for all > >>> the > >>>>>> services to complete, execute optimistic commit in the same thread > >>>> while > >>>>>> keeping this lock and then release it. Notice that all the Ignite > >>>>>> transactions inside of all Services must be optimistic here to be > >>> able > >>>> to > >>>>>> read this locked key. > >>>>>> > >>>>>> But again I do not recommend you using this approach until you > >> have a > >>>>>> reliable deadlock avoidance scheme. > >>>>>> > >>>>>> Sergi > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV < > >>> [hidden email] > >>>>> : > >>>>>> > >>>>>>> Yeah, now i got it. > >>>>>>> There are some doubts on this approach > >>>>>>> 1) During optimistic commit phase, when you assure no one altered > >>> the > >>>>>>> original values, you must check versions of other dependent keys. > >>> How > >>>>>> could > >>>>>>> we obtain those keys(in an automative manner, of course) ? > >>>>>>> 2) How could we lock a key before some Service A introduce > >> changes? > >>>> So > >>>>> no > >>>>>>> other service is allowed to change this key-value?(sort of > >>>> pessimistic > >>>>>>> blocking) > >>>>>>> May be you know some implementations of such approach ? > >>>>>>> > >>>>>>> ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > >>>>> [hidden email] > >>>>>>> : > >>>>>>> > >>>>>>>> Thank you very much for help. I will answer later. > >>>>>>>> > >>>>>>>> ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > >>>>> [hidden email] > >>>>>>> : > >>>>>>>> > >>>>>>>> All the services do not update key in place, but only generate > >>> new > >>>>> keys > >>>>>>>> augmented by otx and store the updated value in the same cache > >> + > >>>>>> remember > >>>>>>>> the keys and versions participating in the transaction in some > >>>>> separate > >>>>>>>> atomic cache. > >>>>>>>> > >>>>>>>> Follow this sequence of changes applied to cache contents by > >> each > >>>>>>> Service: > >>>>>>>> > >>>>>>>> Initial cache contents: > >>>>>>>> [k1 => v1] > >>>>>>>> [k2 => v2] > >>>>>>>> [k3 => v3] > >>>>>>>> > >>>>>>>> Cache contents after Service A: > >>>>>>>> [k1 => v1] > >>>>>>>> [k2 => v2] > >>>>>>>> [k3 => v3] > >>>>>>>> [k1x => v1a] > >>>>>>>> [k2x => v2a] > >>>>>>>> > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2)] in some separate > >>> atomic > >>>>>> cache > >>>>>>>> > >>>>>>>> Cache contents after Service B: > >>>>>>>> [k1 => v1] > >>>>>>>> [k2 => v2] > >>>>>>>> [k3 => v3] > >>>>>>>> [k1x => v1a] > >>>>>>>> [k2x => v2ab] > >>>>>>>> [k3x => v3b] > >>>>>>>> > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > >>>>> separate > >>>>>>>> atomic cache > >>>>>>>> > >>>>>>>> Finally the Committer Service takes this map of updated keys > >> and > >>>>> their > >>>>>>>> versions from some separate atomic cache, starts Ignite > >>> transaction > >>>>> and > >>>>>>>> replaces all the values for k* keys to values taken from k*x > >>> keys. > >>>>> The > >>>>>>>> successful result must be the following: > >>>>>>>> > >>>>>>>> [k1 => v1a] > >>>>>>>> [k2 => v2ab] > >>>>>>>> [k3 => v3b] > >>>>>>>> [k1x => v1a] > >>>>>>>> [k2x => v2ab] > >>>>>>>> [k3x => v3b] > >>>>>>>> > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > >>>>> separate > >>>>>>>> atomic cache > >>>>>>>> > >>>>>>>> But Committer Service also has to check that no one updated the > >>>>>> original > >>>>>>>> values before us, because otherwise we can not give any > >>>>> serializability > >>>>>>>> guarantee for these distributed transactions. Here we may need > >> to > >>>>> check > >>>>>>> not > >>>>>>>> only versions of the updated keys, but also versions of any > >> other > >>>>> keys > >>>>>>> end > >>>>>>>> result depends on. > >>>>>>>> > >>>>>>>> After that Committer Service has to do a cleanup (may be > >> outside > >>> of > >>>>> the > >>>>>>>> committing tx) to come to the following final state: > >>>>>>>> > >>>>>>>> [k1 => v1a] > >>>>>>>> [k2 => v2ab] > >>>>>>>> [k3 => v3b] > >>>>>>>> > >>>>>>>> Makes sense? > >>>>>>>> > >>>>>>>> Sergi > >>>>>>>> > >>>>>>>> > >>>>>>>> 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > >>>>> [hidden email] > >>>>>>> : > >>>>>>>> > >>>>>>>>> - what do u mean by saying " > >>>>>>>>> *in a single transaction checks value versions for all the > >> old > >>>>> values > >>>>>>>>> and replaces them with calculated new ones *"? Every time > >>> you > >>>>>>> change > >>>>>>>>> value(in some service), you store it to *some special > >> atomic > >>>>>> cache* > >>>>>>> , > >>>>>>>> so > >>>>>>>>> when all services ceased working, Service commiter got a > >>>> values > >>>>>> with > >>>>>>>> the > >>>>>>>>> last versions. > >>>>>>>>> - After "*does cleanup of temporary keys and values*" > >>> Service > >>>>>>> commiter > >>>>>>>>> persists them into permanent store, isn't it ? > >>>>>>>>> - I cant grasp your though, you say "*in case of version > >>>>> mismatch > >>>>>> or > >>>>>>>> TX > >>>>>>>>> timeout just rollbacks*". But what versions would it > >> match? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > >>>>>> [hidden email] > >>>>>>>> : > >>>>>>>>> > >>>>>>>>>> Ok, here is what you actually need to implement at the > >>>>> application > >>>>>>>> level. > >>>>>>>>>> > >>>>>>>>>> Lets say we have to call 2 services in the following order: > >>>>>>>>>> - Service A: wants to update keys [k1 => v1, k2 => v2] > >> to > >>>>> [k1 > >>>>>> => > >>>>>>>>> v1a, > >>>>>>>>>> k2 => v2a] > >>>>>>>>>> - Service B: wants to update keys [k2 => v2a, k3 => v3] > >> to > >>>> [k2 > >>>>>> => > >>>>>>>>> v2ab, > >>>>>>>>>> k3 => v3b] > >>>>>>>>>> > >>>>>>>>>> The change > >>>>>>>>>> from [ k1 => v1, k2 => v2, k3 => v3 ] > >>>>>>>>>> to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > >>>>>>>>>> must happen in a single transaction. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Optimistic protocol to solve this: > >>>>>>>>>> > >>>>>>>>>> Each cache key must have a field `otx`, which is a unique > >>>>>>> orchestrator > >>>>>>>> TX > >>>>>>>>>> identifier - it must be a parameter passed to all the > >>> services. > >>>>> If > >>>>>>>> `otx` > >>>>>>>>> is > >>>>>>>>>> set to some value it means that it is an intermediate key > >> and > >>>> is > >>>>>>>> visible > >>>>>>>>>> only inside of some transaction, for the finalized key > >> `otx` > >>>> must > >>>>>> be > >>>>>>>>> null - > >>>>>>>>>> it means the key is committed and visible for everyone. > >>>>>>>>>> > >>>>>>>>>> Each cache value must have a field `ver` which is a version > >>> of > >>>>> that > >>>>>>>>> value. > >>>>>>>>>> > >>>>>>>>>> For both fields (`otx` and `ver`) the safest way is to use > >>>> UUID. > >>>>>>>>>> > >>>>>>>>>> Workflow is the following: > >>>>>>>>>> > >>>>>>>>>> Orchestrator starts the distributed transaction with `otx` > >> = > >>> x > >>>>> and > >>>>>>>> passes > >>>>>>>>>> this parameter to all the services. > >>>>>>>>>> > >>>>>>>>>> Service A: > >>>>>>>>>> - does some computations > >>>>>>>>>> - stores [k1x => v1a, k2x => v2a] with TTL = Za > >>>>>>>>>> where > >>>>>>>>>> Za - left time from max Orchestrator TX duration > >>>> after > >>>>>>>> Service > >>>>>>>>> A > >>>>>>>>>> end > >>>>>>>>>> k1x, k2x - new temporary keys with field `otx` = > >> x > >>>>>>>>>> v2a has updated version `ver` > >>>>>>>>>> - returns a set of updated keys and all the old versions > >> to > >>>> the > >>>>>>>>>> orchestrator > >>>>>>>>>> or just stores it in some special atomic cache like > >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > >>>>>>>>>> > >>>>>>>>>> Service B: > >>>>>>>>>> - retrieves the updated value k2x => v2a because it knows > >>>> `otx` > >>>>> = > >>>>>> x > >>>>>>>>>> - does computations > >>>>>>>>>> - stores [k2x => v2ab, k3x => v3b] TTL = Zb > >>>>>>>>>> - updates the set of updated keys like [x => (k1 -> ver1, > >> k2 > >>>> -> > >>>>>>> ver2, > >>>>>>>> k3 > >>>>>>>>>> -> ver3)] TTL = Zb > >>>>>>>>>> > >>>>>>>>>> Service Committer (may be embedded into Orchestrator): > >>>>>>>>>> - takes all the updated keys and versions for `otx` = x > >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > >>>>>>>>>> - in a single transaction checks value versions for all > >> the > >>>> old > >>>>>>> values > >>>>>>>>>> and replaces them with calculated new ones > >>>>>>>>>> - does cleanup of temporary keys and values > >>>>>>>>>> - in case of version mismatch or TX timeout just rollbacks > >>> and > >>>>>>> signals > >>>>>>>>>> to Orchestrator to restart the job with new `otx` > >>>>>>>>>> > >>>>>>>>>> PROFIT!! > >>>>>>>>>> > >>>>>>>>>> This approach even allows you to run independent parts of > >> the > >>>>> graph > >>>>>>> in > >>>>>>>>>> parallel (with TX transfer you will always run only one at > >> a > >>>>> time). > >>>>>>>> Also > >>>>>>>>> it > >>>>>>>>>> does not require inventing any special fault tolerance > >>> technics > >>>>>>> because > >>>>>>>>>> Ignite caches are already fault tolerant and all the > >>>> intermediate > >>>>>>>> results > >>>>>>>>>> are virtually invisible and stored with TTL, thus in case > >> of > >>>> any > >>>>>>> crash > >>>>>>>>> you > >>>>>>>>>> will not have inconsistent state or garbage. > >>>>>>>>>> > >>>>>>>>>> Sergi > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > >>>>>>> [hidden email] > >>>>>>>>> : > >>>>>>>>>> > >>>>>>>>>>> Okay, we are open for proposals on business task. I mean, > >>> we > >>>>> can > >>>>>>> make > >>>>>>>>> use > >>>>>>>>>>> of some other thing, not distributed transaction. Not > >>>>> transaction > >>>>>>>> yet. > >>>>>>>>>>> > >>>>>>>>>>> ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > >>>>>> [hidden email] > >>>>>>>> : > >>>>>>>>>>> > >>>>>>>>>>>> IMO the use case makes sense. However, as Sergi already > >>>>>>> mentioned, > >>>>>>>>> the > >>>>>>>>>>>> problem is far more complex, than simply passing TX > >> state > >>>>> over > >>>>>> a > >>>>>>>>> wire. > >>>>>>>>>>> Most > >>>>>>>>>>>> probably a kind of coordinator will be required still > >> to > >>>>> manage > >>>>>>> all > >>>>>>>>>> kinds > >>>>>>>>>>>> of failures. This task should be started with clean > >>> design > >>>>>>> proposal > >>>>>>>>>>>> explaining how we handle all these concurrent events. > >> And > >>>>> only > >>>>>>>> then, > >>>>>>>>>> when > >>>>>>>>>>>> we understand all implications, we should move to > >>>> development > >>>>>>>> stage. > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > >>>>>>>>>>>> [hidden email]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Right > >>>>>>>>>>>>> > >>>>>>>>>>>>> ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > >>>>>>>>>> [hidden email] > >>>>>>>>>>>> : > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Good! Basically your orchestrator just takes some > >>>>>> predefined > >>>>>>>>> graph > >>>>>>>>>> of > >>>>>>>>>>>>>> distributed services to be invoked, calls them by > >>> some > >>>>> kind > >>>>>>> of > >>>>>>>>> RPC > >>>>>>>>>>> and > >>>>>>>>>>>>>> passes the needed parameters between them, right? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > >>>>>>>>>>> [hidden email] > >>>>>>>>>>>>> : > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> orchestrator is a custom thing. He is responsible > >>> for > >>>>>>>> managing > >>>>>>>>>>>> business > >>>>>>>>>>>>>>> scenarios flows. Many nodes are involved in > >>>> scenarios. > >>>>>> They > >>>>>>>>>>> exchange > >>>>>>>>>>>>> data > >>>>>>>>>>>>>>> and folow one another. If you acquinted with BPMN > >>>>>>> framework, > >>>>>>>> so > >>>>>>>>>>>>>>> orchestrator is like bpmn engine. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > >>>>>>>>>> [hidden email] > >>>>>>>>>>>> : > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> What is Orchestrator for you? Is it a thing > >> from > >>>>>>> Microsoft > >>>>>>>> or > >>>>>>>>>>> your > >>>>>>>>>>>>>> custom > >>>>>>>>>>>>>>>> in-house software? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > >>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Fine. Let's say we've got multiple servers > >>> which > >>>>>>> fulfills > >>>>>>>>>>> custom > >>>>>>>>>>>>>> logic. > >>>>>>>>>>>>>>>>> This servers compound oriented graph (BPMN > >>>> process) > >>>>>>> which > >>>>>>>>>>>>> controlled > >>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>> Orchestrator. > >>>>>>>>>>>>>>>>> For instance, *server1 *creates *variable A > >>>> *with > >>>>>>> value > >>>>>>>> 1, > >>>>>>>>>>>>> persists > >>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> IGNITE cache and creates *variable B *and > >> sends > >>>> it > >>>>>> to* > >>>>>>>>>> server2. > >>>>>>>>>>>>> *The > >>>>>>>>>>>>>>>>> latests receives *variable B*, do some logic > >>> with > >>>>> it > >>>>>>> and > >>>>>>>>>> stores > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>> IGNITE. > >>>>>>>>>>>>>>>>> All the work made by both servers must be > >>>> fulfilled > >>>>>> in > >>>>>>>>> *one* > >>>>>>>>>>>>>>> transaction. > >>>>>>>>>>>>>>>>> Because we need all information done, or > >>>>>>>>> nothing(rollbacked). > >>>>>>>>>>> The > >>>>>>>>>>>>>>>> scenario > >>>>>>>>>>>>>>>>> is managed by orchestrator. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > >>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Ok, it is not a business case, it is your > >>> wrong > >>>>>>>> solution > >>>>>>>>>> for > >>>>>>>>>>>> it. > >>>>>>>>>>>>>>>>>> Lets try again, what is the business case? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 2017-03-14 16:42 GMT+03:00 ALEKSEY > >> KUZNETSOV > >>> < > >>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> The case is the following, One starts > >>>>> transaction > >>>>>>> in > >>>>>>>>> one > >>>>>>>>>>>> node, > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> commit > >>>>>>>>>>>>>>>>>>> this transaction in another jvm node(or > >>>>> rollback > >>>>>> it > >>>>>>>>>>>> remotely). > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 16:30, Sergi > >>> Vladykin < > >>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Because even if you make it work for > >> some > >>>>>>>> simplistic > >>>>>>>>>>>>> scenario, > >>>>>>>>>>>>>>> get > >>>>>>>>>>>>>>>>>> ready > >>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>> write many fault tolerance tests and > >> make > >>>>> sure > >>>>>>> that > >>>>>>>>> you > >>>>>>>>>>> TXs > >>>>>>>>>>>>>> work > >>>>>>>>>>>>>>>>>>> gracefully > >>>>>>>>>>>>>>>>>>>> in all modes in case of crashes. Also > >>> make > >>>>> sure > >>>>>>>> that > >>>>>>>>> we > >>>>>>>>>>> do > >>>>>>>>>>>>> not > >>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>> any > >>>>>>>>>>>>>>>>>>>> performance drops after all your > >> changes > >>> in > >>>>>>>> existing > >>>>>>>>>>>>>> benchmarks. > >>>>>>>>>>>>>>>> All > >>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>>> I don't believe these conditions will > >> be > >>>> met > >>>>>> and > >>>>>>>> your > >>>>>>>>>>>>>>> contribution > >>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>> accepted. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Better solution to what problem? > >> Sending > >>> TX > >>>>> to > >>>>>>>>> another > >>>>>>>>>>>> node? > >>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>> problem > >>>>>>>>>>>>>>>>>>>> statement itself is already wrong. What > >>>>>> business > >>>>>>>> case > >>>>>>>>>> you > >>>>>>>>>>>> are > >>>>>>>>>>>>>>>> trying > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>> solve? I'm sure everything you need can > >>> be > >>>>> done > >>>>>>> in > >>>>>>>> a > >>>>>>>>>> much > >>>>>>>>>>>>> more > >>>>>>>>>>>>>>>> simple > >>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>> efficient way at the application level. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> 2017-03-14 16:03 GMT+03:00 ALEKSEY > >>>> KUZNETSOV > >>>>> < > >>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Why wrong ? You know the better > >>> solution? > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 15:46, Sergi > >>>>> Vladykin < > >>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Just serializing TX object and > >>>>>> deserializing > >>>>>>> it > >>>>>>>>> on > >>>>>>>>>>>>> another > >>>>>>>>>>>>>>> node > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>> meaningless, because other nodes > >>>>>>> participating > >>>>>>>> in > >>>>>>>>>> the > >>>>>>>>>>>> TX > >>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>> know > >>>>>>>>>>>>>>>>>>>>> about > >>>>>>>>>>>>>>>>>>>>>> the new coordinator. This will > >>> require > >>>>>>> protocol > >>>>>>>>>>>> changes, > >>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>> definitely > >>>>>>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>>>>> have fault tolerance and > >> performance > >>>>>> issues. > >>>>>>>> IMO > >>>>>>>>>> the > >>>>>>>>>>>>> whole > >>>>>>>>>>>>>>> idea > >>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>> wrong > >>>>>>>>>>>>>>>>>>>>>> and it makes no sense to waste time > >>> on > >>>>> it. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Sergi > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 2017-03-14 10:57 GMT+03:00 ALEKSEY > >>>>>> KUZNETSOV > >>>>>>> < > >>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> IgniteTransactionState > >>>> implememntation > >>>>>>>> contains > >>>>>>>>>>>>>>>> IgniteTxEntry's > >>>>>>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>> supposed to be transferable > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 19:32, > >>> Dmitriy > >>>>>>>> Setrakyan > >>>>>>>>> < > >>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> It sounds a little scary to me > >>> that > >>>>> we > >>>>>>> are > >>>>>>>>>>> passing > >>>>>>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>>>> objects > >>>>>>>>>>>>>>>>>>>>>>>> around. Such object may contain > >>> all > >>>>>> sorts > >>>>>>>> of > >>>>>>>>>>> Ignite > >>>>>>>>>>>>>>>> context. > >>>>>>>>>>>>>>>>> If > >>>>>>>>>>>>>>>>>>>> some > >>>>>>>>>>>>>>>>>>>>>> data > >>>>>>>>>>>>>>>>>>>>>>>> needs to be passed across, we > >>>> should > >>>>>>>> create a > >>>>>>>>>>>> special > >>>>>>>>>>>>>>>>> transfer > >>>>>>>>>>>>>>>>>>>> object > >>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>> this case. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> D. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 13, 2017 at 9:10 > >> AM, > >>>>>> ALEKSEY > >>>>>>>>>>> KUZNETSOV > >>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> well, there a couple of > >> issues > >>>>>>> preventing > >>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>> proceeding. > >>>>>>>>>>>>>>>>>>>>>>>>> At first, After transaction > >>>>>>> serialization > >>>>>>>>> and > >>>>>>>>>>>>>>>>> deserialization > >>>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> remote > >>>>>>>>>>>>>>>>>>>>>>>>> server, there is no txState. > >> So > >>>> im > >>>>>>> going > >>>>>>>> to > >>>>>>>>>> put > >>>>>>>>>>>> it > >>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>> > >> writeExternal()\readExternal() > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> The last one is Deserialized > >>>>>>> transaction > >>>>>>>>>> lacks > >>>>>>>>>>> of > >>>>>>>>>>>>>>> shared > >>>>>>>>>>>>>>>>>> cache > >>>>>>>>>>>>>>>>>>>>>> context > >>>>>>>>>>>>>>>>>>>>>>>>> field at > >> TransactionProxyImpl. > >>>>>> Perhaps, > >>>>>>>> it > >>>>>>>>>> must > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>> injected > >>>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>>>>>>>> GridResourceProcessor ? > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 17:27, > >>>>> ALEKSEY > >>>>>>>>>> KUZNETSOV > >>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> while starting and > >> continuing > >>>>>>>> transaction > >>>>>>>>>> in > >>>>>>>>>>>>>>> different > >>>>>>>>>>>>>>>>> jvms > >>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>> run > >>>>>>>>>>>>>>>>>>>>>>> into > >>>>>>>>>>>>>>>>>>>>>>>>>> serialization exception in > >>>>>>>>>> writeExternalMeta > >>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> @Override public void > >>>>>>>>>>>> writeExternal(ObjectOutput > >>>>>>>>>>>>>> out) > >>>>>>>>>>>>>>>>>> throws > >>>>>>>>>>>>>>>>>>>>>>>> IOException > >>>>>>>>>>>>>>>>>>>>>>>>> { > >>>>>>>>>>>>>>>>>>>>>>>>>> writeExternalMeta(out); > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> some meta is cannot be > >>>>> serialized. > >>>>>>>>>>>>>>>>>>>>>>>>>> пт, 10 мар. 2017 г. в > >> 17:25, > >>>>> Alexey > >>>>>>>>>>> Goncharuk < > >>>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Aleksey, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think I am starting to > >> get > >>>> what > >>>>>> you > >>>>>>>>> want, > >>>>>>>>>>>> but I > >>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>> few > >>>>>>>>>>>>>>>>>>>>>>> concerns: > >>>>>>>>>>>>>>>>>>>>>>>>>> - What is the API for the > >>>>> proposed > >>>>>>>>> change? > >>>>>>>>>>> In > >>>>>>>>>>>>> your > >>>>>>>>>>>>>>>> test, > >>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>> pass > >>>>>>>>>>>>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>>>>>> instance of transaction > >>> created > >>>>> on > >>>>>>>>>> ignite(0) > >>>>>>>>>>> to > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> ignite > >>>>>>>>>>>>>>>>>>>>> instance > >>>>>>>>>>>>>>>>>>>>>>>>>> ignite(1). This is > >> obviously > >>>> not > >>>>>>>> possible > >>>>>>>>>> in > >>>>>>>>>>> a > >>>>>>>>>>>>>> truly > >>>>>>>>>>>>>>>>>>>> distributed > >>>>>>>>>>>>>>>>>>>>>>>>>> (multi-jvm) environment. > >>>>>>>>>>>>>>>>>>>>>>>>>> - How will you synchronize > >>>> cache > >>>>>>> update > >>>>>>>>>>> actions > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>>>>>> commit? > >>>>>>>>>>>>>>>>>>>>>>>>>> Say, you have one node that > >>>>> decided > >>>>>>> to > >>>>>>>>>>> commit, > >>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>> another > >>>>>>>>>>>>>>>>>>> node > >>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>>>>>>>>>>> writing within this > >>>> transaction. > >>>>>> How > >>>>>>> do > >>>>>>>>> you > >>>>>>>>>>>> make > >>>>>>>>>>>>>> sure > >>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>> two > >>>>>>>>>>>>>>>>>>>>>> nodes > >>>>>>>>>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>>>>>>>>> not call commit() and > >>>> rollback() > >>>>>>>>>>>> simultaneously? > >>>>>>>>>>>>>>>>>>>>>>>>>> - How do you make sure > >> that > >>>>> either > >>>>>>>>>> commit() > >>>>>>>>>>> or > >>>>>>>>>>>>>>>>> rollback() > >>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>> called > >>>>>>>>>>>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>>>>>> originator failed? > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:38 GMT+03:00 > >>>>> Дмитрий > >>>>>>>> Рябов > >>>>>>>>> < > >>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Alexey Goncharuk, heh, my > >>>>> initial > >>>>>>>>>>>> understanding > >>>>>>>>>>>>>> was > >>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>> transferring > >>>>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>> tx > >>>>>>>>>>>>>>>>>>>>>>>>>>> ownership from one node > >> to > >>>>>> another > >>>>>>>> will > >>>>>>>>>> be > >>>>>>>>>>>>>> happened > >>>>>>>>>>>>>>>>>>>>> automatically > >>>>>>>>>>>>>>>>>>>>>>>> when > >>>>>>>>>>>>>>>>>>>>>>>>>>> originating node is gone > >>>> down. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:36 > >> GMT+03:00 > >>>>>> ALEKSEY > >>>>>>>>>>> KUZNETSOV > >>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Im aiming to span > >>>> transaction > >>>>>> on > >>>>>>>>>> multiple > >>>>>>>>>>>>>>> threads, > >>>>>>>>>>>>>>>>>> nodes, > >>>>>>>>>>>>>>>>>>>>>>>> jvms(soon). > >>>>>>>>>>>>>>>>>>>>>>>>>> So > >>>>>>>>>>>>>>>>>>>>>>>>>>>> every node is able to > >>>>> rollback, > >>>>>>> or > >>>>>>>>>> commit > >>>>>>>>>>>>>> common > >>>>>>>>>>>>>>>>>>>>> transaction.It > >>>>>>>>>>>>>>>>>>>>>>>>> turned > >>>>>>>>>>>>>>>>>>>>>>>>>>> up i > >>>>>>>>>>>>>>>>>>>>>>>>>>>> need to transfer tx > >>> between > >>>>>> nodes > >>>>>>>> in > >>>>>>>>>>> order > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>> commit > >>>>>>>>>>>>>>>>>>>>>> transaction > >>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>>>> different node(in the > >>> same > >>>>>> jvm). > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> пт, 10 мар. 2017 г. в > >>>> 15:20, > >>>>>>> Alexey > >>>>>>>>>>>>> Goncharuk < > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aleksey, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do you mean that you > >>>> want a > >>>>>>>> concept > >>>>>>>>>> of > >>>>>>>>>>>>>>>> transferring > >>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>> tx > >>>>>>>>>>>>>>>>>>>>>>>> ownership > >>>>>>>>>>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> one node to another? > >> My > >>>>>> initial > >>>>>>>>>>>>> understanding > >>>>>>>>>>>>>>> was > >>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>>> want > >>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>>>> able > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to update keys in a > >>>>>> transaction > >>>>>>>>> from > >>>>>>>>>>>>> multiple > >>>>>>>>>>>>>>>>> threads > >>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>> parallel. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --AG > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:01 > >>>> GMT+03:00 > >>>>>>>> ALEKSEY > >>>>>>>>>>>>> KUZNETSOV > >>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > >>>>>>>>>>>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Well. Consider > >>>>> transaction > >>>>>>>>> started > >>>>>>>>>> in > >>>>>>>>>>>> one > >>>>>>>>>>>>>>> node, > >>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> continued > >>>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>>>> another > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following test > >>>>>> describes > >>>>>>> my > >>>>>>>>>> idea: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ignite ignite1 = > >>>>> ignite(0); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteTransactions > >>>>>>>> transactions = > >>>>>>>>>>>>>>>>>>>> ignite1.transactions(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteCache<String, > >>>>>> Integer> > >>>>>>>>> cache > >>>>>>>>>> = > >>>>>>>>>>>>>>>>>>>>>>> ignite1.getOrCreateCache(" > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> testCache"); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Transaction tx = > >>>>>>>>>>> transactions.txStart( > >>>>>>>>>>>>>>>>> concurrency, > >>>>>>>>>>>>>>>>>>>>>>> isolation); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cache.put("key1", > >> 1); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cache.put("key2", > >> 2); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tx.stop(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> IgniteInternalFuture<Boolean> > >>>>>>>>> fut = > >>>>>>>>>>>>>>>>>>>>>> GridTestUtils.runAsync(() > >>>>>>>>>>>>>>>>>>>>>>>> -> > >>>>>>>>>>>>>>>>>>>>>>>>> { > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>> IgniteTransactions > >>>>> ts = > >>>>>>>>>>>>>>>>>> ignite(1).transactions(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>> Assert.assertNull(ts.tx()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Assert.assertEquals( > >>>>>>>>>>>>>>>>> TransactionState.STOPPED, > >>>>>>>>>>>>>>>>>>>>>>> tx.state()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ts.txStart(tx); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Assert.assertEquals(TransactionState.ACTIVE, > >>>>>>>>>>>>>>>>>>>>>>> tx.state()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> cache.put("key3", > >>>> 3); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> Assert.assertTrue(cache.remove("key2")); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tx.commit(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> return true; > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> }); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fut.get(); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> Assert.assertEquals( > >>>>>>>>>>>>>>> TransactionState.COMMITTED, > >>>>>>>>>>>>>>>>>>>>>> tx.state()); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Assert.assertEquals((long)1, > >>>>>>>>>>>>>>>>>>> (long)cache.get("key1")); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>> Assert.assertEquals((long)3, > >>>>>>>>>>>>>>>>>>> (long)cache.get("key3")); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> Assert.assertFalse(cache. > >>>>>>>>>>>>>>> containsKey("key2")); > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In method > >>>>> *ts.txStart(...)* > >>>>>>> we > >>>>>>>>> just > >>>>>>>>>>>>> rebind > >>>>>>>>>>>>>>> *tx* > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> current > >>>>>>>>>>>>>>>>>>>>>>>>> thread: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > > -- *Best Regards,* *Kuznetsov Aleksey* |
Re: distributed transaction of non-single coordinator
|
|
Aleksey,
I doubt your approach works as expected. Current transaction recovery protocol heavily relies on the originating node ID in its internal logic. For example, currently a transaction will be rolled back if you want to transfer a transaction ownership to another node and original tx owner fails. An attempt to commit such a transaction on another node may fail with all sorts of assertions. After transaction ownership changed, you need to notify all current transaction participants about this change, and it should also be done failover-safe, let alone that you did not add any tests for these cases. I back Denis here. Please create a ticket first and come up with clear use-cases, API and protocol changes design. It is hard to reason about the changes you've made when we do not even understand why you are making these changes and how they are supposed to work. --AG 2017-03-30 10:43 GMT+03:00 ALEKSEY KUZNETSOV <[hidden email]>: > So, what do u think on my idea ? > > ср, 29 мар. 2017 г. в 10:35, ALEKSEY KUZNETSOV <[hidden email]>: > > > Hi! No, i dont have ticket for this. > > In the ticket i have implemented methods that change transaction status > to > > STOP, thus letting it to commit transaction in another thread. In another > > thread you r going to restart transaction in order to commit it. > > The mechanism behind it is obvious : we change thread id to newer one in > > ThreadMap, and make use of serialization of txState, transactions itself > to > > transfer them into another thread. > > > > > > вт, 28 мар. 2017 г. в 20:15, Denis Magda <[hidden email]>: > > > > Aleksey, > > > > Do you have a ticket for this? Could you briefly list what exactly was > > done and how the things work. > > > > — > > Denis > > > > > On Mar 28, 2017, at 8:32 AM, ALEKSEY KUZNETSOV < > [hidden email]> > > wrote: > > > > > > Hi, Igniters! I 've made implementation of transactions of non-single > > > coordinator. Here you can start transaction in one thread and commit it > > in > > > another thread. > > > Take a look on it. Give your thoughts on it. > > > > > > > > https://github.com/voipp/ignite/pull/10/commits/ > 3a3d90aa6ac84f125e4c3ce4ced4f269a695ef45 > > > > > > пт, 17 мар. 2017 г. в 19:26, Sergi Vladykin <[hidden email] > >: > > > > > >> You know better, go ahead! :) > > >> > > >> Sergi > > >> > > >> 2017-03-17 16:16 GMT+03:00 ALEKSEY KUZNETSOV < > [hidden email] > > >: > > >> > > >>> we've discovered several problems regarding your "accumulation" > > >>> approach.These are > > >>> > > >>> 1. perfomance issues when transfering data from temporary cache to > > >>> permanent one. Keep in mind big deal of concurent transactions in > > >>> Service > > >>> commiter > > >>> 2. extreme memory load when keeping temporary cache in memory > > >>> 3. As long as user is not acquainted with ignite, working with > cache > > >>> must be transparent for him. Keep this in mind. User's node can > > >> evaluate > > >>> logic with no transaction at all, so we should deal with both types > > of > > >>> execution flow : transactional and non-transactional.Another one > > >>> problem is > > >>> transaction id support at the user node. We would have handled all > > >> this > > >>> issues and many more. > > >>> 4. we cannot pessimistically lock entity. > > >>> > > >>> As a result, we decided to move on building distributed transaction. > We > > >> put > > >>> aside your "accumulation" approach until we realize how to solve > > >>> difficulties above . > > >>> > > >>> чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin < > [hidden email] > > >: > > >>> > > >>>> The problem "How to run millions of entities, and millions of > > >> operations > > >>> on > > >>>> a single Pentium3" is out of scope here. Do the math, plan capacity > > >>>> reasonably. > > >>>> > > >>>> Sergi > > >>>> > > >>>> 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV < > > [hidden email] > > >>> : > > >>>> > > >>>>> hmm, If we have millions of entities, and millions of operations, > > >> would > > >>>> not > > >>>>> this approache lead to memory overflow and perfomance degradation > > >>>>> > > >>>>> чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin < > > >> [hidden email] > > >>>> : > > >>>>> > > >>>>>> 1. Actually you have to check versions on all the values you have > > >>> read > > >>>>>> during the tx. > > >>>>>> > > >>>>>> For example if we have [k1 => v1, k2 => v2] and do: > > >>>>>> > > >>>>>> put(k1, get(k2) + 5) > > >>>>>> > > >>>>>> We have to remember the version for k2. This logic can be > > >> relatively > > >>>>> easily > > >>>>>> encapsulated in a framework atop of Ignite. You need to implement > > >> one > > >>>> to > > >>>>>> make all this stuff usable. > > >>>>>> > > >>>>>> 2. I suggest to avoid any locking here, because you easily will > end > > >>> up > > >>>>> with > > >>>>>> deadlocks. If you do not have too frequent updates for your keys, > > >>>>>> optimistic approach will work just fine. > > >>>>>> > > >>>>>> Theoretically in the Committer Service you can start a thread for > > >> the > > >>>>>> lifetime of the whole distributed transaction, take a lock on the > > >> key > > >>>>> using > > >>>>>> IgniteCache.lock(K key) before executing any Services, wait for > all > > >>> the > > >>>>>> services to complete, execute optimistic commit in the same thread > > >>>> while > > >>>>>> keeping this lock and then release it. Notice that all the Ignite > > >>>>>> transactions inside of all Services must be optimistic here to be > > >>> able > > >>>> to > > >>>>>> read this locked key. > > >>>>>> > > >>>>>> But again I do not recommend you using this approach until you > > >> have a > > >>>>>> reliable deadlock avoidance scheme. > > >>>>>> > > >>>>>> Sergi > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV < > > >>> [hidden email] > > >>>>> : > > >>>>>> > > >>>>>>> Yeah, now i got it. > > >>>>>>> There are some doubts on this approach > > >>>>>>> 1) During optimistic commit phase, when you assure no one altered > > >>> the > > >>>>>>> original values, you must check versions of other dependent keys. > > >>> How > > >>>>>> could > > >>>>>>> we obtain those keys(in an automative manner, of course) ? > > >>>>>>> 2) How could we lock a key before some Service A introduce > > >> changes? > > >>>> So > > >>>>> no > > >>>>>>> other service is allowed to change this key-value?(sort of > > >>>> pessimistic > > >>>>>>> blocking) > > >>>>>>> May be you know some implementations of such approach ? > > >>>>>>> > > >>>>>>> ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > > >>>>> [hidden email] > > >>>>>>> : > > >>>>>>> > > >>>>>>>> Thank you very much for help. I will answer later. > > >>>>>>>> > > >>>>>>>> ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > > >>>>> [hidden email] > > >>>>>>> : > > >>>>>>>> > > >>>>>>>> All the services do not update key in place, but only generate > > >>> new > > >>>>> keys > > >>>>>>>> augmented by otx and store the updated value in the same cache > > >> + > > >>>>>> remember > > >>>>>>>> the keys and versions participating in the transaction in some > > >>>>> separate > > >>>>>>>> atomic cache. > > >>>>>>>> > > >>>>>>>> Follow this sequence of changes applied to cache contents by > > >> each > > >>>>>>> Service: > > >>>>>>>> > > >>>>>>>> Initial cache contents: > > >>>>>>>> [k1 => v1] > > >>>>>>>> [k2 => v2] > > >>>>>>>> [k3 => v3] > > >>>>>>>> > > >>>>>>>> Cache contents after Service A: > > >>>>>>>> [k1 => v1] > > >>>>>>>> [k2 => v2] > > >>>>>>>> [k3 => v3] > > >>>>>>>> [k1x => v1a] > > >>>>>>>> [k2x => v2a] > > >>>>>>>> > > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2)] in some separate > > >>> atomic > > >>>>>> cache > > >>>>>>>> > > >>>>>>>> Cache contents after Service B: > > >>>>>>>> [k1 => v1] > > >>>>>>>> [k2 => v2] > > >>>>>>>> [k3 => v3] > > >>>>>>>> [k1x => v1a] > > >>>>>>>> [k2x => v2ab] > > >>>>>>>> [k3x => v3b] > > >>>>>>>> > > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > >>>>> separate > > >>>>>>>> atomic cache > > >>>>>>>> > > >>>>>>>> Finally the Committer Service takes this map of updated keys > > >> and > > >>>>> their > > >>>>>>>> versions from some separate atomic cache, starts Ignite > > >>> transaction > > >>>>> and > > >>>>>>>> replaces all the values for k* keys to values taken from k*x > > >>> keys. > > >>>>> The > > >>>>>>>> successful result must be the following: > > >>>>>>>> > > >>>>>>>> [k1 => v1a] > > >>>>>>>> [k2 => v2ab] > > >>>>>>>> [k3 => v3b] > > >>>>>>>> [k1x => v1a] > > >>>>>>>> [k2x => v2ab] > > >>>>>>>> [k3x => v3b] > > >>>>>>>> > > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] in some > > >>>>> separate > > >>>>>>>> atomic cache > > >>>>>>>> > > >>>>>>>> But Committer Service also has to check that no one updated the > > >>>>>> original > > >>>>>>>> values before us, because otherwise we can not give any > > >>>>> serializability > > >>>>>>>> guarantee for these distributed transactions. Here we may need > > >> to > > >>>>> check > > >>>>>>> not > > >>>>>>>> only versions of the updated keys, but also versions of any > > >> other > > >>>>> keys > > >>>>>>> end > > >>>>>>>> result depends on. > > >>>>>>>> > > >>>>>>>> After that Committer Service has to do a cleanup (may be > > >> outside > > >>> of > > >>>>> the > > >>>>>>>> committing tx) to come to the following final state: > > >>>>>>>> > > >>>>>>>> [k1 => v1a] > > >>>>>>>> [k2 => v2ab] > > >>>>>>>> [k3 => v3b] > > >>>>>>>> > > >>>>>>>> Makes sense? > > >>>>>>>> > > >>>>>>>> Sergi > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > > >>>>> [hidden email] > > >>>>>>> : > > >>>>>>>> > > >>>>>>>>> - what do u mean by saying " > > >>>>>>>>> *in a single transaction checks value versions for all the > > >> old > > >>>>> values > > >>>>>>>>> and replaces them with calculated new ones *"? Every time > > >>> you > > >>>>>>> change > > >>>>>>>>> value(in some service), you store it to *some special > > >> atomic > > >>>>>> cache* > > >>>>>>> , > > >>>>>>>> so > > >>>>>>>>> when all services ceased working, Service commiter got a > > >>>> values > > >>>>>> with > > >>>>>>>> the > > >>>>>>>>> last versions. > > >>>>>>>>> - After "*does cleanup of temporary keys and values*" > > >>> Service > > >>>>>>> commiter > > >>>>>>>>> persists them into permanent store, isn't it ? > > >>>>>>>>> - I cant grasp your though, you say "*in case of version > > >>>>> mismatch > > >>>>>> or > > >>>>>>>> TX > > >>>>>>>>> timeout just rollbacks*". But what versions would it > > >> match? > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > > >>>>>> [hidden email] > > >>>>>>>> : > > >>>>>>>>> > > >>>>>>>>>> Ok, here is what you actually need to implement at the > > >>>>> application > > >>>>>>>> level. > > >>>>>>>>>> > > >>>>>>>>>> Lets say we have to call 2 services in the following order: > > >>>>>>>>>> - Service A: wants to update keys [k1 => v1, k2 => v2] > > >> to > > >>>>> [k1 > > >>>>>> => > > >>>>>>>>> v1a, > > >>>>>>>>>> k2 => v2a] > > >>>>>>>>>> - Service B: wants to update keys [k2 => v2a, k3 => v3] > > >> to > > >>>> [k2 > > >>>>>> => > > >>>>>>>>> v2ab, > > >>>>>>>>>> k3 => v3b] > > >>>>>>>>>> > > >>>>>>>>>> The change > > >>>>>>>>>> from [ k1 => v1, k2 => v2, k3 => v3 ] > > >>>>>>>>>> to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > > >>>>>>>>>> must happen in a single transaction. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Optimistic protocol to solve this: > > >>>>>>>>>> > > >>>>>>>>>> Each cache key must have a field `otx`, which is a unique > > >>>>>>> orchestrator > > >>>>>>>> TX > > >>>>>>>>>> identifier - it must be a parameter passed to all the > > >>> services. > > >>>>> If > > >>>>>>>> `otx` > > >>>>>>>>> is > > >>>>>>>>>> set to some value it means that it is an intermediate key > > >> and > > >>>> is > > >>>>>>>> visible > > >>>>>>>>>> only inside of some transaction, for the finalized key > > >> `otx` > > >>>> must > > >>>>>> be > > >>>>>>>>> null - > > >>>>>>>>>> it means the key is committed and visible for everyone. > > >>>>>>>>>> > > >>>>>>>>>> Each cache value must have a field `ver` which is a version > > >>> of > > >>>>> that > > >>>>>>>>> value. > > >>>>>>>>>> > > >>>>>>>>>> For both fields (`otx` and `ver`) the safest way is to use > > >>>> UUID. > > >>>>>>>>>> > > >>>>>>>>>> Workflow is the following: > > >>>>>>>>>> > > >>>>>>>>>> Orchestrator starts the distributed transaction with `otx` > > >> = > > >>> x > > >>>>> and > > >>>>>>>> passes > > >>>>>>>>>> this parameter to all the services. > > >>>>>>>>>> > > >>>>>>>>>> Service A: > > >>>>>>>>>> - does some computations > > >>>>>>>>>> - stores [k1x => v1a, k2x => v2a] with TTL = Za > > >>>>>>>>>> where > > >>>>>>>>>> Za - left time from max Orchestrator TX duration > > >>>> after > > >>>>>>>> Service > > >>>>>>>>> A > > >>>>>>>>>> end > > >>>>>>>>>> k1x, k2x - new temporary keys with field `otx` = > > >> x > > >>>>>>>>>> v2a has updated version `ver` > > >>>>>>>>>> - returns a set of updated keys and all the old versions > > >> to > > >>>> the > > >>>>>>>>>> orchestrator > > >>>>>>>>>> or just stores it in some special atomic cache like > > >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > > >>>>>>>>>> > > >>>>>>>>>> Service B: > > >>>>>>>>>> - retrieves the updated value k2x => v2a because it knows > > >>>> `otx` > > >>>>> = > > >>>>>> x > > >>>>>>>>>> - does computations > > >>>>>>>>>> - stores [k2x => v2ab, k3x => v3b] TTL = Zb > > >>>>>>>>>> - updates the set of updated keys like [x => (k1 -> ver1, > > >> k2 > > >>>> -> > > >>>>>>> ver2, > > >>>>>>>> k3 > > >>>>>>>>>> -> ver3)] TTL = Zb > > >>>>>>>>>> > > >>>>>>>>>> Service Committer (may be embedded into Orchestrator): > > >>>>>>>>>> - takes all the updated keys and versions for `otx` = x > > >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > > >>>>>>>>>> - in a single transaction checks value versions for all > > >> the > > >>>> old > > >>>>>>> values > > >>>>>>>>>> and replaces them with calculated new ones > > >>>>>>>>>> - does cleanup of temporary keys and values > > >>>>>>>>>> - in case of version mismatch or TX timeout just rollbacks > > >>> and > > >>>>>>> signals > > >>>>>>>>>> to Orchestrator to restart the job with new `otx` > > >>>>>>>>>> > > >>>>>>>>>> PROFIT!! > > >>>>>>>>>> > > >>>>>>>>>> This approach even allows you to run independent parts of > > >> the > > >>>>> graph > > >>>>>>> in > > >>>>>>>>>> parallel (with TX transfer you will always run only one at > > >> a > > >>>>> time). > > >>>>>>>> Also > > >>>>>>>>> it > > >>>>>>>>>> does not require inventing any special fault tolerance > > >>> technics > > >>>>>>> because > > >>>>>>>>>> Ignite caches are already fault tolerant and all the > > >>>> intermediate > > >>>>>>>> results > > >>>>>>>>>> are virtually invisible and stored with TTL, thus in case > > >> of > > >>>> any > > >>>>>>> crash > > >>>>>>>>> you > > >>>>>>>>>> will not have inconsistent state or garbage. > > >>>>>>>>>> > > >>>>>>>>>> Sergi > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > > >>>>>>> [hidden email] > > >>>>>>>>> : > > >>>>>>>>>> > > >>>>>>>>>>> Okay, we are open for proposals on business task. I mean, > > >>> we > > >>>>> can > > >>>>>>> make > > >>>>>>>>> use > > >>>>>>>>>>> of some other thing, not distributed transaction. Not > > >>>>> transaction > > >>>>>>>> yet. > > >>>>>>>>>>> > > >>>>>>>>>>> ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > > >>>>>> [hidden email] > > >>>>>>>> : > > >>>>>>>>>>> > > >>>>>>>>>>>> IMO the use case makes sense. However, as Sergi already > > >>>>>>> mentioned, > > >>>>>>>>> the > > >>>>>>>>>>>> problem is far more complex, than simply passing TX > > >> state > > >>>>> over > > >>>>>> a > > >>>>>>>>> wire. > > >>>>>>>>>>> Most > > >>>>>>>>>>>> probably a kind of coordinator will be required still > > >> to > > >>>>> manage > > >>>>>>> all > > >>>>>>>>>> kinds > > >>>>>>>>>>>> of failures. This task should be started with clean > > >>> design > > >>>>>>> proposal > > >>>>>>>>>>>> explaining how we handle all these concurrent events. > > >> And > > >>>>> only > > >>>>>>>> then, > > >>>>>>>>>> when > > >>>>>>>>>>>> we understand all implications, we should move to > > >>>> development > > >>>>>>>> stage. > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV < > > >>>>>>>>>>>> [hidden email]> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Right > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > > >>>>>>>>>> [hidden email] > > >>>>>>>>>>>> : > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Good! Basically your orchestrator just takes some > > >>>>>> predefined > > >>>>>>>>> graph > > >>>>>>>>>> of > > >>>>>>>>>>>>>> distributed services to be invoked, calls them by > > >>> some > > >>>>> kind > > >>>>>>> of > > >>>>>>>>> RPC > > >>>>>>>>>>> and > > >>>>>>>>>>>>>> passes the needed parameters between them, right? > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Sergi > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > > >>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>> : > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> orchestrator is a custom thing. He is responsible > > >>> for > > >>>>>>>> managing > > >>>>>>>>>>>> business > > >>>>>>>>>>>>>>> scenarios flows. Many nodes are involved in > > >>>> scenarios. > > >>>>>> They > > >>>>>>>>>>> exchange > > >>>>>>>>>>>>> data > > >>>>>>>>>>>>>>> and folow one another. If you acquinted with BPMN > > >>>>>>> framework, > > >>>>>>>> so > > >>>>>>>>>>>>>>> orchestrator is like bpmn engine. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > > >>>>>>>>>> [hidden email] > > >>>>>>>>>>>> : > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> What is Orchestrator for you? Is it a thing > > >> from > > >>>>>>> Microsoft > > >>>>>>>> or > > >>>>>>>>>>> your > > >>>>>>>>>>>>>> custom > > >>>>>>>>>>>>>>>> in-house software? > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Sergi > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > > >>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Fine. Let's say we've got multiple servers > > >>> which > > >>>>>>> fulfills > > >>>>>>>>>>> custom > > >>>>>>>>>>>>>> logic. > > >>>>>>>>>>>>>>>>> This servers compound oriented graph (BPMN > > >>>> process) > > >>>>>>> which > > >>>>>>>>>>>>> controlled > > >>>>>>>>>>>>>> by > > >>>>>>>>>>>>>>>>> Orchestrator. > > >>>>>>>>>>>>>>>>> For instance, *server1 *creates *variable A > > >>>> *with > > >>>>>>> value > > >>>>>>>> 1, > > >>>>>>>>>>>>> persists > > >>>>>>>>>>>>>> it > > >>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>> IGNITE cache and creates *variable B *and > > >> sends > > >>>> it > > >>>>>> to* > > >>>>>>>>>> server2. > > >>>>>>>>>>>>> *The > > >>>>>>>>>>>>>>>>> latests receives *variable B*, do some logic > > >>> with > > >>>>> it > > >>>>>>> and > > >>>>>>>>>> stores > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>>>> IGNITE. > > >>>>>>>>>>>>>>>>> All the work made by both servers must be > > >>>> fulfilled > > >>>>>> in > > >>>>>>>>> *one* > > >>>>>>>>>>>>>>> transaction. > > >>>>>>>>>>>>>>>>> Because we need all information done, or > > >>>>>>>>> nothing(rollbacked). > > >>>>>>>>>>> The > > >>>>>>>>>>>>>>>> scenario > > >>>>>>>>>>>>>>>>> is managed by orchestrator. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > > >>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Ok, it is not a business case, it is your > > >>> wrong > > >>>>>>>> solution > > >>>>>>>>>> for > > >>>>>>>>>>>> it. > > >>>>>>>>>>>>>>>>>> Lets try again, what is the business case? > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Sergi > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> 2017-03-14 16:42 GMT+03:00 ALEKSEY > > >> KUZNETSOV > > >>> < > > >>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> The case is the following, One starts > > >>>>> transaction > > >>>>>>> in > > >>>>>>>>> one > > >>>>>>>>>>>> node, > > >>>>>>>>>>>>>> and > > >>>>>>>>>>>>>>>>> commit > > >>>>>>>>>>>>>>>>>>> this transaction in another jvm node(or > > >>>>> rollback > > >>>>>> it > > >>>>>>>>>>>> remotely). > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 16:30, Sergi > > >>> Vladykin < > > >>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Because even if you make it work for > > >> some > > >>>>>>>> simplistic > > >>>>>>>>>>>>> scenario, > > >>>>>>>>>>>>>>> get > > >>>>>>>>>>>>>>>>>> ready > > >>>>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>> write many fault tolerance tests and > > >> make > > >>>>> sure > > >>>>>>> that > > >>>>>>>>> you > > >>>>>>>>>>> TXs > > >>>>>>>>>>>>>> work > > >>>>>>>>>>>>>>>>>>> gracefully > > >>>>>>>>>>>>>>>>>>>> in all modes in case of crashes. Also > > >>> make > > >>>>> sure > > >>>>>>>> that > > >>>>>>>>> we > > >>>>>>>>>>> do > > >>>>>>>>>>>>> not > > >>>>>>>>>>>>>>> have > > >>>>>>>>>>>>>>>>> any > > >>>>>>>>>>>>>>>>>>>> performance drops after all your > > >> changes > > >>> in > > >>>>>>>> existing > > >>>>>>>>>>>>>> benchmarks. > > >>>>>>>>>>>>>>>> All > > >>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>> all > > >>>>>>>>>>>>>>>>>>>> I don't believe these conditions will > > >> be > > >>>> met > > >>>>>> and > > >>>>>>>> your > > >>>>>>>>>>>>>>> contribution > > >>>>>>>>>>>>>>>>> will > > >>>>>>>>>>>>>>>>>>> be > > >>>>>>>>>>>>>>>>>>>> accepted. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Better solution to what problem? > > >> Sending > > >>> TX > > >>>>> to > > >>>>>>>>> another > > >>>>>>>>>>>> node? > > >>>>>>>>>>>>>> The > > >>>>>>>>>>>>>>>>>> problem > > >>>>>>>>>>>>>>>>>>>> statement itself is already wrong. What > > >>>>>> business > > >>>>>>>> case > > >>>>>>>>>> you > > >>>>>>>>>>>> are > > >>>>>>>>>>>>>>>> trying > > >>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>> solve? I'm sure everything you need can > > >>> be > > >>>>> done > > >>>>>>> in > > >>>>>>>> a > > >>>>>>>>>> much > > >>>>>>>>>>>>> more > > >>>>>>>>>>>>>>>> simple > > >>>>>>>>>>>>>>>>>> and > > >>>>>>>>>>>>>>>>>>>> efficient way at the application level. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Sergi > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> 2017-03-14 16:03 GMT+03:00 ALEKSEY > > >>>> KUZNETSOV > > >>>>> < > > >>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Why wrong ? You know the better > > >>> solution? > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 15:46, Sergi > > >>>>> Vladykin < > > >>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Just serializing TX object and > > >>>>>> deserializing > > >>>>>>> it > > >>>>>>>>> on > > >>>>>>>>>>>>> another > > >>>>>>>>>>>>>>> node > > >>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>> meaningless, because other nodes > > >>>>>>> participating > > >>>>>>>> in > > >>>>>>>>>> the > > >>>>>>>>>>>> TX > > >>>>>>>>>>>>>> have > > >>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>> know > > >>>>>>>>>>>>>>>>>>>>> about > > >>>>>>>>>>>>>>>>>>>>>> the new coordinator. This will > > >>> require > > >>>>>>> protocol > > >>>>>>>>>>>> changes, > > >>>>>>>>>>>>> we > > >>>>>>>>>>>>>>>>>>> definitely > > >>>>>>>>>>>>>>>>>>>>> will > > >>>>>>>>>>>>>>>>>>>>>> have fault tolerance and > > >> performance > > >>>>>> issues. > > >>>>>>>> IMO > > >>>>>>>>>> the > > >>>>>>>>>>>>> whole > > >>>>>>>>>>>>>>> idea > > >>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>> wrong > > >>>>>>>>>>>>>>>>>>>>>> and it makes no sense to waste time > > >>> on > > >>>>> it. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Sergi > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> 2017-03-14 10:57 GMT+03:00 ALEKSEY > > >>>>>> KUZNETSOV > > >>>>>>> < > > >>>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> IgniteTransactionState > > >>>> implememntation > > >>>>>>>> contains > > >>>>>>>>>>>>>>>> IgniteTxEntry's > > >>>>>>>>>>>>>>>>>>> which > > >>>>>>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>>> supposed to be transferable > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 19:32, > > >>> Dmitriy > > >>>>>>>> Setrakyan > > >>>>>>>>> < > > >>>>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> It sounds a little scary to me > > >>> that > > >>>>> we > > >>>>>>> are > > >>>>>>>>>>> passing > > >>>>>>>>>>>>>>>>> transaction > > >>>>>>>>>>>>>>>>>>>>> objects > > >>>>>>>>>>>>>>>>>>>>>>>> around. Such object may contain > > >>> all > > >>>>>> sorts > > >>>>>>>> of > > >>>>>>>>>>> Ignite > > >>>>>>>>>>>>>>>> context. > > >>>>>>>>>>>>>>>>> If > > >>>>>>>>>>>>>>>>>>>> some > > >>>>>>>>>>>>>>>>>>>>>> data > > >>>>>>>>>>>>>>>>>>>>>>>> needs to be passed across, we > > >>>> should > > >>>>>>>> create a > > >>>>>>>>>>>> special > > >>>>>>>>>>>>>>>>> transfer > > >>>>>>>>>>>>>>>>>>>> object > > >>>>>>>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>>>>> this case. > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> D. > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 13, 2017 at 9:10 > > >> AM, > > >>>>>> ALEKSEY > > >>>>>>>>>>> KUZNETSOV > > >>>>>>>>>>>> < > > >>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> well, there a couple of > > >> issues > > >>>>>>> preventing > > >>>>>>>>>>>>> transaction > > >>>>>>>>>>>>>>>>>>> proceeding. > > >>>>>>>>>>>>>>>>>>>>>>>>> At first, After transaction > > >>>>>>> serialization > > >>>>>>>>> and > > >>>>>>>>>>>>>>>>> deserialization > > >>>>>>>>>>>>>>>>>>> on > > >>>>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>>> remote > > >>>>>>>>>>>>>>>>>>>>>>>>> server, there is no txState. > > >> So > > >>>> im > > >>>>>>> going > > >>>>>>>> to > > >>>>>>>>>> put > > >>>>>>>>>>>> it > > >>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> writeExternal()\readExternal() > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> The last one is Deserialized > > >>>>>>> transaction > > >>>>>>>>>> lacks > > >>>>>>>>>>> of > > >>>>>>>>>>>>>>> shared > > >>>>>>>>>>>>>>>>>> cache > > >>>>>>>>>>>>>>>>>>>>>> context > > >>>>>>>>>>>>>>>>>>>>>>>>> field at > > >> TransactionProxyImpl. > > >>>>>> Perhaps, > > >>>>>>>> it > > >>>>>>>>>> must > > >>>>>>>>>>>> be > > >>>>>>>>>>>>>>>> injected > > >>>>>>>>>>>>>>>>>> by > > >>>>>>>>>>>>>>>>>>>>>>>>> GridResourceProcessor ? > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 17:27, > > >>>>> ALEKSEY > > >>>>>>>>>> KUZNETSOV > > >>>>>>>>>>> < > > >>>>>>>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> while starting and > > >> continuing > > >>>>>>>> transaction > > >>>>>>>>>> in > > >>>>>>>>>>>>>>> different > > >>>>>>>>>>>>>>>>> jvms > > >>>>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>> run > > >>>>>>>>>>>>>>>>>>>>>>> into > > >>>>>>>>>>>>>>>>>>>>>>>>>> serialization exception in > > >>>>>>>>>> writeExternalMeta > > >>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> @Override public void > > >>>>>>>>>>>> writeExternal(ObjectOutput > > >>>>>>>>>>>>>> out) > > >>>>>>>>>>>>>>>>>> throws > > >>>>>>>>>>>>>>>>>>>>>>>> IOException > > >>>>>>>>>>>>>>>>>>>>>>>>> { > > >>>>>>>>>>>>>>>>>>>>>>>>>> writeExternalMeta(out); > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> some meta is cannot be > > >>>>> serialized. > > >>>>>>>>>>>>>>>>>>>>>>>>>> пт, 10 мар. 2017 г. в > > >> 17:25, > > >>>>> Alexey > > >>>>>>>>>>> Goncharuk < > > >>>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Aleksey, > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I think I am starting to > > >> get > > >>>> what > > >>>>>> you > > >>>>>>>>> want, > > >>>>>>>>>>>> but I > > >>>>>>>>>>>>>>> have > > >>>>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>>> few > > >>>>>>>>>>>>>>>>>>>>>>> concerns: > > >>>>>>>>>>>>>>>>>>>>>>>>>> - What is the API for the > > >>>>> proposed > > >>>>>>>>> change? > > >>>>>>>>>>> In > > >>>>>>>>>>>>> your > > >>>>>>>>>>>>>>>> test, > > >>>>>>>>>>>>>>>>>> you > > >>>>>>>>>>>>>>>>>>>>> pass > > >>>>>>>>>>>>>>>>>>>>>> an > > >>>>>>>>>>>>>>>>>>>>>>>>>> instance of transaction > > >>> created > > >>>>> on > > >>>>>>>>>> ignite(0) > > >>>>>>>>>>> to > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>> ignite > > >>>>>>>>>>>>>>>>>>>>> instance > > >>>>>>>>>>>>>>>>>>>>>>>>>> ignite(1). This is > > >> obviously > > >>>> not > > >>>>>>>> possible > > >>>>>>>>>> in > > >>>>>>>>>>> a > > >>>>>>>>>>>>>> truly > > >>>>>>>>>>>>>>>>>>>> distributed > > >>>>>>>>>>>>>>>>>>>>>>>>>> (multi-jvm) environment. > > >>>>>>>>>>>>>>>>>>>>>>>>>> - How will you synchronize > > >>>> cache > > >>>>>>> update > > >>>>>>>>>>> actions > > >>>>>>>>>>>>> and > > >>>>>>>>>>>>>>>>>>> transaction > > >>>>>>>>>>>>>>>>>>>>>>> commit? > > >>>>>>>>>>>>>>>>>>>>>>>>>> Say, you have one node that > > >>>>> decided > > >>>>>>> to > > >>>>>>>>>>> commit, > > >>>>>>>>>>>>> but > > >>>>>>>>>>>>>>>>> another > > >>>>>>>>>>>>>>>>>>> node > > >>>>>>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>>>> still > > >>>>>>>>>>>>>>>>>>>>>>>>>> writing within this > > >>>> transaction. > > >>>>>> How > > >>>>>>> do > > >>>>>>>>> you > > >>>>>>>>>>>> make > > >>>>>>>>>>>>>> sure > > >>>>>>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>>> two > > >>>>>>>>>>>>>>>>>>>>>> nodes > > >>>>>>>>>>>>>>>>>>>>>>>> will > > >>>>>>>>>>>>>>>>>>>>>>>>>> not call commit() and > > >>>> rollback() > > >>>>>>>>>>>> simultaneously? > > >>>>>>>>>>>>>>>>>>>>>>>>>> - How do you make sure > > >> that > > >>>>> either > > >>>>>>>>>> commit() > > >>>>>>>>>>> or > > >>>>>>>>>>>>>>>>> rollback() > > >>>>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>>>> called > > >>>>>>>>>>>>>>>>>>>>>>> if > > >>>>>>>>>>>>>>>>>>>>>>>>> an > > >>>>>>>>>>>>>>>>>>>>>>>>>> originator failed? > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:38 GMT+03:00 > > >>>>> Дмитрий > > >>>>>>>> Рябов > > >>>>>>>>> < > > >>>>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Alexey Goncharuk, heh, my > > >>>>> initial > > >>>>>>>>>>>> understanding > > >>>>>>>>>>>>>> was > > >>>>>>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>>>>>>> transferring > > >>>>>>>>>>>>>>>>>>>>>>>>> of > > >>>>>>>>>>>>>>>>>>>>>>>>>> tx > > >>>>>>>>>>>>>>>>>>>>>>>>>>> ownership from one node > > >> to > > >>>>>> another > > >>>>>>>> will > > >>>>>>>>>> be > > >>>>>>>>>>>>>> happened > > >>>>>>>>>>>>>>>>>>>>> automatically > > >>>>>>>>>>>>>>>>>>>>>>>> when > > >>>>>>>>>>>>>>>>>>>>>>>>>>> originating node is gone > > >>>> down. > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:36 > > >> GMT+03:00 > > >>>>>> ALEKSEY > > >>>>>>>>>>> KUZNETSOV > > >>>>>>>>>>>> < > > >>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Im aiming to span > > >>>> transaction > > >>>>>> on > > >>>>>>>>>> multiple > > >>>>>>>>>>>>>>> threads, > > >>>>>>>>>>>>>>>>>> nodes, > > >>>>>>>>>>>>>>>>>>>>>>>> jvms(soon). > > >>>>>>>>>>>>>>>>>>>>>>>>>> So > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> every node is able to > > >>>>> rollback, > > >>>>>>> or > > >>>>>>>>>> commit > > >>>>>>>>>>>>>> common > > >>>>>>>>>>>>>>>>>>>>> transaction.It > > >>>>>>>>>>>>>>>>>>>>>>>>> turned > > >>>>>>>>>>>>>>>>>>>>>>>>>>> up i > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> need to transfer tx > > >>> between > > >>>>>> nodes > > >>>>>>>> in > > >>>>>>>>>>> order > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>>>> commit > > >>>>>>>>>>>>>>>>>>>>>> transaction > > >>>>>>>>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> different node(in the > > >>> same > > >>>>>> jvm). > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> пт, 10 мар. 2017 г. в > > >>>> 15:20, > > >>>>>>> Alexey > > >>>>>>>>>>>>> Goncharuk < > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> [hidden email] > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aleksey, > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do you mean that you > > >>>> want a > > >>>>>>>> concept > > >>>>>>>>>> of > > >>>>>>>>>>>>>>>> transferring > > >>>>>>>>>>>>>>>>>> of > > >>>>>>>>>>>>>>>>>>> tx > > >>>>>>>>>>>>>>>>>>>>>>>> ownership > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> one node to another? > > >> My > > >>>>>> initial > > >>>>>>>>>>>>> understanding > > >>>>>>>>>>>>>>> was > > >>>>>>>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>>>> you > > >>>>>>>>>>>>>>>>>>>>>> want > > >>>>>>>>>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>>>>>>> be > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> able > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to update keys in a > > >>>>>> transaction > > >>>>>>>>> from > > >>>>>>>>>>>>> multiple > > >>>>>>>>>>>>>>>>> threads > > >>>>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>>>>> parallel. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --AG > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-03-10 15:01 > > >>>> GMT+03:00 > > >>>>>>>> ALEKSEY > > >>>>>>>>>>>>> KUZNETSOV > > >>>>>>>>>>>>>> < > > >>>>>>>>>>>>>>>>>>>>>>>>>> [hidden email] > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Well. Consider > > >>>>> transaction > > >>>>>>>>> started > > >>>>>>>>>> in > > >>>>>>>>>>>> one > > >>>>>>>>>>>>>>> node, > > >>>>>>>>>>>>>>>>> and > > >>>>>>>>>>>>>>>>>>>>>> continued > > >>>>>>>>>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> another > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one. > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following test > > >>>>>> describes > > >>>>>>> my > > >>>>>>>>>> idea: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ignite ignite1 = > > >>>>> ignite(0); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteTransactions > > >>>>>>>> transactions = > > >>>>>>>>>>>>>>>>>>>> ignite1.transactions(); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteCache<String, > > >>>>>> Integer> > > >>>>>>>>> cache > > >>>>>>>>>> = > > >>>>>>>>>>>>>>>>>>>>>>> ignite1.getOrCreateCache(" > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> testCache"); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Transaction tx = > > >>>>>>>>>>> transactions.txStart( > > >>>>>>>>>>>>>>>>> concurrency, > > >>>>>>>>>>>>>>>>>>>>>>> isolation); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cache.put("key1", > > >> 1); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cache.put("key2", > > >> 2); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tx.stop(); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>> IgniteInternalFuture<Boolean> > > >>>>>>>>> fut = > > >>>>>>>>>>>>>>>>>>>>>> GridTestUtils.runAsync(() > > >>>>>>>>>>>>>>>>>>>>>>>> -> > > >>>>>>>>>>>>>>>>>>>>>>>>> { > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>> IgniteTransactions > > >>>>> ts = > > >>>>>>>>>>>>>>>>>> ignite(1).transactions(); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>> Assert.assertNull(ts.tx()); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>> Assert.assertEquals( > > >>>>>>>>>>>>>>>>> TransactionState.STOPPED, > > >>>>>>>>>>>>>>>>>>>>>>> tx.state()); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ts.txStart(tx); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Assert.assertEquals(TransactionState.ACTIVE, > > >>>>>>>>>>>>>>>>>>>>>>> tx.state()); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> cache.put("key3", > > >>>> 3); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Assert.assertTrue(cache.remove("key2")); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tx.commit(); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> return true; > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> }); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fut.get(); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> Assert.assertEquals( > > >>>>>>>>>>>>>>> TransactionState.COMMITTED, > > >>>>>>>>>>>>>>>>>>>>>> tx.state()); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>> Assert.assertEquals((long)1, > > >>>>>>>>>>>>>>>>>>> (long)cache.get("key1")); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>> Assert.assertEquals((long)3, > > >>>>>>>>>>>>>>>>>>> (long)cache.get("key3")); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>> Assert.assertFalse(cache. > > >>>>>>>>>>>>>>> containsKey("key2")); > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In method > > >>>>> *ts.txStart(...)* > > >>>>>>> we > > >>>>>>>>> just > > >>>>>>>>>>>>> rebind > > >>>>>>>>>>>>>>> *tx* > > >>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>>> current > > >>>>>>>>>>>>>>>>>>>>>>>>> thread: > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > > > -- > > *Best Regards,* > > *Kuznetsov Aleksey* > |
«
Return to Apache Ignite Developers - Legacy Mail Archive
|
1 view|%1 views
| Free forum by Nabble | Edit this page |