DataStreamer Transactional and Timestamp Implementation

[fatih]

Hi

We have implemented some receivers to be able to update data in data nodes by ignite datastreamer api.

There is an associated ticket as below already exist. We thought it would be useful to use that implementation from ignite directly. Maybe can be added to ignite directly.

http://apache-ignite-users.70518.x6.nabble.com/Transaction-Boundary-Data-Streamer-td13803.html#a14078
AbstractTransactionalStreamReceiver.java

TimestampBasedUpdateStreamReceiver.java

[dmagda]

Faith,

Thanks for sharing it with us.

In general, we can always wrap this code up in a form of an example to be delivered with every Ignite release.

But probably it makes sense to add it to the core streaming functionality. *Igniters*, what’d you think?

—
Denis

> On Jun 27, 2017, at 12:29 AM, fatih <[hidden email]> wrote:
>
> Hi
>
> We have implemented some receivers to be able to update data in data nodes
> by ignite datastreamer api.
>
> There is an associated ticket as below already exist. We thought it would be
> useful to use that implementation from ignite directly. Maybe can be added
> to ignite directly.
>
> http://apache-ignite-users.70518.x6.nabble.com/Transaction-Boundary-Data-Streamer-td13803.html#a14078
> AbstractTransactionalStreamReceiver.java
> <http://apache-ignite-developers.2346864.n4.nabble.com/file/n19129/AbstractTransactionalStreamReceiver.java>  
>
> TimestampBasedUpdateStreamReceiver.java
> <http://apache-ignite-developers.2346864.n4.nabble.com/file/n19129/TimestampBasedUpdateStreamReceiver.java>  
>
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/DataStreamer-Transactional-and-Timestamp-Implementation-tp19129.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[Valentin Kulichenko]

Fatih,

Can you give more details about the use case addressed by this
implementation?

-Val

On Tue, Jun 27, 2017 at 3:45 PM, Denis Magda <[hidden email]> wrote:

> Faith,
>
> Thanks for sharing it with us.
>
> In general, we can always wrap this code up in a form of an example to be
> delivered with every Ignite release.
>
> But probably it makes sense to add it to the core streaming functionality.
> *Igniters*, what’d you think?
>
> —
> Denis
>
> > On Jun 27, 2017, at 12:29 AM, fatih <[hidden email]> wrote:
> >
> > Hi
> >
> > We have implemented some receivers to be able to update data in data
> nodes
> > by ignite datastreamer api.
> >
> > There is an associated ticket as below already exist. We thought it
> would be
> > useful to use that implementation from ignite directly. Maybe can be
> added
> > to ignite directly.
> >
> > http://apache-ignite-users.70518.x6.nabble.com/
> Transaction-Boundary-Data-Streamer-td13803.html#a14078
> > AbstractTransactionalStreamReceiver.java
> > <http://apache-ignite-developers.2346864.n4.nabble.com/file/n19129/
> AbstractTransactionalStreamReceiver.java>
> >
> > TimestampBasedUpdateStreamReceiver.java
> > <http://apache-ignite-developers.2346864.n4.nabble.com/file/n19129/
> TimestampBasedUpdateStreamReceiver.java>
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-ignite-
> developers.2346864.n4.nabble.com/DataStreamer-Transactional-and-Timestamp-
> Implementation-tp19129.html
> > Sent from the Apache Ignite Developers mailing list archive at
> Nabble.com.
>
>

[fatih]

Hi

Sure. Thanks for your interest by the way.

We needed to update our cache objects concurrently and every time we do the update on cache objects, we wanted to calculate some results by comparing before and after the update. To make this use case fast enough we wanted to use IgniteDataStreamer to send the update events from Kafka as batches to the Ignite Server Nodes to do the calculation where the data is.

But we come across a concurrency issue where 2 different threads were updating the cache object with the same key. The last thread finishing was overriding the other threads changes. So we decided to use Deadlock Free Optimistic Transactional support from Ignite as it is much better performing than using pessimistic transactions. To achieve that we implemented AbstractTransactionalStreamReceiver. In this implementation, we are caching OptimisticTransactionalExceptions and then trying to reapply the changes by retrying up to a certain limit.

Later, we also had another problem which was related to the update sequence. Imagine you get update for fieldX for cacheObjectX at timeStamp=100 and then also get an update to the same fieldX and same cacheObjectX at timeStamp=200. In distributed environment, we don't know which update will run first. If the first update(timeStamp=100) happens first then we are safe but what happens if the second update(timeStamp=200) happens first then the timeStamp=100 update happens? In that case, we will override more recent data in our cacheObjectX. So to solve that we design our model to have "fieldX__timestamp"  so this applies to other columns as well. First, check the update is more recent if not then don't do the update. To achieve that we implemented TimestampBasedUpdateStreamReceiver class.

I hope that I managed to explain, please let us know if you have questions?
 

[fatih]

Hi

May I kindly ask if you had a chance to look into the implementation and the use case description in our previous post

[Valentin Kulichenko]

Hi Fatih,

This makes sense to me, but frankly I don't see anything that can be
included in the product here. It's very specific to your case and doesn't
add much value in general.

Do you have a blog by any chance? :) It looks like a very good topic for an
article (describing the use case, proposing solution, etc.).

-Val

On Fri, Jun 30, 2017 at 11:17 AM, fatih <[hidden email]> wrote:

> Hi
>
> May I kindly ask if you had a chance to look into the implementation and
> the
> use case description in our previous post
>
>
>
> --
> View this message in context: http://apache-ignite-
> developers.2346864.n4.nabble.com/DataStreamer-Transactional-and-Timestamp-
> Implementation-tp19129p19312.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>