Re: [External] Re: Triggering cache updates in Ignite

[Bosko Devetak]

Hi Denis,

On Wed, Jun 28, 2017 at 12:13 AM, Denis Magda <[hidden email]> wrote:

Hello Daniel,
>
> That looks promising!
>
> So, in case with Apache Ignite we need to subscribe to Kafka to get all
> the updates while the Kafka will receive them from the replicator, right?
> Is there any way to provide a direct support for Ignite by implementing a
> dedicating replicator?
>

Short answer: Yes, that is relatively easy to achieve by implementing
Applier interface.

A bit of details: Basically replicator pipeline looks like this:

    [binlog parser] -> [schema decorator] -> [applier]

Currently there are three appliers implemented:

1. HBase
2. Kafka
3. STDOUT

They all implement the Applier interface. STDOUT applier is a simple
implementation used for debugging, but it is also a good place to start
when writing a new applier. We are open for pull requests and we can help
with advice.

The Applier Interface:

https://github.com/mysql-time-machine/replicator/blob/master/src/main/java/com/booking/replication/applier/Applier.java

STDOUT Applier Implementation:

https://github.com/mysql-time-machine/replicator/blob/master/src/main/java/com/booking/replication/applier/StdoutJsonApplier.java

When you run the replicator you can specify the applier type in startup
parameters:

java -jar mysql-replicator.jar \
    --applier STDOUT \
    --schema $schema \
    --binlog-filename $binlog-filename \
    --config-path $config-path


> In general, may I ask you to send this message to Ignite dev list or I can
> forward it there copying you? It makes sense to discuss the project with
> the whole community.
>

I have added  [hidden email]  to cc.

I am out of the office now, but we can talk more once I am back from 10th
of July.

Cheers,
Bosko.

>
> —
> Denis
>
> On Jun 22, 2017, at 6:08 AM, Daniël van Eeden <[hidden email]>
> wrote:
>
> Hello Denis,
>
>
> This is the project I mentioned at the MySQL meetup:
> https://mysql-time-machine.github.io/
>
> This allows you to consume a MySQL binlog stream, which then can be used
> to update the cache entries in Ignite when they are created, updated or
> removed in MySQL directly.
> This is opensource and *much* cheaper than Golden Gate.
>
>
> Cheers,
>
> --
> Daniël van Eeden
> Database Administrator
>
> Booking.com B.V.
> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
> Direct [phone] <+31%2020%20703%203812>
> [image: Booking.com] <http://www.booking.com/>
> The world's #1 accommodation site
> 43 languages, 187+ offices worldwide, 96,000+ global destinations,
> 1,200,000+ room nights booked every day
> No booking fees, best price always guaranteed
> Subsidiary of the Priceline Group (NASDAQ: PCLN)
>
>
>

--
Bosko Devetak
Senior Developer

Booking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
Direct [phone]
[image: Booking.com] <http://www.booking.com/>
The world's #1 accommodation site
43 languages, 199+ offices worldwide, 96,000+ global destinations,
1,400,000+ room nights booked every day
No booking fees, best price always guaranteed
Subsidiary of the Priceline Group (NASDAQ: PCLN)

[dmagda]

Hi Bosko,

Thanks for extra details! I’ve created a ticket in Ignite’s JIRA. Hope someone from the community gets interested in this and implements the feature:
https://issues.apache.org/jira/browse/IGNITE-5610

Igniters, in short, this integration will allow to send updates from MySQL to an Ignite cluster automatically. That might be useful when MySQL is used as a persistent store by Ignite and the database is being updated directly by some other application and all such updates have to end up in the cluster eventually.

Is the anyone who is willing to take over the task?


—
Denis

> On Jun 28, 2017, at 4:34 AM, Bosko Devetak <[hidden email]> wrote:
>
> Hi Denis,
>
> On Wed, Jun 28, 2017 at 12:13 AM, Denis Magda <[hidden email] <mailto:[hidden email]>> wrote:
>
> Hello Daniel,
>
> That looks promising!
>
> So, in case with Apache Ignite we need to subscribe to Kafka to get all the updates while the Kafka will receive them from the replicator, right? Is there any way to provide a direct support for Ignite by implementing a dedicating replicator?
>  
> Short answer: Yes, that is relatively easy to achieve by implementing Applier interface.
>
> A bit of details: Basically replicator pipeline looks like this:
>
>     [binlog parser] -> [schema decorator] -> [applier]
>
> Currently there are three appliers implemented:
>
> 1. HBase
> 2. Kafka
> 3. STDOUT
>
> They all implement the Applier interface. STDOUT applier is a simple implementation used for debugging, but it is also a good place to start when writing a new applier. We are open for pull requests and we can help with advice.
>
> The Applier Interface:
>
> https://github.com/mysql-time-machine/replicator/blob/master/src/main/java/com/booking/replication/applier/Applier.java <https://github.com/mysql-time-machine/replicator/blob/master/src/main/java/com/booking/replication/applier/Applier.java>
>
> STDOUT Applier Implementation:
>
> https://github.com/mysql-time-machine/replicator/blob/master/src/main/java/com/booking/replication/applier/StdoutJsonApplier.java <https://github.com/mysql-time-machine/replicator/blob/master/src/main/java/com/booking/replication/applier/StdoutJsonApplier.java>
>
> When you run the replicator you can specify the applier type in startup parameters:
>
> java -jar mysql-replicator.jar \
>     --applier STDOUT \
>     --schema $schema \
>     --binlog-filename $binlog-filename \
>     --config-path $config-path
>
>
> In general, may I ask you to send this message to Ignite dev list or I can forward it there copying you? It makes sense to discuss the project with the whole community.
>
> I have added  [hidden email] <mailto:[hidden email]>  to cc.
>
> I am out of the office now, but we can talk more once I am back from 10th of July.
>
> Cheers,
> Bosko.
>
>  
>
> —
> Denis
>
>> On Jun 22, 2017, at 6:08 AM, Daniël van Eeden <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> Hello Denis,
>>
>>
>> This is the project I mentioned at the MySQL meetup:
>> https://mysql-time-machine.github.io/ <https://mysql-time-machine.github.io/>
>>
>> This allows you to consume a MySQL binlog stream, which then can be used
>> to update the cache entries in Ignite when they are created, updated or removed in MySQL directly.
>> This is opensource and *much* cheaper than Golden Gate.
>>
>>
>> Cheers,
>>
>> --
>> Daniël van Eeden
>> Database Administrator
>>
>> Booking.com <http://booking.com/> B.V.
>> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
>> Direct [phone] <tel:+31%2020%20703%203812>
>>  <http://www.booking.com/>The world's #1 accommodation site
>> 43 languages, 187+ offices worldwide, 96,000+ global destinations, 1,200,000+ room nights booked every day
>> No booking fees, best price always guaranteed
>> Subsidiary of the Priceline Group (NASDAQ: PCLN)
>
>
>
>
> --
> Bosko Devetak
> Senior Developer
>
> Booking.com <http://booking.com/> B.V.
> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
> Direct [phone]
>  <http://www.booking.com/>The world's #1 accommodation site
> 43 languages, 199+ offices worldwide, 96,000+ global destinations, 1,400,000+ room nights booked every day
> No booking fees, best price always guaranteed
> Subsidiary of the Priceline Group (NASDAQ: PCLN)