Cassandra cache store [IGNITE-1371]

[Alexey Kuznetsov-2]

During review I faced the problem, that I have not enough experience to
answer.

So, it will be great, if someone who has deep experience with Binary
marshaller could help us.

---------------------------

> How about such limitations of Binary Marshaller:

> 1) Fields or types with the same name hash are not allowed.
> 2) BinaryObject format does not allow same field names on different
levels of a class hierarchy.
> 3) Externalizable interface is ignored by default. If BinaryObject format
is used, Externalizable
>  classes will be written the same way as if they were Serializable,
without writeExternal() and
> readExternal() methods. If for some reason this does not work for you,
you should implement
> Binarylizable interface for your classes, plug in a custom
BinarySerializer or switch to the
> OptimizedMarshaller.

> There are no such limitations in Kryo and Java serialization.

> The next concern is that you need *Ignite Core* module, which is rather
huge (about 7.3MB) if
>  you want to build ETL script which will consume data persisted into
Cassandra by Ignite Binary
>  Marshaller.

> By the way, does Ignite plan to support backward compatibility for data
persisted using Binary
>  Marshaller? I mean the situation when some objects were persisted into
Cassandra using old
>  version of Binary Marshaller and then Ignite cluster was upgraded to new
version. Kryo and
>  Java serialization handles this situation and provides backward
compatibility.

> May be it's better just to add one more serializer implementation which
will use Binary Marshaller?

> By the way are there any samples in the code how to use Binary Marshaller
just to
> serialize/deserialize arbitrary object? Binary Marshaller documentation
says that all such
>  operation performed internally inside Ignite when we using BinaryObject,
but in my case I am
>  interested in rather low-level serialization/deserialization API/

Regards,
Igor Rudyak

--------------------------------------------
Hi Igor!

I moved discussion to dev list.

>>> The next concern is that you need *Ignite Core* module
We could not avoid adding this module because All base cache store classes
declared in that module, so I think this module will *be always imported*.

As for other points, I hope community will help us.

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

[Valentin Kulichenko]

Alexey, Igor,

Binary format is internal format used by Ignite, it can't be used directly
or outside of Ignite. So there is no way and no reason to have special
binary serializer in Cassandra store implementation. If user wants to save
BinaryObject as a BLOB to the store (to later load it back to Ignite), he
can set CacheConfiguration.setStoreKeepBinary(true) flag and cache will
give BinaryObject to the store without deserializing it.

Actually, I'm looking at the current API and I don't like
PersistenceStrategy enum and Serializer interface. I understand what you
try to achieve here and it makes sense to me, but the abstraction is not
generic enough in my opinion. We give user only two options, but what if he
needs something else? For example, he may need to add special processing
for some specific fields (make some conversions, etc.).

I propose to change it a bit and do something like this:

interface PersistenceCallback {
    CassandraData onWrite(K key, V value);
    IgniteBiTuple<K, V> onRead(CassandraData);
}

CassandraData is an entity that Cassandra should be able to write and load
(essentially, set of column values). We can reuse something from Cassandra
API here if that's possible (I'm not familiar enough with Cassandra to make
specific suggestions right away) or create our own.

We will have a default implementation that will introspect POJOs. The one
that converts value to BLOB can be also provided out of the box. And user
is free to implement his own logic.

All namings and API itself are arguable, of course, but I hope that idea is
clear.

Thoughts?

On Thu, Jan 28, 2016 at 12:41 AM, Alexey Kuznetsov <[hidden email]>
wrote:

> During review I faced the problem, that I have not enough experience to
> answer.
>
> So, it will be great, if someone who has deep experience with Binary
> marshaller could help us.
>
> ---------------------------
>
> > How about such limitations of Binary Marshaller:
>
> > 1) Fields or types with the same name hash are not allowed.
> > 2) BinaryObject format does not allow same field names on different
> levels of a class hierarchy.
> > 3) Externalizable interface is ignored by default. If BinaryObject format
> is used, Externalizable
> >  classes will be written the same way as if they were Serializable,
> without writeExternal() and
> > readExternal() methods. If for some reason this does not work for you,
> you should implement
> > Binarylizable interface for your classes, plug in a custom
> BinarySerializer or switch to the
> > OptimizedMarshaller.
>
> > There are no such limitations in Kryo and Java serialization.
>
> > The next concern is that you need *Ignite Core* module, which is rather
> huge (about 7.3MB) if
> >  you want to build ETL script which will consume data persisted into
> Cassandra by Ignite Binary
> >  Marshaller.
>
> > By the way, does Ignite plan to support backward compatibility for data
> persisted using Binary
> >  Marshaller? I mean the situation when some objects were persisted into
> Cassandra using old
> >  version of Binary Marshaller and then Ignite cluster was upgraded to new
> version. Kryo and
> >  Java serialization handles this situation and provides backward
> compatibility.
>
> > May be it's better just to add one more serializer implementation which
> will use Binary Marshaller?
>
> > By the way are there any samples in the code how to use Binary Marshaller
> just to
> > serialize/deserialize arbitrary object? Binary Marshaller documentation
> says that all such
> >  operation performed internally inside Ignite when we using BinaryObject,
> but in my case I am
> >  interested in rather low-level serialization/deserialization API/
>
> Regards,
> Igor Rudyak
>
> --------------------------------------------
> Hi Igor!
>
> I moved discussion to dev list.
>
> >>> The next concern is that you need *Ignite Core* module
> We could not avoid adding this module because All base cache store classes
> declared in that module, so I think this module will *be always imported*.
>
> As for other points, I hope community will help us.
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>

[Alexey Kuznetsov-2]

I like Val suggestion.

Also in this case we could move external things like Kryo into separate
module/dependency (for those who do not want to depend on Kryo).
This could be "ignite-cassandra-kryo" module that will contain smth. like
"KryoPersistenceCallback".

Thoughts?


On Fri, Jan 29, 2016 at 10:27 AM, Valentin Kulichenko <
[hidden email]> wrote:

> Alexey, Igor,
>
> Binary format is internal format used by Ignite, it can't be used directly
> or outside of Ignite. So there is no way and no reason to have special
> binary serializer in Cassandra store implementation. If user wants to save
> BinaryObject as a BLOB to the store (to later load it back to Ignite), he
> can set CacheConfiguration.setStoreKeepBinary(true) flag and cache will
> give BinaryObject to the store without deserializing it.
>
> Actually, I'm looking at the current API and I don't like
> PersistenceStrategy enum and Serializer interface. I understand what you
> try to achieve here and it makes sense to me, but the abstraction is not
> generic enough in my opinion. We give user only two options, but what if he
> needs something else? For example, he may need to add special processing
> for some specific fields (make some conversions, etc.).
>
> I propose to change it a bit and do something like this:
>
> interface PersistenceCallback {
>     CassandraData onWrite(K key, V value);
>     IgniteBiTuple<K, V> onRead(CassandraData);
> }
>
> CassandraData is an entity that Cassandra should be able to write and load
> (essentially, set of column values). We can reuse something from Cassandra
> API here if that's possible (I'm not familiar enough with Cassandra to make
> specific suggestions right away) or create our own.
>
> We will have a default implementation that will introspect POJOs. The one
> that converts value to BLOB can be also provided out of the box. And user
> is free to implement his own logic.
>
> All namings and API itself are arguable, of course, but I hope that idea is
> clear.
>
> Thoughts?
>
> On Thu, Jan 28, 2016 at 12:41 AM, Alexey Kuznetsov <
> [hidden email]>
> wrote:
>
> > During review I faced the problem, that I have not enough experience to
> > answer.
> >
> > So, it will be great, if someone who has deep experience with Binary
> > marshaller could help us.
> >
> > ---------------------------
> >
> > > How about such limitations of Binary Marshaller:
> >
> > > 1) Fields or types with the same name hash are not allowed.
> > > 2) BinaryObject format does not allow same field names on different
> > levels of a class hierarchy.
> > > 3) Externalizable interface is ignored by default. If BinaryObject
> format
> > is used, Externalizable
> > >  classes will be written the same way as if they were Serializable,
> > without writeExternal() and
> > > readExternal() methods. If for some reason this does not work for you,
> > you should implement
> > > Binarylizable interface for your classes, plug in a custom
> > BinarySerializer or switch to the
> > > OptimizedMarshaller.
> >
> > > There are no such limitations in Kryo and Java serialization.
> >
> > > The next concern is that you need *Ignite Core* module, which is rather
> > huge (about 7.3MB) if
> > >  you want to build ETL script which will consume data persisted into
> > Cassandra by Ignite Binary
> > >  Marshaller.
> >
> > > By the way, does Ignite plan to support backward compatibility for data
> > persisted using Binary
> > >  Marshaller? I mean the situation when some objects were persisted into
> > Cassandra using old
> > >  version of Binary Marshaller and then Ignite cluster was upgraded to
> new
> > version. Kryo and
> > >  Java serialization handles this situation and provides backward
> > compatibility.
> >
> > > May be it's better just to add one more serializer implementation which
> > will use Binary Marshaller?
> >
> > > By the way are there any samples in the code how to use Binary
> Marshaller
> > just to
> > > serialize/deserialize arbitrary object? Binary Marshaller documentation
> > says that all such
> > >  operation performed internally inside Ignite when we using
> BinaryObject,
> > but in my case I am
> > >  interested in rather low-level serialization/deserialization API/
> >
> > Regards,
> > Igor Rudyak
> >
> > --------------------------------------------
> > Hi Igor!
> >
> > I moved discussion to dev list.
> >
> > >>> The next concern is that you need *Ignite Core* module
> > We could not avoid adding this module because All base cache store
> classes
> > declared in that module, so I think this module will *be always
> imported*.
> >
> > As for other points, I hope community will help us.
> >
> > --
> > Alexey Kuznetsov
> > GridGain Systems
> > www.gridgain.com
> >
>

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

[irudyak]

Actually the idea of PersistenceCallback interface looks good. I'll try to think how it's better to implement CassandraData class.

[dsetrakyan]

Would be nice to get it into the next release.

On Wed, Feb 17, 2016 at 10:40 PM, irudyak <[hidden email]> wrote:

> Actually the idea of PersistenceCallback interface looks good. I'll try to
> think how it's better to implement CassandraData class.
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7436.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

[Denis Magda]

I think it makes sense to make this feature the one that drives the next
release.

There is a discussion started by Yakov on this list named "Ignite 1.6
release timelines".
https://mail-archives.apache.org/mod_mbox/ignite-dev/201602.mbox/%3CCAGcMBHiS-9SaqURK8pKQ98VKh-Ynoq_y723%2BotLDuNJJAjKgow%40mail.gmail.com%3E

Igor, could you ask to consider this ticket for the next release as well
answering in that thread?

--
Denis

On 2/18/2016 10:37 AM, Dmitriy Setrakyan wrote:

> Would be nice to get it into the next release.
>
> On Wed, Feb 17, 2016 at 10:40 PM, irudyak <[hidden email]> wrote:
>
>> Actually the idea of PersistenceCallback interface looks good. I'll try to
>> think how it's better to implement CassandraData class.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7436.html
>> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>>

[irudyak]

Well, it just depends on how soon you are going to publish release 1.6. It will take about 1 month for me to implement PersistenceCallback.

[Denis Magda]

I'm quite sure that 1.6 release won't be ready in a month. So you have a
plenty of time to finish your ticket ;)


On 2/19/2016 10:54 AM, irudyak wrote:
> Well, it just depends on how soon you are going to publish release 1.6. It
> will take about 1 month for me to implement PersistenceCallback.
>
>
>
> --
> View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7466.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

[irudyak]

Hi guys, I am quite busy now. It looks like I'll just start implementing PersistenceCallback in a month.

[irudyak]

Hi guys,

Just a short update for Cassandra persistent store implementation:

1) We switched to the newest version of Cassandra 3.3
2) Unit tests now using embedded Cassandra starting in the same process with Ignite cluster.
3) All the latest changes from Ignite 1.6 upstream were merged into repository: https://github.com/irudyak/ignite
4) Current implementation is available in branch "ignite-1371"

[Alexey Kuznetsov-2]

Igor, good news!

I take a look shortly.

Val, could you also take a look?

On Thu, Apr 7, 2016 at 1:35 PM, irudyak <[hidden email]> wrote:

> Hi guys,
>
> Just a short update for Cassandra persistent store implementation:
>
> 1) We switched to the newest version of Cassandra 3.3
> 2) Unit tests now using embedded Cassandra starting in the same process
> with
> Ignite cluster.
> 3) All the latest changes from Ignite 1.6 upstream were merged into
> repository: https://github.com/irudyak/ignite
> 4) Current implementation is available in branch "ignite-1371"
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p8313.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com