OSGi migration may require a special marshaller

[Romain Gilles]

Hi all,

I'm really interested in this issue: https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to make it work in our osgi environment. The main issue for us now is the serialization. I think it you will have to rework the OptimizedMarshaller or any other marshaller that works with object that come from outside your class space.

We have try kryo that works. Kryo provide an extension point in order to resolve the classes: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java . With this extension we are able to solve the problem of external classes. The only issue with kryo is that some classes need a certain care in the serialization process and therefore a specialized serializer.

So I would like to know from the community what do think of changing the way the optimized marshaller works or introducing the support of yet another marshaller based on a kryo like technology?


Thanks in advance,


Best regards,


Romain.


PS: I'm ready to help in the both cases.
"Misys" is the trade name of the Misys group of companies. This email and any attachments have been scanned for known viruses using multiple scanners. This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person. This email does not constitute the commencement of legal relations between you and Misys. Please refer to the executed contract between you and the relevant member of the Misys group for the identity of the contracting party with which you are dealing.

[Yakov Zhdanov-2]

Romain,

Is that correct that you obtain instance of Ignite through cache manager
and pass your classloader to which gets ignored at some point? Or you
use IgniteConfiguration#setClassLoader?

Can you or anyone in community please provide a short example or junit so I
can reproduce this?

Thanks!

Thanks!
--
Yakov Zhdanov, Director R&D
*GridGain Systems*
www.gridgain.com

2015-11-02 12:56 GMT+03:00 Gilles, Romain <[hidden email]>:

> Hi all,
>
> I'm really interested in this issue:
> https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to make
> it work in our osgi environment. The main issue for us now is the
> serialization. I think it you will have to rework the OptimizedMarshaller
> or any other marshaller that works with object that come from outside your
> class space.
>
> We have try kryo that works. Kryo provide an extension point in order to
> resolve the classes:
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> . With this extension we are able to solve the problem of external classes.
> The only issue with kryo is that some classes need a certain care in the
> serialization process and therefore a specialized serializer.
>
> So I would like to know from the community what do think of changing the
> way the optimized marshaller works or introducing the support of yet
> another marshaller based on a kryo like technology?
>
>
> Thanks in advance,
>
>
> Best regards,
>
>
> Romain.
>
>
> PS: I'm ready to help in the both cases.
> "Misys" is the trade name of the Misys group of companies. This email and
> any attachments have been scanned for known viruses using multiple
> scanners. This email message is intended for the named recipient only. It
> may be privileged and/or confidential. If you are not the named recipient
> of this email please notify us immediately and do not copy it or use it for
> any purpose, nor disclose its contents to any other person. This email does
> not constitute the commencement of legal relations between you and Misys.
> Please refer to the executed contract between you and the relevant member
> of the Misys group for the identity of the contracting party with which you
> are dealing.
>

[Raul Kripalani]

Hi Romain,

I'm working on the OSGi compatibility of Ignite. I appreciate your input.

I'm thinking about the situation you describe and I wonder if you're
exporting Ignite as an OSGi service which is then consumed from other
bundles. Under this situation, it would be quite easy to reproduce the
behaviour you describe if Ignite is not resolving classes via the TCCL.
Need to dig deeper into that.

Off the top of my head, there are two alternatives to solve it:

1. Use the TCCL for marshalling/unmarshalling (if not already used) – we
gotta be wary of possible regressions.
2. Create a special OSGi header Ignite-Export-Package so that bundles
containing DTOs can expose packages to Ignite's marshallers.

Regards,

*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <[hidden email]>
wrote:

> Hi all,
>
> I'm really interested in this issue:
> https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to make
> it work in our osgi environment. The main issue for us now is the
> serialization. I think it you will have to rework the OptimizedMarshaller
> or any other marshaller that works with object that come from outside your
> class space.
>
> We have try kryo that works. Kryo provide an extension point in order to
> resolve the classes:
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> . With this extension we are able to solve the problem of external classes.
> The only issue with kryo is that some classes need a certain care in the
> serialization process and therefore a specialized serializer.
>
> So I would like to know from the community what do think of changing the
> way the optimized marshaller works or introducing the support of yet
> another marshaller based on a kryo like technology?
>
>
> Thanks in advance,
>
>
> Best regards,
>
>
> Romain.
>
>
> PS: I'm ready to help in the both cases.
> "Misys" is the trade name of the Misys group of companies. This email and
> any attachments have been scanned for known viruses using multiple
> scanners. This email message is intended for the named recipient only. It
> may be privileged and/or confidential. If you are not the named recipient
> of this email please notify us immediately and do not copy it or use it for
> any purpose, nor disclose its contents to any other person. This email does
> not constitute the commencement of legal relations between you and Misys.
> Please refer to the executed contract between you and the relevant member
> of the Misys group for the identity of the contracting party with which you
> are dealing.
>

[Andrey Kornev]

Raul,

The fundamental hurdle we need to jump over to make Ignite OSGi-enabled is the marshalling. More specifically the issue is with deserialization of the classes that are provided by the bundles other than the Ignite bundle itself.

When the Ignite transport layer receives a message it needs to figure out how to deserialize the bytes and for that it needs to know the bundle that provides the class to be deserailized. At this point TCCL is of no use. To make things more complex, the class may contain other classes that come from other bundles, and so on recursively. This means that each object in the hierarchy must be serialized with its bundle name (or bundle id), so that the deserializer will then be able to correctly resolve the class while traversing the object hierarchy during deserialization.

Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to plug in a custom class resolver. Romain's solution was to use Kryo that does provide a way to customize class resolution. It has solved the problem of capturing the bundle info and he was able to successfully run Ignite as a bundle in an OSGi container (after some repackaging and inclusion of the manifest). But Kryo-based marshalling introduced a lot of complexity to the code and incorrect use of Kryo's numerous serializers caused some weird hard-to-debug issues in the Ignite core (like duplicate cache entries due to incorrect marshalling of the GridDhtPArtitonFullMap class -- go figure!). Overall the Kryo-based solution is brittle and hard to maintain.

I feel the correct solution to OSGi problem would be to
1) enhance the OptimizedMarshaller to allow custom class resolution.
2) provide an OSGi-enabled OptimizedMarshaller (in addition to the original one) to be used in OSGi environment.

Regards
Andrey

> From: [hidden email]
> Date: Mon, 2 Nov 2015 12:41:47 +0000
> Subject: Re: OSGi migration may require a special marshaller
> To: [hidden email]
>
> Hi Romain,
>
> I'm working on the OSGi compatibility of Ignite. I appreciate your input.
>
> I'm thinking about the situation you describe and I wonder if you're
> exporting Ignite as an OSGi service which is then consumed from other
> bundles. Under this situation, it would be quite easy to reproduce the
> behaviour you describe if Ignite is not resolving classes via the TCCL.
> Need to dig deeper into that.
>
> Off the top of my head, there are two alternatives to solve it:
>
> 1. Use the TCCL for marshalling/unmarshalling (if not already used) – we
> gotta be wary of possible regressions.
> 2. Create a special OSGi header Ignite-Export-Package so that bundles
> containing DTOs can expose packages to Ignite's marshallers.
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <[hidden email]>
> wrote:
>
> > Hi all,
> >
> > I'm really interested in this issue:
> > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to make
> > it work in our osgi environment. The main issue for us now is the
> > serialization. I think it you will have to rework the OptimizedMarshaller
> > or any other marshaller that works with object that come from outside your
> > class space.
> >
> > We have try kryo that works. Kryo provide an extension point in order to
> > resolve the classes:
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > . With this extension we are able to solve the problem of external classes.
> > The only issue with kryo is that some classes need a certain care in the
> > serialization process and therefore a specialized serializer.
> >
> > So I would like to know from the community what do think of changing the
> > way the optimized marshaller works or introducing the support of yet
> > another marshaller based on a kryo like technology?
> >
> >
> > Thanks in advance,
> >
> >
> > Best regards,
> >
> >
> > Romain.
> >
> >
> > PS: I'm ready to help in the both cases.
> > "Misys" is the trade name of the Misys group of companies. This email and
> > any attachments have been scanned for known viruses using multiple
> > scanners. This email message is intended for the named recipient only. It
> > may be privileged and/or confidential. If you are not the named recipient
> > of this email please notify us immediately and do not copy it or use it for
> > any purpose, nor disclose its contents to any other person. This email does
> > not constitute the commencement of legal relations between you and Misys.
> > Please refer to the executed contract between you and the relevant member
> > of the Misys group for the identity of the contracting party with which you
> > are dealing.
> >

     

[Raul Kripalani-2]

Hi Andrey,

Thanks for the participation in this topic.

I don't like the solution to incorporate the bundle symbolic name in the
serialised form. Nothing guarantees that the classdef will be located under
the same bundle in both source and target machines. We also have to take
into account that OSGi allows for multiple versions of the same
bundle/packages to co-exist in the same  container. So it becomes more
complex.

Using the TCCL should work when serialising, but it'll probably be of no
use when deserialising on the other end.

I need to enhance Ignite to:

1. Use the TCCL when marshalling on one end.
2. Incorporate the package version of the class in the serialised form if
Ignite is running in an OSGi environment.
3. On the receiver end, discover cache entities / DTOs in all bundles
through a custom OSGi manifest header or the like, as I explained before.
Package version must be taken into account.

What do you think?

Raúl.
On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:

> Raul,
>
> The fundamental hurdle we need to jump over to make Ignite OSGi-enabled is
> the marshalling. More specifically the issue is with deserialization of the
> classes that are provided by the bundles other than the Ignite bundle
> itself.
>
> When the Ignite transport layer receives a message it needs to figure out
> how to deserialize the bytes and for that it needs to know the bundle that
> provides the class to be deserailized. At this point TCCL is of no use. To
> make things more complex, the class may contain other classes that come
> from other bundles, and so on recursively. This means that each object in
> the hierarchy must be serialized with its bundle name (or bundle id), so
> that the deserializer will then be able to correctly resolve the class
> while traversing the object hierarchy during deserialization.
>
> Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to plug
> in a custom class resolver. Romain's solution was to use Kryo that does
> provide a way to customize class resolution. It has solved the problem of
> capturing the bundle info and he was able to successfully run Ignite as a
> bundle in an OSGi container (after some repackaging and inclusion of the
> manifest). But Kryo-based marshalling introduced a lot of complexity to the
> code and incorrect use of Kryo's numerous serializers caused some weird
> hard-to-debug issues in the Ignite core (like duplicate cache entries due
> to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> figure!). Overall the Kryo-based solution is brittle and hard to maintain.
>
> I feel the correct solution to OSGi problem would be to
> 1) enhance the OptimizedMarshaller to allow custom class resolution.
> 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> original one) to be used in OSGi environment.
>
> Regards
> Andrey
>
> > From: [hidden email]
> > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > Subject: Re: OSGi migration may require a special marshaller
> > To: [hidden email]
> >
> > Hi Romain,
> >
> > I'm working on the OSGi compatibility of Ignite. I appreciate your input.
> >
> > I'm thinking about the situation you describe and I wonder if you're
> > exporting Ignite as an OSGi service which is then consumed from other
> > bundles. Under this situation, it would be quite easy to reproduce the
> > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > Need to dig deeper into that.
> >
> > Off the top of my head, there are two alternatives to solve it:
> >
> > 1. Use the TCCL for marshalling/unmarshalling (if not already used) – we
> > gotta be wary of possible regressions.
> > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > containing DTOs can expose packages to Ignite's marshallers.
> >
> > Regards,
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <[hidden email]>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'm really interested in this issue:
> > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to
> make
> > > it work in our osgi environment. The main issue for us now is the
> > > serialization. I think it you will have to rework the
> OptimizedMarshaller
> > > or any other marshaller that works with object that come from outside
> your
> > > class space.
> > >
> > > We have try kryo that works. Kryo provide an extension point in order
> to
> > > resolve the classes:
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > . With this extension we are able to solve the problem of external
> classes.
> > > The only issue with kryo is that some classes need a certain care in
> the
> > > serialization process and therefore a specialized serializer.
> > >
> > > So I would like to know from the community what do think of changing
> the
> > > way the optimized marshaller works or introducing the support of yet
> > > another marshaller based on a kryo like technology?
> > >
> > >
> > > Thanks in advance,
> > >
> > >
> > > Best regards,
> > >
> > >
> > > Romain.
> > >
> > >
> > > PS: I'm ready to help in the both cases.
> > > "Misys" is the trade name of the Misys group of companies. This email
> and
> > > any attachments have been scanned for known viruses using multiple
> > > scanners. This email message is intended for the named recipient only.
> It
> > > may be privileged and/or confidential. If you are not the named
> recipient
> > > of this email please notify us immediately and do not copy it or use
> it for
> > > any purpose, nor disclose its contents to any other person. This email
> does
> > > not constitute the commencement of legal relations between you and
> Misys.
> > > Please refer to the executed contract between you and the relevant
> member
> > > of the Misys group for the identity of the contracting party with
> which you
> > > are dealing.
> > >
>

[dsetrakyan]

Raul,

Why do we need TCCL for serializing? We already have the Object we want to
serialize, so to get its class loader we can call
myObject.getClass().getClassLoader(), no?

D.

On Mon, Nov 2, 2015 at 3:20 PM, Raul Kripalani <[hidden email]> wrote:

> Hi Andrey,
>
> Thanks for the participation in this topic.
>
> I don't like the solution to incorporate the bundle symbolic name in the
> serialised form. Nothing guarantees that the classdef will be located under
> the same bundle in both source and target machines. We also have to take
> into account that OSGi allows for multiple versions of the same
> bundle/packages to co-exist in the same  container. So it becomes more
> complex.
>
> Using the TCCL should work when serialising, but it'll probably be of no
> use when deserialising on the other end.
>
> I need to enhance Ignite to:
>
> 1. Use the TCCL when marshalling on one end.
> 2. Incorporate the package version of the class in the serialised form if
> Ignite is running in an OSGi environment.
> 3. On the receiver end, discover cache entities / DTOs in all bundles
> through a custom OSGi manifest header or the like, as I explained before.
> Package version must be taken into account.
>
> What do you think?
>
> Raúl.
> On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
>
> > Raul,
> >
> > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> is
> > the marshalling. More specifically the issue is with deserialization of
> the
> > classes that are provided by the bundles other than the Ignite bundle
> > itself.
> >
> > When the Ignite transport layer receives a message it needs to figure out
> > how to deserialize the bytes and for that it needs to know the bundle
> that
> > provides the class to be deserailized. At this point TCCL is of no use.
> To
> > make things more complex, the class may contain other classes that come
> > from other bundles, and so on recursively. This means that each object in
> > the hierarchy must be serialized with its bundle name (or bundle id), so
> > that the deserializer will then be able to correctly resolve the class
> > while traversing the object hierarchy during deserialization.
> >
> > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> plug
> > in a custom class resolver. Romain's solution was to use Kryo that does
> > provide a way to customize class resolution. It has solved the problem of
> > capturing the bundle info and he was able to successfully run Ignite as a
> > bundle in an OSGi container (after some repackaging and inclusion of the
> > manifest). But Kryo-based marshalling introduced a lot of complexity to
> the
> > code and incorrect use of Kryo's numerous serializers caused some weird
> > hard-to-debug issues in the Ignite core (like duplicate cache entries due
> > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > figure!). Overall the Kryo-based solution is brittle and hard to
> maintain.
> >
> > I feel the correct solution to OSGi problem would be to
> > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > original one) to be used in OSGi environment.
> >
> > Regards
> > Andrey
> >
> > > From: [hidden email]
> > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: [hidden email]
> > >
> > > Hi Romain,
> > >
> > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> input.
> > >
> > > I'm thinking about the situation you describe and I wonder if you're
> > > exporting Ignite as an OSGi service which is then consumed from other
> > > bundles. Under this situation, it would be quite easy to reproduce the
> > > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > > Need to dig deeper into that.
> > >
> > > Off the top of my head, there are two alternatives to solve it:
> > >
> > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) –
> we
> > > gotta be wary of possible regressions.
> > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > containing DTOs can expose packages to Ignite's marshallers.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> and
> > > Messaging Engineer
> > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> [hidden email]>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm really interested in this issue:
> > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to
> > make
> > > > it work in our osgi environment. The main issue for us now is the
> > > > serialization. I think it you will have to rework the
> > OptimizedMarshaller
> > > > or any other marshaller that works with object that come from outside
> > your
> > > > class space.
> > > >
> > > > We have try kryo that works. Kryo provide an extension point in order
> > to
> > > > resolve the classes:
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > . With this extension we are able to solve the problem of external
> > classes.
> > > > The only issue with kryo is that some classes need a certain care in
> > the
> > > > serialization process and therefore a specialized serializer.
> > > >
> > > > So I would like to know from the community what do think of changing
> > the
> > > > way the optimized marshaller works or introducing the support of yet
> > > > another marshaller based on a kryo like technology?
> > > >
> > > >
> > > > Thanks in advance,
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Romain.
> > > >
> > > >
> > > > PS: I'm ready to help in the both cases.
> > > > "Misys" is the trade name of the Misys group of companies. This email
> > and
> > > > any attachments have been scanned for known viruses using multiple
> > > > scanners. This email message is intended for the named recipient
> only.
> > It
> > > > may be privileged and/or confidential. If you are not the named
> > recipient
> > > > of this email please notify us immediately and do not copy it or use
> > it for
> > > > any purpose, nor disclose its contents to any other person. This
> email
> > does
> > > > not constitute the commencement of legal relations between you and
> > Misys.
> > > > Please refer to the executed contract between you and the relevant
> > member
> > > > of the Misys group for the identity of the contracting party with
> > which you
> > > > are dealing.
> > > >
> >
>

[Andrey Kornev]

Hey Raul,

What kind of use case do you have in mind when you suggest that a classdef may be located in different bundles on different machines? What would one have to do to end up in such a predicament? It feels more like a configuration/deployment issue rather than a desirable feature.

A more realistic assumption I believe is that the nodes will have (mostly) the same set of bundles deployed. (We even enforce it at runtime  by leveraging Ignite's PluginProvider.validateNewNode() method: we do not allow new nodes join the cluster if their OSGi environment is inconsistent.)

The "bundle name" in my original email really meant "bundle symbolic name + bundle version". Sorry for not being clear.

I'm not sure I understand why TCCL is even required for serialization?

Finally, I didn't quite grok your thinking about the custom manifest header. Could you please elaborate? What type of information do you propose to store in the header? How will the serializer be able to provide that info, and how will the deserializer use the info to load the classes? Also, who is responsible for making sure that the custom header includes all classes to be serialized? And how/when/by whom would such a header be generated?

Regards
Andrey

> Date: Mon, 2 Nov 2015 23:20:41 +0000
> Subject: RE: OSGi migration may require a special marshaller
> From: [hidden email]
> To: [hidden email]
>
> Hi Andrey,
>
> Thanks for the participation in this topic.
>
> I don't like the solution to incorporate the bundle symbolic name in the
> serialised form. Nothing guarantees that the classdef will be located under
> the same bundle in both source and target machines. We also have to take
> into account that OSGi allows for multiple versions of the same
> bundle/packages to co-exist in the same  container. So it becomes more
> complex.
>
> Using the TCCL should work when serialising, but it'll probably be of no
> use when deserialising on the other end.
>
> I need to enhance Ignite to:
>
> 1. Use the TCCL when marshalling on one end.
> 2. Incorporate the package version of the class in the serialised form if
> Ignite is running in an OSGi environment.
> 3. On the receiver end, discover cache entities / DTOs in all bundles
> through a custom OSGi manifest header or the like, as I explained before.
> Package version must be taken into account.
>
> What do you think?
>
> Raúl.
> On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
>
> > Raul,
> >
> > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled is
> > the marshalling. More specifically the issue is with deserialization of the
> > classes that are provided by the bundles other than the Ignite bundle
> > itself.
> >
> > When the Ignite transport layer receives a message it needs to figure out
> > how to deserialize the bytes and for that it needs to know the bundle that
> > provides the class to be deserailized. At this point TCCL is of no use. To
> > make things more complex, the class may contain other classes that come
> > from other bundles, and so on recursively. This means that each object in
> > the hierarchy must be serialized with its bundle name (or bundle id), so
> > that the deserializer will then be able to correctly resolve the class
> > while traversing the object hierarchy during deserialization.
> >
> > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to plug
> > in a custom class resolver. Romain's solution was to use Kryo that does
> > provide a way to customize class resolution. It has solved the problem of
> > capturing the bundle info and he was able to successfully run Ignite as a
> > bundle in an OSGi container (after some repackaging and inclusion of the
> > manifest). But Kryo-based marshalling introduced a lot of complexity to the
> > code and incorrect use of Kryo's numerous serializers caused some weird
> > hard-to-debug issues in the Ignite core (like duplicate cache entries due
> > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > figure!). Overall the Kryo-based solution is brittle and hard to maintain.
> >
> > I feel the correct solution to OSGi problem would be to
> > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > original one) to be used in OSGi environment.
> >
> > Regards
> > Andrey
> >
> > > From: [hidden email]
> > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: [hidden email]
> > >
> > > Hi Romain,
> > >
> > > I'm working on the OSGi compatibility of Ignite. I appreciate your input.
> > >
> > > I'm thinking about the situation you describe and I wonder if you're
> > > exporting Ignite as an OSGi service which is then consumed from other
> > > bundles. Under this situation, it would be quite easy to reproduce the
> > > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > > Need to dig deeper into that.
> > >
> > > Off the top of my head, there are two alternatives to solve it:
> > >
> > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) – we
> > > gotta be wary of possible regressions.
> > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > containing DTOs can expose packages to Ignite's marshallers.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > > Messaging Engineer
> > > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <[hidden email]>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm really interested in this issue:
> > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to
> > make
> > > > it work in our osgi environment. The main issue for us now is the
> > > > serialization. I think it you will have to rework the
> > OptimizedMarshaller
> > > > or any other marshaller that works with object that come from outside
> > your
> > > > class space.
> > > >
> > > > We have try kryo that works. Kryo provide an extension point in order
> > to
> > > > resolve the classes:
> > > >
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > . With this extension we are able to solve the problem of external
> > classes.
> > > > The only issue with kryo is that some classes need a certain care in
> > the
> > > > serialization process and therefore a specialized serializer.
> > > >
> > > > So I would like to know from the community what do think of changing
> > the
> > > > way the optimized marshaller works or introducing the support of yet
> > > > another marshaller based on a kryo like technology?
> > > >
> > > >
> > > > Thanks in advance,
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Romain.
> > > >
> > > >
> > > > PS: I'm ready to help in the both cases.
> > > > "Misys" is the trade name of the Misys group of companies. This email
> > and
> > > > any attachments have been scanned for known viruses using multiple
> > > > scanners. This email message is intended for the named recipient only.
> > It
> > > > may be privileged and/or confidential. If you are not the named
> > recipient
> > > > of this email please notify us immediately and do not copy it or use
> > it for
> > > > any purpose, nor disclose its contents to any other person. This email
> > does
> > > > not constitute the commencement of legal relations between you and
> > Misys.
> > > > Please refer to the executed contract between you and the relevant
> > member
> > > > of the Misys group for the identity of the contracting party with
> > which you
> > > > are dealing.
> > > >
> >

     

[dsetrakyan]

I think I can see Andrey’s point about the bundles. Having same classdef in
different bundles on sender and receiver sides sounds rather like a bug,
not a feature we should support.

Having read this thread, I think we need to have ClassLoaderResolver on the
sending and receiving sides. Something like this:

*public interface ClassLoaderResolver {*
*    // This method is invoked on the sender side to *
*    // marshal the information about the class loader.*
*    public byte[] writeClassLoader(Object o) throws IgniteException;*

*    // This method is invoked on the receiving side to*
*    // retrieve the class loader based on provided information.*
*    public ClassLoader readClassLoader(byte[]) throws IgniteException;*
*}*

This resolver should be a pluggable configuration for marshallers or
provided in IgniteConfiguration directly. Once we have such resolver, then
we can have a special OSGI implementation for it (perhaps a different one
for different OSGI containers).

Thoughts?

D.

On Mon, Nov 2, 2015 at 4:23 PM, Andrey Kornev <[hidden email]>
wrote:

> Hey Raul,
>
> What kind of use case do you have in mind when you suggest that a classdef
> may be located in different bundles on different machines? What would one
> have to do to end up in such a predicament? It feels more like a
> configuration/deployment issue rather than a desirable feature.
>
> A more realistic assumption I believe is that the nodes will have (mostly)
> the same set of bundles deployed. (We even enforce it at runtime  by
> leveraging Ignite's PluginProvider.validateNewNode() method: we do not
> allow new nodes join the cluster if their OSGi environment is inconsistent.)
>
> The "bundle name" in my original email really meant "bundle symbolic name
> + bundle version". Sorry for not being clear.
>
> I'm not sure I understand why TCCL is even required for serialization?
>
> Finally, I didn't quite grok your thinking about the custom manifest
> header. Could you please elaborate? What type of information do you propose
> to store in the header? How will the serializer be able to provide that
> info, and how will the deserializer use the info to load the classes? Also,
> who is responsible for making sure that the custom header includes all
> classes to be serialized? And how/when/by whom would such a header be
> generated?
>
> Regards
> Andrey
>
> > Date: Mon, 2 Nov 2015 23:20:41 +0000
> > Subject: RE: OSGi migration may require a special marshaller
> > From: [hidden email]
> > To: [hidden email]
> >
> > Hi Andrey,
> >
> > Thanks for the participation in this topic.
> >
> > I don't like the solution to incorporate the bundle symbolic name in the
> > serialised form. Nothing guarantees that the classdef will be located
> under
> > the same bundle in both source and target machines. We also have to take
> > into account that OSGi allows for multiple versions of the same
> > bundle/packages to co-exist in the same  container. So it becomes more
> > complex.
> >
> > Using the TCCL should work when serialising, but it'll probably be of no
> > use when deserialising on the other end.
> >
> > I need to enhance Ignite to:
> >
> > 1. Use the TCCL when marshalling on one end.
> > 2. Incorporate the package version of the class in the serialised form if
> > Ignite is running in an OSGi environment.
> > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > through a custom OSGi manifest header or the like, as I explained before.
> > Package version must be taken into account.
> >
> > What do you think?
> >
> > Raúl.
> > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
> >
> > > Raul,
> > >
> > > The fundamental hurdle we need to jump over to make Ignite
> OSGi-enabled is
> > > the marshalling. More specifically the issue is with deserialization
> of the
> > > classes that are provided by the bundles other than the Ignite bundle
> > > itself.
> > >
> > > When the Ignite transport layer receives a message it needs to figure
> out
> > > how to deserialize the bytes and for that it needs to know the bundle
> that
> > > provides the class to be deserailized. At this point TCCL is of no
> use. To
> > > make things more complex, the class may contain other classes that come
> > > from other bundles, and so on recursively. This means that each object
> in
> > > the hierarchy must be serialized with its bundle name (or bundle id),
> so
> > > that the deserializer will then be able to correctly resolve the class
> > > while traversing the object hierarchy during deserialization.
> > >
> > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> plug
> > > in a custom class resolver. Romain's solution was to use Kryo that does
> > > provide a way to customize class resolution. It has solved the problem
> of
> > > capturing the bundle info and he was able to successfully run Ignite
> as a
> > > bundle in an OSGi container (after some repackaging and inclusion of
> the
> > > manifest). But Kryo-based marshalling introduced a lot of complexity
> to the
> > > code and incorrect use of Kryo's numerous serializers caused some weird
> > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> due
> > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > figure!). Overall the Kryo-based solution is brittle and hard to
> maintain.
> > >
> > > I feel the correct solution to OSGi problem would be to
> > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > original one) to be used in OSGi environment.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: [hidden email]
> > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: [hidden email]
> > > >
> > > > Hi Romain,
> > > >
> > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> input.
> > > >
> > > > I'm thinking about the situation you describe and I wonder if you're
> > > > exporting Ignite as an OSGi service which is then consumed from other
> > > > bundles. Under this situation, it would be quite easy to reproduce
> the
> > > > behaviour you describe if Ignite is not resolving classes via the
> TCCL.
> > > > Need to dig deeper into that.
> > > >
> > > > Off the top of my head, there are two alternatives to solve it:
> > > >
> > > > 1. Use the TCCL for marshalling/unmarshalling (if not already used)
> – we
> > > > gotta be wary of possible regressions.
> > > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > > containing DTOs can expose packages to Ignite's marshallers.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm really interested in this issue:
> > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff
> to
> > > make
> > > > > it work in our osgi environment. The main issue for us now is the
> > > > > serialization. I think it you will have to rework the
> > > OptimizedMarshaller
> > > > > or any other marshaller that works with object that come from
> outside
> > > your
> > > > > class space.
> > > > >
> > > > > We have try kryo that works. Kryo provide an extension point in
> order
> > > to
> > > > > resolve the classes:
> > > > >
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > . With this extension we are able to solve the problem of external
> > > classes.
> > > > > The only issue with kryo is that some classes need a certain care
> in
> > > the
> > > > > serialization process and therefore a specialized serializer.
> > > > >
> > > > > So I would like to know from the community what do think of
> changing
> > > the
> > > > > way the optimized marshaller works or introducing the support of
> yet
> > > > > another marshaller based on a kryo like technology?
> > > > >
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Romain.
> > > > >
> > > > >
> > > > > PS: I'm ready to help in the both cases.
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > > it for
> > > > > any purpose, nor disclose its contents to any other person. This
> email
> > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > > which you
> > > > > are dealing.
> > > > >
> > >
>
>

[Romain Gilles-2]

Hi Raul,
 - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
 - I like the point 2. Maybe for a graph of object that come from different
packages / bundles you may have to recursively capture the package version
of the individual element of your graph.
 - For the point 3 I wonder if it will not over-complexify the solution. As
a developer you will have to think about it. And it is not obvious in the
code where things are serialized or not. You may use lambda in your
application code therefore the current package become what you call a DTO
package. The current state of ignite does not enforce you to specify it for
"classical" classloading application. If you introduce this extra step for
OSGi ready application you will maybe discourage people to use OSGi.

To comeback to our solution. We start we a strong assumption: our cluster
is homogeneous in term of code so, of course, it simplify our life :).

BTW if you can introduce an extension point in the class resolution
mechanism it can be interesting for end user in order to allow them to
optimize it based on there specific use cases.

Romain.

Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a écrit :

> Hi Andrey,
>
> Thanks for the participation in this topic.
>
> I don't like the solution to incorporate the bundle symbolic name in the
> serialised form. Nothing guarantees that the classdef will be located under
> the same bundle in both source and target machines. We also have to take
> into account that OSGi allows for multiple versions of the same
> bundle/packages to co-exist in the same  container. So it becomes more
> complex.
>
> Using the TCCL should work when serialising, but it'll probably be of no
> use when deserialising on the other end.
>
> I need to enhance Ignite to:
>
> 1. Use the TCCL when marshalling on one end.
> 2. Incorporate the package version of the class in the serialised form if
> Ignite is running in an OSGi environment.
> 3. On the receiver end, discover cache entities / DTOs in all bundles
> through a custom OSGi manifest header or the like, as I explained before.
> Package version must be taken into account.
>
> What do you think?
>
> Raúl.
> On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
>
> > Raul,
> >
> > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> is
> > the marshalling. More specifically the issue is with deserialization of
> the
> > classes that are provided by the bundles other than the Ignite bundle
> > itself.
> >
> > When the Ignite transport layer receives a message it needs to figure out
> > how to deserialize the bytes and for that it needs to know the bundle
> that
> > provides the class to be deserailized. At this point TCCL is of no use.
> To
> > make things more complex, the class may contain other classes that come
> > from other bundles, and so on recursively. This means that each object in
> > the hierarchy must be serialized with its bundle name (or bundle id), so
> > that the deserializer will then be able to correctly resolve the class
> > while traversing the object hierarchy during deserialization.
> >
> > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> plug
> > in a custom class resolver. Romain's solution was to use Kryo that does
> > provide a way to customize class resolution. It has solved the problem of
> > capturing the bundle info and he was able to successfully run Ignite as a
> > bundle in an OSGi container (after some repackaging and inclusion of the
> > manifest). But Kryo-based marshalling introduced a lot of complexity to
> the
> > code and incorrect use of Kryo's numerous serializers caused some weird
> > hard-to-debug issues in the Ignite core (like duplicate cache entries due
> > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > figure!). Overall the Kryo-based solution is brittle and hard to
> maintain.
> >
> > I feel the correct solution to OSGi problem would be to
> > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > original one) to be used in OSGi environment.
> >
> > Regards
> > Andrey
> >
> > > From: [hidden email]
> > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: [hidden email]
> > >
> > > Hi Romain,
> > >
> > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> input.
> > >
> > > I'm thinking about the situation you describe and I wonder if you're
> > > exporting Ignite as an OSGi service which is then consumed from other
> > > bundles. Under this situation, it would be quite easy to reproduce the
> > > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > > Need to dig deeper into that.
> > >
> > > Off the top of my head, there are two alternatives to solve it:
> > >
> > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) –
> we
> > > gotta be wary of possible regressions.
> > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > containing DTOs can expose packages to Ignite's marshallers.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> and
> > > Messaging Engineer
> > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> [hidden email]>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm really interested in this issue:
> > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to
> > make
> > > > it work in our osgi environment. The main issue for us now is the
> > > > serialization. I think it you will have to rework the
> > OptimizedMarshaller
> > > > or any other marshaller that works with object that come from outside
> > your
> > > > class space.
> > > >
> > > > We have try kryo that works. Kryo provide an extension point in order
> > to
> > > > resolve the classes:
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > . With this extension we are able to solve the problem of external
> > classes.
> > > > The only issue with kryo is that some classes need a certain care in
> > the
> > > > serialization process and therefore a specialized serializer.
> > > >
> > > > So I would like to know from the community what do think of changing
> > the
> > > > way the optimized marshaller works or introducing the support of yet
> > > > another marshaller based on a kryo like technology?
> > > >
> > > >
> > > > Thanks in advance,
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Romain.
> > > >
> > > >
> > > > PS: I'm ready to help in the both cases.
> > > > "Misys" is the trade name of the Misys group of companies. This email
> > and
> > > > any attachments have been scanned for known viruses using multiple
> > > > scanners. This email message is intended for the named recipient
> only.
> > It
> > > > may be privileged and/or confidential. If you are not the named
> > recipient
> > > > of this email please notify us immediately and do not copy it or use
> > it for
> > > > any purpose, nor disclose its contents to any other person. This
> email
> > does
> > > > not constitute the commencement of legal relations between you and
> > Misys.
> > > > Please refer to the executed contract between you and the relevant
> > member
> > > > of the Misys group for the identity of the contracting party with
> > which you
> > > > are dealing.
> > > >
> >
>

[dsetrakyan]

Romain,

Can you comment on the ClassLoaderResolver suggestion that I proposed
earlier? Will it solve your problem?

D.

On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <[hidden email]>
wrote:

> Hi Raul,
>  - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
>  - I like the point 2. Maybe for a graph of object that come from different
> packages / bundles you may have to recursively capture the package version
> of the individual element of your graph.
>  - For the point 3 I wonder if it will not over-complexify the solution. As
> a developer you will have to think about it. And it is not obvious in the
> code where things are serialized or not. You may use lambda in your
> application code therefore the current package become what you call a DTO
> package. The current state of ignite does not enforce you to specify it for
> "classical" classloading application. If you introduce this extra step for
> OSGi ready application you will maybe discourage people to use OSGi.
>
> To comeback to our solution. We start we a strong assumption: our cluster
> is homogeneous in term of code so, of course, it simplify our life :).
>
> BTW if you can introduce an extension point in the class resolution
> mechanism it can be interesting for end user in order to allow them to
> optimize it based on there specific use cases.
>
> Romain.
>
> Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a écrit :
>
> > Hi Andrey,
> >
> > Thanks for the participation in this topic.
> >
> > I don't like the solution to incorporate the bundle symbolic name in the
> > serialised form. Nothing guarantees that the classdef will be located
> under
> > the same bundle in both source and target machines. We also have to take
> > into account that OSGi allows for multiple versions of the same
> > bundle/packages to co-exist in the same  container. So it becomes more
> > complex.
> >
> > Using the TCCL should work when serialising, but it'll probably be of no
> > use when deserialising on the other end.
> >
> > I need to enhance Ignite to:
> >
> > 1. Use the TCCL when marshalling on one end.
> > 2. Incorporate the package version of the class in the serialised form if
> > Ignite is running in an OSGi environment.
> > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > through a custom OSGi manifest header or the like, as I explained before.
> > Package version must be taken into account.
> >
> > What do you think?
> >
> > Raúl.
> > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
> >
> > > Raul,
> > >
> > > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> > is
> > > the marshalling. More specifically the issue is with deserialization of
> > the
> > > classes that are provided by the bundles other than the Ignite bundle
> > > itself.
> > >
> > > When the Ignite transport layer receives a message it needs to figure
> out
> > > how to deserialize the bytes and for that it needs to know the bundle
> > that
> > > provides the class to be deserailized. At this point TCCL is of no use.
> > To
> > > make things more complex, the class may contain other classes that come
> > > from other bundles, and so on recursively. This means that each object
> in
> > > the hierarchy must be serialized with its bundle name (or bundle id),
> so
> > > that the deserializer will then be able to correctly resolve the class
> > > while traversing the object hierarchy during deserialization.
> > >
> > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > plug
> > > in a custom class resolver. Romain's solution was to use Kryo that does
> > > provide a way to customize class resolution. It has solved the problem
> of
> > > capturing the bundle info and he was able to successfully run Ignite
> as a
> > > bundle in an OSGi container (after some repackaging and inclusion of
> the
> > > manifest). But Kryo-based marshalling introduced a lot of complexity to
> > the
> > > code and incorrect use of Kryo's numerous serializers caused some weird
> > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> due
> > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > figure!). Overall the Kryo-based solution is brittle and hard to
> > maintain.
> > >
> > > I feel the correct solution to OSGi problem would be to
> > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > original one) to be used in OSGi environment.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: [hidden email]
> > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: [hidden email]
> > > >
> > > > Hi Romain,
> > > >
> > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > input.
> > > >
> > > > I'm thinking about the situation you describe and I wonder if you're
> > > > exporting Ignite as an OSGi service which is then consumed from other
> > > > bundles. Under this situation, it would be quite easy to reproduce
> the
> > > > behaviour you describe if Ignite is not resolving classes via the
> TCCL.
> > > > Need to dig deeper into that.
> > > >
> > > > Off the top of my head, there are two alternatives to solve it:
> > > >
> > > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) –
> > we
> > > > gotta be wary of possible regressions.
> > > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > > containing DTOs can expose packages to Ignite's marshallers.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> > and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm really interested in this issue:
> > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff
> to
> > > make
> > > > > it work in our osgi environment. The main issue for us now is the
> > > > > serialization. I think it you will have to rework the
> > > OptimizedMarshaller
> > > > > or any other marshaller that works with object that come from
> outside
> > > your
> > > > > class space.
> > > > >
> > > > > We have try kryo that works. Kryo provide an extension point in
> order
> > > to
> > > > > resolve the classes:
> > > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > . With this extension we are able to solve the problem of external
> > > classes.
> > > > > The only issue with kryo is that some classes need a certain care
> in
> > > the
> > > > > serialization process and therefore a specialized serializer.
> > > > >
> > > > > So I would like to know from the community what do think of
> changing
> > > the
> > > > > way the optimized marshaller works or introducing the support of
> yet
> > > > > another marshaller based on a kryo like technology?
> > > > >
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Romain.
> > > > >
> > > > >
> > > > > PS: I'm ready to help in the both cases.
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> > only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > > it for
> > > > > any purpose, nor disclose its contents to any other person. This
> > email
> > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > > which you
> > > > > are dealing.
> > > > >
> > >
> >
>

[Raul Kripalani]

Hey guys,

About the TCCL, I need to dig deeper into how serialisation is being done
now. If, at any point, we are resolving a class through the classloader of
a particular class, e.g. Ignition.getClass().getClassLoader(), etc. we are
using the class space of the bundle containing that class. If we are using
a class from Ignite, we obviously wouldn't be able to find a custom DTO
provided by a user.

In Camel we have found this problem several times and we've solved it by
changing the TCCL, or using the TCCL [1] [2] to resolve classes.

As I said, I need to look into this deeper and I'll have some time on
Thursday, so I hope you don't mind waiting a little bit. I have already
"bundle-fied" most of Ignite modules in the corresponding branch [3], so my
next step is to test out the Ignite examples inside an OSGi environment
(Karaf 4).

[1] https://issues.apache.org/jira/browse/CAMEL-5748
[2] https://issues.apache.org/jira/browse/CAMEL-5722
[3] https://github.com/apache/ignite/tree/ignite-1527

Regards,

*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Tue, Nov 3, 2015 at 10:38 AM, Romain Gilles <[hidden email]>
wrote:

> Hi Raul,
>  - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
>  - I like the point 2. Maybe for a graph of object that come from different
> packages / bundles you may have to recursively capture the package version
> of the individual element of your graph.
>  - For the point 3 I wonder if it will not over-complexify the solution. As
> a developer you will have to think about it. And it is not obvious in the
> code where things are serialized or not. You may use lambda in your
> application code therefore the current package become what you call a DTO
> package. The current state of ignite does not enforce you to specify it for
> "classical" classloading application. If you introduce this extra step for
> OSGi ready application you will maybe discourage people to use OSGi.
>
> To comeback to our solution. We start we a strong assumption: our cluster
> is homogeneous in term of code so, of course, it simplify our life :).
>
> BTW if you can introduce an extension point in the class resolution
> mechanism it can be interesting for end user in order to allow them to
> optimize it based on there specific use cases.
>
> Romain.
>
> Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a écrit :
>
> > Hi Andrey,
> >
> > Thanks for the participation in this topic.
> >
> > I don't like the solution to incorporate the bundle symbolic name in the
> > serialised form. Nothing guarantees that the classdef will be located
> under
> > the same bundle in both source and target machines. We also have to take
> > into account that OSGi allows for multiple versions of the same
> > bundle/packages to co-exist in the same  container. So it becomes more
> > complex.
> >
> > Using the TCCL should work when serialising, but it'll probably be of no
> > use when deserialising on the other end.
> >
> > I need to enhance Ignite to:
> >
> > 1. Use the TCCL when marshalling on one end.
> > 2. Incorporate the package version of the class in the serialised form if
> > Ignite is running in an OSGi environment.
> > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > through a custom OSGi manifest header or the like, as I explained before.
> > Package version must be taken into account.
> >
> > What do you think?
> >
> > Raúl.
> > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
> >
> > > Raul,
> > >
> > > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> > is
> > > the marshalling. More specifically the issue is with deserialization of
> > the
> > > classes that are provided by the bundles other than the Ignite bundle
> > > itself.
> > >
> > > When the Ignite transport layer receives a message it needs to figure
> out
> > > how to deserialize the bytes and for that it needs to know the bundle
> > that
> > > provides the class to be deserailized. At this point TCCL is of no use.
> > To
> > > make things more complex, the class may contain other classes that come
> > > from other bundles, and so on recursively. This means that each object
> in
> > > the hierarchy must be serialized with its bundle name (or bundle id),
> so
> > > that the deserializer will then be able to correctly resolve the class
> > > while traversing the object hierarchy during deserialization.
> > >
> > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > plug
> > > in a custom class resolver. Romain's solution was to use Kryo that does
> > > provide a way to customize class resolution. It has solved the problem
> of
> > > capturing the bundle info and he was able to successfully run Ignite
> as a
> > > bundle in an OSGi container (after some repackaging and inclusion of
> the
> > > manifest). But Kryo-based marshalling introduced a lot of complexity to
> > the
> > > code and incorrect use of Kryo's numerous serializers caused some weird
> > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> due
> > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > figure!). Overall the Kryo-based solution is brittle and hard to
> > maintain.
> > >
> > > I feel the correct solution to OSGi problem would be to
> > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > original one) to be used in OSGi environment.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: [hidden email]
> > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: [hidden email]
> > > >
> > > > Hi Romain,
> > > >
> > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > input.
> > > >
> > > > I'm thinking about the situation you describe and I wonder if you're
> > > > exporting Ignite as an OSGi service which is then consumed from other
> > > > bundles. Under this situation, it would be quite easy to reproduce
> the
> > > > behaviour you describe if Ignite is not resolving classes via the
> TCCL.
> > > > Need to dig deeper into that.
> > > >
> > > > Off the top of my head, there are two alternatives to solve it:
> > > >
> > > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) –
> > we
> > > > gotta be wary of possible regressions.
> > > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > > containing DTOs can expose packages to Ignite's marshallers.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> > and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm really interested in this issue:
> > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff
> to
> > > make
> > > > > it work in our osgi environment. The main issue for us now is the
> > > > > serialization. I think it you will have to rework the
> > > OptimizedMarshaller
> > > > > or any other marshaller that works with object that come from
> outside
> > > your
> > > > > class space.
> > > > >
> > > > > We have try kryo that works. Kryo provide an extension point in
> order
> > > to
> > > > > resolve the classes:
> > > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > . With this extension we are able to solve the problem of external
> > > classes.
> > > > > The only issue with kryo is that some classes need a certain care
> in
> > > the
> > > > > serialization process and therefore a specialized serializer.
> > > > >
> > > > > So I would like to know from the community what do think of
> changing
> > > the
> > > > > way the optimized marshaller works or introducing the support of
> yet
> > > > > another marshaller based on a kryo like technology?
> > > > >
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Romain.
> > > > >
> > > > >
> > > > > PS: I'm ready to help in the both cases.
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> > only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > > it for
> > > > > any purpose, nor disclose its contents to any other person. This
> > email
> > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > > which you
> > > > > are dealing.
> > > > >
> > >
> >
>

[Romain Gilles-2]

Hi Dmitriy,
I think your solution is good. Maybe I will change it a little bit... :P
I think you should delegate the Class resolution to the resolver. Because
for example with your current solution the marshaller may before (or after)
store the fqn of the class (maybe only the first time it encounter it) but
in order to save the classloader context resolution the implementation of
the resolver may have to save the package name of the given object and some
extra info therefore the content package name will be serialized two times.
Ok, it's not a big deal.
But now if the resolver identify that it can reuse the same class loader
for a couple of classes. It will be interesting for it to have a
serialization context in order to save, let say, classloader/id mapping in
order to save the id instead of a more longer representation.
So I propose something like that:
*public interface ClassResolver {*
*    // This method is invoked on the sender side to *
*    // marshal the information about the class.*
*    // where the context is a map style object that is reset/new for each
object graph serialization.*
*    public byte[] writeClass(Object o, Context context) throws
IgniteException;*

*    // This method is invoked on the receiving side to*
*    // retrieve the class based on provided information.*
*    // where the context is a map style object that is reset/new for each
object graph serialization.*
*    public Class<?> readClass(byte[], Context context) throws
IgniteException;*
*}*

I think your proposal will solve our issue and maybe also open a door for
the osgi development.
Let me know what do you think about me proposal? :)

Thanks,

Romain

Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <[hidden email]> a
écrit :

> Romain,
>
> Can you comment on the ClassLoaderResolver suggestion that I proposed
> earlier? Will it solve your problem?
>
> D.
>
> On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <[hidden email]>
> wrote:
>
> > Hi Raul,
> >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> >  - I like the point 2. Maybe for a graph of object that come from
> different
> > packages / bundles you may have to recursively capture the package
> version
> > of the individual element of your graph.
> >  - For the point 3 I wonder if it will not over-complexify the solution.
> As
> > a developer you will have to think about it. And it is not obvious in the
> > code where things are serialized or not. You may use lambda in your
> > application code therefore the current package become what you call a DTO
> > package. The current state of ignite does not enforce you to specify it
> for
> > "classical" classloading application. If you introduce this extra step
> for
> > OSGi ready application you will maybe discourage people to use OSGi.
> >
> > To comeback to our solution. We start we a strong assumption: our cluster
> > is homogeneous in term of code so, of course, it simplify our life :).
> >
> > BTW if you can introduce an extension point in the class resolution
> > mechanism it can be interesting for end user in order to allow them to
> > optimize it based on there specific use cases.
> >
> > Romain.
> >
> > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a écrit :
> >
> > > Hi Andrey,
> > >
> > > Thanks for the participation in this topic.
> > >
> > > I don't like the solution to incorporate the bundle symbolic name in
> the
> > > serialised form. Nothing guarantees that the classdef will be located
> > under
> > > the same bundle in both source and target machines. We also have to
> take
> > > into account that OSGi allows for multiple versions of the same
> > > bundle/packages to co-exist in the same  container. So it becomes more
> > > complex.
> > >
> > > Using the TCCL should work when serialising, but it'll probably be of
> no
> > > use when deserialising on the other end.
> > >
> > > I need to enhance Ignite to:
> > >
> > > 1. Use the TCCL when marshalling on one end.
> > > 2. Incorporate the package version of the class in the serialised form
> if
> > > Ignite is running in an OSGi environment.
> > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > through a custom OSGi manifest header or the like, as I explained
> before.
> > > Package version must be taken into account.
> > >
> > > What do you think?
> > >
> > > Raúl.
> > > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
> > >
> > > > Raul,
> > > >
> > > > The fundamental hurdle we need to jump over to make Ignite
> OSGi-enabled
> > > is
> > > > the marshalling. More specifically the issue is with deserialization
> of
> > > the
> > > > classes that are provided by the bundles other than the Ignite bundle
> > > > itself.
> > > >
> > > > When the Ignite transport layer receives a message it needs to figure
> > out
> > > > how to deserialize the bytes and for that it needs to know the bundle
> > > that
> > > > provides the class to be deserailized. At this point TCCL is of no
> use.
> > > To
> > > > make things more complex, the class may contain other classes that
> come
> > > > from other bundles, and so on recursively. This means that each
> object
> > in
> > > > the hierarchy must be serialized with its bundle name (or bundle id),
> > so
> > > > that the deserializer will then be able to correctly resolve the
> class
> > > > while traversing the object hierarchy during deserialization.
> > > >
> > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > > plug
> > > > in a custom class resolver. Romain's solution was to use Kryo that
> does
> > > > provide a way to customize class resolution. It has solved the
> problem
> > of
> > > > capturing the bundle info and he was able to successfully run Ignite
> > as a
> > > > bundle in an OSGi container (after some repackaging and inclusion of
> > the
> > > > manifest). But Kryo-based marshalling introduced a lot of complexity
> to
> > > the
> > > > code and incorrect use of Kryo's numerous serializers caused some
> weird
> > > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> > due
> > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > maintain.
> > > >
> > > > I feel the correct solution to OSGi problem would be to
> > > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > > original one) to be used in OSGi environment.
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > From: [hidden email]
> > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: [hidden email]
> > > > >
> > > > > Hi Romain,
> > > > >
> > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > > input.
> > > > >
> > > > > I'm thinking about the situation you describe and I wonder if
> you're
> > > > > exporting Ignite as an OSGi service which is then consumed from
> other
> > > > > bundles. Under this situation, it would be quite easy to reproduce
> > the
> > > > > behaviour you describe if Ignite is not resolving classes via the
> > TCCL.
> > > > > Need to dig deeper into that.
> > > > >
> > > > > Off the top of my head, there are two alternatives to solve it:
> > > > >
> > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> used) –
> > > we
> > > > > gotta be wary of possible regressions.
> > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> bundles
> > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > >
> > > > > Regards,
> > > > >
> > > > > *Raúl Kripalani*
> > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data
> > > and
> > > > > Messaging Engineer
> > > > > http://about.me/raulkripalani |
> > > http://www.linkedin.com/in/raulkripalani
> > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > >
> > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm really interested in this issue:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> stuff
> > to
> > > > make
> > > > > > it work in our osgi environment. The main issue for us now is the
> > > > > > serialization. I think it you will have to rework the
> > > > OptimizedMarshaller
> > > > > > or any other marshaller that works with object that come from
> > outside
> > > > your
> > > > > > class space.
> > > > > >
> > > > > > We have try kryo that works. Kryo provide an extension point in
> > order
> > > > to
> > > > > > resolve the classes:
> > > > > >
> > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > . With this extension we are able to solve the problem of
> external
> > > > classes.
> > > > > > The only issue with kryo is that some classes need a certain care
> > in
> > > > the
> > > > > > serialization process and therefore a specialized serializer.
> > > > > >
> > > > > > So I would like to know from the community what do think of
> > changing
> > > > the
> > > > > > way the optimized marshaller works or introducing the support of
> > yet
> > > > > > another marshaller based on a kryo like technology?
> > > > > >
> > > > > >
> > > > > > Thanks in advance,
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > >
> > > > > > Romain.
> > > > > >
> > > > > >
> > > > > > PS: I'm ready to help in the both cases.
> > > > > > "Misys" is the trade name of the Misys group of companies. This
> > email
> > > > and
> > > > > > any attachments have been scanned for known viruses using
> multiple
> > > > > > scanners. This email message is intended for the named recipient
> > > only.
> > > > It
> > > > > > may be privileged and/or confidential. If you are not the named
> > > > recipient
> > > > > > of this email please notify us immediately and do not copy it or
> > use
> > > > it for
> > > > > > any purpose, nor disclose its contents to any other person. This
> > > email
> > > > does
> > > > > > not constitute the commencement of legal relations between you
> and
> > > > Misys.
> > > > > > Please refer to the executed contract between you and the
> relevant
> > > > member
> > > > > > of the Misys group for the identity of the contracting party with
> > > > which you
> > > > > > are dealing.
> > > > > >
> > > >
> > >
> >
>

[Romain Gilles-2]

Hi Raul,
I had a look to what has been done in camel. I think here the use cases
came be different.
Let say I want to run a closure against the grid like this:

IgniteCompute compute  = ignite.compute();
// Execute closure on all cluster nodes.Collection<Integer> res = compute.apply(
    String::length,
    Arrays.asList("How many characters".split(" "))
);
     // Add all the word lengths received from cluster nodes.

int total = res.stream().mapToInt(Integer::intValue).sum();

Then I think, even in more complex and realistic use cases, you may don't
want to export the implementation details of your closure. But this closure
will be serialized by the Ignite marshalling framework.

Romain.

Le mar. 3 nov. 2015 à 20:18, Raul Kripalani <[hidden email]> a écrit :

> Hey guys,
>
> About the TCCL, I need to dig deeper into how serialisation is being done
> now. If, at any point, we are resolving a class through the classloader of
> a particular class, e.g. Ignition.getClass().getClassLoader(), etc. we are
> using the class space of the bundle containing that class. If we are using
> a class from Ignite, we obviously wouldn't be able to find a custom DTO
> provided by a user.
>
> In Camel we have found this problem several times and we've solved it by
> changing the TCCL, or using the TCCL [1] [2] to resolve classes.
>
> As I said, I need to look into this deeper and I'll have some time on
> Thursday, so I hope you don't mind waiting a little bit. I have already
> "bundle-fied" most of Ignite modules in the corresponding branch [3], so my
> next step is to test out the Ignite examples inside an OSGi environment
> (Karaf 4).
>
> [1] https://issues.apache.org/jira/browse/CAMEL-5748
> [2] https://issues.apache.org/jira/browse/CAMEL-5722
> [3] https://github.com/apache/ignite/tree/ignite-1527
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Tue, Nov 3, 2015 at 10:38 AM, Romain Gilles <[hidden email]>
> wrote:
>
> > Hi Raul,
> >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> >  - I like the point 2. Maybe for a graph of object that come from
> different
> > packages / bundles you may have to recursively capture the package
> version
> > of the individual element of your graph.
> >  - For the point 3 I wonder if it will not over-complexify the solution.
> As
> > a developer you will have to think about it. And it is not obvious in the
> > code where things are serialized or not. You may use lambda in your
> > application code therefore the current package become what you call a DTO
> > package. The current state of ignite does not enforce you to specify it
> for
> > "classical" classloading application. If you introduce this extra step
> for
> > OSGi ready application you will maybe discourage people to use OSGi.
> >
> > To comeback to our solution. We start we a strong assumption: our cluster
> > is homogeneous in term of code so, of course, it simplify our life :).
> >
> > BTW if you can introduce an extension point in the class resolution
> > mechanism it can be interesting for end user in order to allow them to
> > optimize it based on there specific use cases.
> >
> > Romain.
> >
> > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a écrit :
> >
> > > Hi Andrey,
> > >
> > > Thanks for the participation in this topic.
> > >
> > > I don't like the solution to incorporate the bundle symbolic name in
> the
> > > serialised form. Nothing guarantees that the classdef will be located
> > under
> > > the same bundle in both source and target machines. We also have to
> take
> > > into account that OSGi allows for multiple versions of the same
> > > bundle/packages to co-exist in the same  container. So it becomes more
> > > complex.
> > >
> > > Using the TCCL should work when serialising, but it'll probably be of
> no
> > > use when deserialising on the other end.
> > >
> > > I need to enhance Ignite to:
> > >
> > > 1. Use the TCCL when marshalling on one end.
> > > 2. Incorporate the package version of the class in the serialised form
> if
> > > Ignite is running in an OSGi environment.
> > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > through a custom OSGi manifest header or the like, as I explained
> before.
> > > Package version must be taken into account.
> > >
> > > What do you think?
> > >
> > > Raúl.
> > > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
> > >
> > > > Raul,
> > > >
> > > > The fundamental hurdle we need to jump over to make Ignite
> OSGi-enabled
> > > is
> > > > the marshalling. More specifically the issue is with deserialization
> of
> > > the
> > > > classes that are provided by the bundles other than the Ignite bundle
> > > > itself.
> > > >
> > > > When the Ignite transport layer receives a message it needs to figure
> > out
> > > > how to deserialize the bytes and for that it needs to know the bundle
> > > that
> > > > provides the class to be deserailized. At this point TCCL is of no
> use.
> > > To
> > > > make things more complex, the class may contain other classes that
> come
> > > > from other bundles, and so on recursively. This means that each
> object
> > in
> > > > the hierarchy must be serialized with its bundle name (or bundle id),
> > so
> > > > that the deserializer will then be able to correctly resolve the
> class
> > > > while traversing the object hierarchy during deserialization.
> > > >
> > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > > plug
> > > > in a custom class resolver. Romain's solution was to use Kryo that
> does
> > > > provide a way to customize class resolution. It has solved the
> problem
> > of
> > > > capturing the bundle info and he was able to successfully run Ignite
> > as a
> > > > bundle in an OSGi container (after some repackaging and inclusion of
> > the
> > > > manifest). But Kryo-based marshalling introduced a lot of complexity
> to
> > > the
> > > > code and incorrect use of Kryo's numerous serializers caused some
> weird
> > > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> > due
> > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > maintain.
> > > >
> > > > I feel the correct solution to OSGi problem would be to
> > > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > > original one) to be used in OSGi environment.
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > From: [hidden email]
> > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: [hidden email]
> > > > >
> > > > > Hi Romain,
> > > > >
> > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > > input.
> > > > >
> > > > > I'm thinking about the situation you describe and I wonder if
> you're
> > > > > exporting Ignite as an OSGi service which is then consumed from
> other
> > > > > bundles. Under this situation, it would be quite easy to reproduce
> > the
> > > > > behaviour you describe if Ignite is not resolving classes via the
> > TCCL.
> > > > > Need to dig deeper into that.
> > > > >
> > > > > Off the top of my head, there are two alternatives to solve it:
> > > > >
> > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> used) –
> > > we
> > > > > gotta be wary of possible regressions.
> > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> bundles
> > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > >
> > > > > Regards,
> > > > >
> > > > > *Raúl Kripalani*
> > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data
> > > and
> > > > > Messaging Engineer
> > > > > http://about.me/raulkripalani |
> > > http://www.linkedin.com/in/raulkripalani
> > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > >
> > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm really interested in this issue:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> stuff
> > to
> > > > make
> > > > > > it work in our osgi environment. The main issue for us now is the
> > > > > > serialization. I think it you will have to rework the
> > > > OptimizedMarshaller
> > > > > > or any other marshaller that works with object that come from
> > outside
> > > > your
> > > > > > class space.
> > > > > >
> > > > > > We have try kryo that works. Kryo provide an extension point in
> > order
> > > > to
> > > > > > resolve the classes:
> > > > > >
> > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > . With this extension we are able to solve the problem of
> external
> > > > classes.
> > > > > > The only issue with kryo is that some classes need a certain care
> > in
> > > > the
> > > > > > serialization process and therefore a specialized serializer.
> > > > > >
> > > > > > So I would like to know from the community what do think of
> > changing
> > > > the
> > > > > > way the optimized marshaller works or introducing the support of
> > yet
> > > > > > another marshaller based on a kryo like technology?
> > > > > >
> > > > > >
> > > > > > Thanks in advance,
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > >
> > > > > > Romain.
> > > > > >
> > > > > >
> > > > > > PS: I'm ready to help in the both cases.
> > > > > > "Misys" is the trade name of the Misys group of companies. This
> > email
> > > > and
> > > > > > any attachments have been scanned for known viruses using
> multiple
> > > > > > scanners. This email message is intended for the named recipient
> > > only.
> > > > It
> > > > > > may be privileged and/or confidential. If you are not the named
> > > > recipient
> > > > > > of this email please notify us immediately and do not copy it or
> > use
> > > > it for
> > > > > > any purpose, nor disclose its contents to any other person. This
> > > email
> > > > does
> > > > > > not constitute the commencement of legal relations between you
> and
> > > > Misys.
> > > > > > Please refer to the executed contract between you and the
> relevant
> > > > member
> > > > > > of the Misys group for the identity of the contracting party with
> > > > which you
> > > > > > are dealing.
> > > > > >
> > > >
> > >
> >
>

[Andrey Kornev]

Romain/Dmitriy,

I prefer Romain's approach, but just curious, in the API you guys are proposing why use a byte[] rather than OutputStream/InputStream? With a byte[], one would inevitably end up wrapping it into a byte stream class. Also, the stream-based interface would be more in line with the Marshaller API.

Also for symmetry with the readClass() method, I suggest the writeClass() take a Class<?> rather than an object.

Regards
Andrey

> From: [hidden email]
> Date: Tue, 3 Nov 2015 21:24:01 +0000
> Subject: Re: OSGi migration may require a special marshaller
> To: [hidden email]
>
> Hi Dmitriy,
> I think your solution is good. Maybe I will change it a little bit... :P
> I think you should delegate the Class resolution to the resolver. Because
> for example with your current solution the marshaller may before (or after)
> store the fqn of the class (maybe only the first time it encounter it) but
> in order to save the classloader context resolution the implementation of
> the resolver may have to save the package name of the given object and some
> extra info therefore the content package name will be serialized two times.
> Ok, it's not a big deal.
> But now if the resolver identify that it can reuse the same class loader
> for a couple of classes. It will be interesting for it to have a
> serialization context in order to save, let say, classloader/id mapping in
> order to save the id instead of a more longer representation.
> So I propose something like that:
> *public interface ClassResolver {*
> *    // This method is invoked on the sender side to *
> *    // marshal the information about the class.*
> *    // where the context is a map style object that is reset/new for each
> object graph serialization.*
> *    public byte[] writeClass(Object o, Context context) throws
> IgniteException;*
>
> *    // This method is invoked on the receiving side to*
> *    // retrieve the class based on provided information.*
> *    // where the context is a map style object that is reset/new for each
> object graph serialization.*
> *    public Class<?> readClass(byte[], Context context) throws
> IgniteException;*
> *}*
>
> I think your proposal will solve our issue and maybe also open a door for
> the osgi development.
> Let me know what do you think about me proposal? :)
>
> Thanks,
>
> Romain
>
> Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <[hidden email]> a
> écrit :
>
> > Romain,
> >
> > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > earlier? Will it solve your problem?
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <[hidden email]>
> > wrote:
> >
> > > Hi Raul,
> > >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> > >  - I like the point 2. Maybe for a graph of object that come from
> > different
> > > packages / bundles you may have to recursively capture the package
> > version
> > > of the individual element of your graph.
> > >  - For the point 3 I wonder if it will not over-complexify the solution.
> > As
> > > a developer you will have to think about it. And it is not obvious in the
> > > code where things are serialized or not. You may use lambda in your
> > > application code therefore the current package become what you call a DTO
> > > package. The current state of ignite does not enforce you to specify it
> > for
> > > "classical" classloading application. If you introduce this extra step
> > for
> > > OSGi ready application you will maybe discourage people to use OSGi.
> > >
> > > To comeback to our solution. We start we a strong assumption: our cluster
> > > is homogeneous in term of code so, of course, it simplify our life :).
> > >
> > > BTW if you can introduce an extension point in the class resolution
> > > mechanism it can be interesting for end user in order to allow them to
> > > optimize it based on there specific use cases.
> > >
> > > Romain.
> > >
> > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a écrit :
> > >
> > > > Hi Andrey,
> > > >
> > > > Thanks for the participation in this topic.
> > > >
> > > > I don't like the solution to incorporate the bundle symbolic name in
> > the
> > > > serialised form. Nothing guarantees that the classdef will be located
> > > under
> > > > the same bundle in both source and target machines. We also have to
> > take
> > > > into account that OSGi allows for multiple versions of the same
> > > > bundle/packages to co-exist in the same  container. So it becomes more
> > > > complex.
> > > >
> > > > Using the TCCL should work when serialising, but it'll probably be of
> > no
> > > > use when deserialising on the other end.
> > > >
> > > > I need to enhance Ignite to:
> > > >
> > > > 1. Use the TCCL when marshalling on one end.
> > > > 2. Incorporate the package version of the class in the serialised form
> > if
> > > > Ignite is running in an OSGi environment.
> > > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > > through a custom OSGi manifest header or the like, as I explained
> > before.
> > > > Package version must be taken into account.
> > > >
> > > > What do you think?
> > > >
> > > > Raúl.
> > > > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]> wrote:
> > > >
> > > > > Raul,
> > > > >
> > > > > The fundamental hurdle we need to jump over to make Ignite
> > OSGi-enabled
> > > > is
> > > > > the marshalling. More specifically the issue is with deserialization
> > of
> > > > the
> > > > > classes that are provided by the bundles other than the Ignite bundle
> > > > > itself.
> > > > >
> > > > > When the Ignite transport layer receives a message it needs to figure
> > > out
> > > > > how to deserialize the bytes and for that it needs to know the bundle
> > > > that
> > > > > provides the class to be deserailized. At this point TCCL is of no
> > use.
> > > > To
> > > > > make things more complex, the class may contain other classes that
> > come
> > > > > from other bundles, and so on recursively. This means that each
> > object
> > > in
> > > > > the hierarchy must be serialized with its bundle name (or bundle id),
> > > so
> > > > > that the deserializer will then be able to correctly resolve the
> > class
> > > > > while traversing the object hierarchy during deserialization.
> > > > >
> > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > > > plug
> > > > > in a custom class resolver. Romain's solution was to use Kryo that
> > does
> > > > > provide a way to customize class resolution. It has solved the
> > problem
> > > of
> > > > > capturing the bundle info and he was able to successfully run Ignite
> > > as a
> > > > > bundle in an OSGi container (after some repackaging and inclusion of
> > > the
> > > > > manifest). But Kryo-based marshalling introduced a lot of complexity
> > to
> > > > the
> > > > > code and incorrect use of Kryo's numerous serializers caused some
> > weird
> > > > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> > > due
> > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > > maintain.
> > > > >
> > > > > I feel the correct solution to OSGi problem would be to
> > > > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > > > original one) to be used in OSGi environment.
> > > > >
> > > > > Regards
> > > > > Andrey
> > > > >
> > > > > > From: [hidden email]
> > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > To: [hidden email]
> > > > > >
> > > > > > Hi Romain,
> > > > > >
> > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > > > input.
> > > > > >
> > > > > > I'm thinking about the situation you describe and I wonder if
> > you're
> > > > > > exporting Ignite as an OSGi service which is then consumed from
> > other
> > > > > > bundles. Under this situation, it would be quite easy to reproduce
> > > the
> > > > > > behaviour you describe if Ignite is not resolving classes via the
> > > TCCL.
> > > > > > Need to dig deeper into that.
> > > > > >
> > > > > > Off the top of my head, there are two alternatives to solve it:
> > > > > >
> > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > used) –
> > > > we
> > > > > > gotta be wary of possible regressions.
> > > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> > bundles
> > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > *Raúl Kripalani*
> > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> > Data
> > > > and
> > > > > > Messaging Engineer
> > > > > > http://about.me/raulkripalani |
> > > > http://www.linkedin.com/in/raulkripalani
> > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > >
> > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm really interested in this issue:
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> > stuff
> > > to
> > > > > make
> > > > > > > it work in our osgi environment. The main issue for us now is the
> > > > > > > serialization. I think it you will have to rework the
> > > > > OptimizedMarshaller
> > > > > > > or any other marshaller that works with object that come from
> > > outside
> > > > > your
> > > > > > > class space.
> > > > > > >
> > > > > > > We have try kryo that works. Kryo provide an extension point in
> > > order
> > > > > to
> > > > > > > resolve the classes:
> > > > > > >
> > > > >
> > > >
> > >
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > . With this extension we are able to solve the problem of
> > external
> > > > > classes.
> > > > > > > The only issue with kryo is that some classes need a certain care
> > > in
> > > > > the
> > > > > > > serialization process and therefore a specialized serializer.
> > > > > > >
> > > > > > > So I would like to know from the community what do think of
> > > changing
> > > > > the
> > > > > > > way the optimized marshaller works or introducing the support of
> > > yet
> > > > > > > another marshaller based on a kryo like technology?
> > > > > > >
> > > > > > >
> > > > > > > Thanks in advance,
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > >
> > > > > > > Romain.
> > > > > > >
> > > > > > >
> > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > "Misys" is the trade name of the Misys group of companies. This
> > > email
> > > > > and
> > > > > > > any attachments have been scanned for known viruses using
> > multiple
> > > > > > > scanners. This email message is intended for the named recipient
> > > > only.
> > > > > It
> > > > > > > may be privileged and/or confidential. If you are not the named
> > > > > recipient
> > > > > > > of this email please notify us immediately and do not copy it or
> > > use
> > > > > it for
> > > > > > > any purpose, nor disclose its contents to any other person. This
> > > > email
> > > > > does
> > > > > > > not constitute the commencement of legal relations between you
> > and
> > > > > Misys.
> > > > > > > Please refer to the executed contract between you and the
> > relevant
> > > > > member
> > > > > > > of the Misys group for the identity of the contracting party with
> > > > > which you
> > > > > > > are dealing.
> > > > > > >
> > > > >
> > > >
> > >
> >

     

[Romain Gilles-2]

Hi,
Maybe a missing point. I think but I'm not sure that in the
OptimizedMarshaller there is a caching of already serialized classes. Due
to the dynamic nature of OSGi this may lead to memory leak. In fact if a
bundle is refreshed, it will produce a new BundleRevision and therefore a
new classloader. And if you don't release the class from the previous
BundleRevision then you endup with memory leak. So maybe the Marshaller
interface or somewhere should provide a way to free those classes /
classloaders.

Regards,

Romain

Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <[hidden email]> a
écrit :

> Romain/Dmitriy,
>
> I prefer Romain's approach, but just curious, in the API you guys are
> proposing why use a byte[] rather than OutputStream/InputStream? With a
> byte[], one would inevitably end up wrapping it into a byte stream class.
> Also, the stream-based interface would be more in line with the Marshaller
> API.
>
> Also for symmetry with the readClass() method, I suggest the writeClass()
> take a Class<?> rather than an object.
>
> Regards
> Andrey
>
> > From: [hidden email]
> > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > Subject: Re: OSGi migration may require a special marshaller
> > To: [hidden email]
> >
> > Hi Dmitriy,
> > I think your solution is good. Maybe I will change it a little bit... :P
> > I think you should delegate the Class resolution to the resolver. Because
> > for example with your current solution the marshaller may before (or
> after)
> > store the fqn of the class (maybe only the first time it encounter it)
> but
> > in order to save the classloader context resolution the implementation of
> > the resolver may have to save the package name of the given object and
> some
> > extra info therefore the content package name will be serialized two
> times.
> > Ok, it's not a big deal.
> > But now if the resolver identify that it can reuse the same class loader
> > for a couple of classes. It will be interesting for it to have a
> > serialization context in order to save, let say, classloader/id mapping
> in
> > order to save the id instead of a more longer representation.
> > So I propose something like that:
> > *public interface ClassResolver {*
> > *    // This method is invoked on the sender side to *
> > *    // marshal the information about the class.*
> > *    // where the context is a map style object that is reset/new for
> each
> > object graph serialization.*
> > *    public byte[] writeClass(Object o, Context context) throws
> > IgniteException;*
> >
> > *    // This method is invoked on the receiving side to*
> > *    // retrieve the class based on provided information.*
> > *    // where the context is a map style object that is reset/new for
> each
> > object graph serialization.*
> > *    public Class<?> readClass(byte[], Context context) throws
> > IgniteException;*
> > *}*
> >
> > I think your proposal will solve our issue and maybe also open a door for
> > the osgi development.
> > Let me know what do you think about me proposal? :)
> >
> > Thanks,
> >
> > Romain
> >
> > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <[hidden email]> a
> > écrit :
> >
> > > Romain,
> > >
> > > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > > earlier? Will it solve your problem?
> > >
> > > D.
> > >
> > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <[hidden email]
> >
> > > wrote:
> > >
> > > > Hi Raul,
> > > >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes
> how?
> > > >  - I like the point 2. Maybe for a graph of object that come from
> > > different
> > > > packages / bundles you may have to recursively capture the package
> > > version
> > > > of the individual element of your graph.
> > > >  - For the point 3 I wonder if it will not over-complexify the
> solution.
> > > As
> > > > a developer you will have to think about it. And it is not obvious
> in the
> > > > code where things are serialized or not. You may use lambda in your
> > > > application code therefore the current package become what you call
> a DTO
> > > > package. The current state of ignite does not enforce you to specify
> it
> > > for
> > > > "classical" classloading application. If you introduce this extra
> step
> > > for
> > > > OSGi ready application you will maybe discourage people to use OSGi.
> > > >
> > > > To comeback to our solution. We start we a strong assumption: our
> cluster
> > > > is homogeneous in term of code so, of course, it simplify our life
> :).
> > > >
> > > > BTW if you can introduce an extension point in the class resolution
> > > > mechanism it can be interesting for end user in order to allow them
> to
> > > > optimize it based on there specific use cases.
> > > >
> > > > Romain.
> > > >
> > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a
> écrit :
> > > >
> > > > > Hi Andrey,
> > > > >
> > > > > Thanks for the participation in this topic.
> > > > >
> > > > > I don't like the solution to incorporate the bundle symbolic name
> in
> > > the
> > > > > serialised form. Nothing guarantees that the classdef will be
> located
> > > > under
> > > > > the same bundle in both source and target machines. We also have to
> > > take
> > > > > into account that OSGi allows for multiple versions of the same
> > > > > bundle/packages to co-exist in the same  container. So it becomes
> more
> > > > > complex.
> > > > >
> > > > > Using the TCCL should work when serialising, but it'll probably be
> of
> > > no
> > > > > use when deserialising on the other end.
> > > > >
> > > > > I need to enhance Ignite to:
> > > > >
> > > > > 1. Use the TCCL when marshalling on one end.
> > > > > 2. Incorporate the package version of the class in the serialised
> form
> > > if
> > > > > Ignite is running in an OSGi environment.
> > > > > 3. On the receiver end, discover cache entities / DTOs in all
> bundles
> > > > > through a custom OSGi manifest header or the like, as I explained
> > > before.
> > > > > Package version must be taken into account.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Raúl.
> > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]>
> wrote:
> > > > >
> > > > > > Raul,
> > > > > >
> > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > OSGi-enabled
> > > > > is
> > > > > > the marshalling. More specifically the issue is with
> deserialization
> > > of
> > > > > the
> > > > > > classes that are provided by the bundles other than the Ignite
> bundle
> > > > > > itself.
> > > > > >
> > > > > > When the Ignite transport layer receives a message it needs to
> figure
> > > > out
> > > > > > how to deserialize the bytes and for that it needs to know the
> bundle
> > > > > that
> > > > > > provides the class to be deserailized. At this point TCCL is of
> no
> > > use.
> > > > > To
> > > > > > make things more complex, the class may contain other classes
> that
> > > come
> > > > > > from other bundles, and so on recursively. This means that each
> > > object
> > > > in
> > > > > > the hierarchy must be serialized with its bundle name (or bundle
> id),
> > > > so
> > > > > > that the deserializer will then be able to correctly resolve the
> > > class
> > > > > > while traversing the object hierarchy during deserialization.
> > > > > >
> > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> ability to
> > > > > plug
> > > > > > in a custom class resolver. Romain's solution was to use Kryo
> that
> > > does
> > > > > > provide a way to customize class resolution. It has solved the
> > > problem
> > > > of
> > > > > > capturing the bundle info and he was able to successfully run
> Ignite
> > > > as a
> > > > > > bundle in an OSGi container (after some repackaging and
> inclusion of
> > > > the
> > > > > > manifest). But Kryo-based marshalling introduced a lot of
> complexity
> > > to
> > > > > the
> > > > > > code and incorrect use of Kryo's numerous serializers caused some
> > > weird
> > > > > > hard-to-debug issues in the Ignite core (like duplicate cache
> entries
> > > > due
> > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class --
> go
> > > > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > > > maintain.
> > > > > >
> > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> resolution.
> > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to
> the
> > > > > > original one) to be used in OSGi environment.
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > > From: [hidden email]
> > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: [hidden email]
> > > > > > >
> > > > > > > Hi Romain,
> > > > > > >
> > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate
> your
> > > > > input.
> > > > > > >
> > > > > > > I'm thinking about the situation you describe and I wonder if
> > > you're
> > > > > > > exporting Ignite as an OSGi service which is then consumed from
> > > other
> > > > > > > bundles. Under this situation, it would be quite easy to
> reproduce
> > > > the
> > > > > > > behaviour you describe if Ignite is not resolving classes via
> the
> > > > TCCL.
> > > > > > > Need to dig deeper into that.
> > > > > > >
> > > > > > > Off the top of my head, there are two alternatives to solve it:
> > > > > > >
> > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > > used) –
> > > > > we
> > > > > > > gotta be wary of possible regressions.
> > > > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> > > bundles
> > > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > *Raúl Kripalani*
> > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> Big
> > > Data
> > > > > and
> > > > > > > Messaging Engineer
> > > > > > > http://about.me/raulkripalani |
> > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > >
> > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'm really interested in this issue:
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> > > stuff
> > > > to
> > > > > > make
> > > > > > > > it work in our osgi environment. The main issue for us now
> is the
> > > > > > > > serialization. I think it you will have to rework the
> > > > > > OptimizedMarshaller
> > > > > > > > or any other marshaller that works with object that come from
> > > > outside
> > > > > > your
> > > > > > > > class space.
> > > > > > > >
> > > > > > > > We have try kryo that works. Kryo provide an extension point
> in
> > > > order
> > > > > > to
> > > > > > > > resolve the classes:
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > . With this extension we are able to solve the problem of
> > > external
> > > > > > classes.
> > > > > > > > The only issue with kryo is that some classes need a certain
> care
> > > > in
> > > > > > the
> > > > > > > > serialization process and therefore a specialized serializer.
> > > > > > > >
> > > > > > > > So I would like to know from the community what do think of
> > > > changing
> > > > > > the
> > > > > > > > way the optimized marshaller works or introducing the
> support of
> > > > yet
> > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks in advance,
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > >
> > > > > > > > Romain.
> > > > > > > >
> > > > > > > >
> > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > "Misys" is the trade name of the Misys group of companies.
> This
> > > > email
> > > > > > and
> > > > > > > > any attachments have been scanned for known viruses using
> > > multiple
> > > > > > > > scanners. This email message is intended for the named
> recipient
> > > > > only.
> > > > > > It
> > > > > > > > may be privileged and/or confidential. If you are not the
> named
> > > > > > recipient
> > > > > > > > of this email please notify us immediately and do not copy
> it or
> > > > use
> > > > > > it for
> > > > > > > > any purpose, nor disclose its contents to any other person.
> This
> > > > > email
> > > > > > does
> > > > > > > > not constitute the commencement of legal relations between
> you
> > > and
> > > > > > Misys.
> > > > > > > > Please refer to the executed contract between you and the
> > > relevant
> > > > > > member
> > > > > > > > of the Misys group for the identity of the contracting party
> with
> > > > > > which you
> > > > > > > > are dealing.
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

[dsetrakyan]

Romain,

In the upcoming release we will be deprecating the OptimizedMarshaller and
will be switching to a default internal marshaller (which is based on the
new PortableMarshaller donated by GridGain).

Having said that, we may be able to pass BinaryWriter and BinaryReader
instead of byte arrays. This will be pretty close to passing the stream, as
suggested by Andrey.

Also, I still think that we should only resolve class loaders and not the
class itself. The main reason is that Ignite will encode class names into
an integer hash code and will store the integer->class-fqn mapping in
internal replicated cache. I doubt users will get more efficient than an
integer (4 bytes) for a class name.

On the receiving side, once we are able to get the right class loader, we
can easily get the proper class by calling ClassLoader.findClass(class-fqn).

Thoughts?

D.

On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <[hidden email]>
wrote:

> Hi,
> Maybe a missing point. I think but I'm not sure that in the
> OptimizedMarshaller there is a caching of already serialized classes. Due
> to the dynamic nature of OSGi this may lead to memory leak. In fact if a
> bundle is refreshed, it will produce a new BundleRevision and therefore a
> new classloader. And if you don't release the class from the previous
> BundleRevision then you endup with memory leak. So maybe the Marshaller
> interface or somewhere should provide a way to free those classes /
> classloaders.
>
> Regards,
>
> Romain
>
> Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <[hidden email]> a
> écrit :
>
> > Romain/Dmitriy,
> >
> > I prefer Romain's approach, but just curious, in the API you guys are
> > proposing why use a byte[] rather than OutputStream/InputStream? With a
> > byte[], one would inevitably end up wrapping it into a byte stream class.
> > Also, the stream-based interface would be more in line with the
> Marshaller
> > API.
> >
> > Also for symmetry with the readClass() method, I suggest the writeClass()
> > take a Class<?> rather than an object.
> >
> > Regards
> > Andrey
> >
> > > From: [hidden email]
> > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: [hidden email]
> > >
> > > Hi Dmitriy,
> > > I think your solution is good. Maybe I will change it a little bit...
> :P
> > > I think you should delegate the Class resolution to the resolver.
> Because
> > > for example with your current solution the marshaller may before (or
> > after)
> > > store the fqn of the class (maybe only the first time it encounter it)
> > but
> > > in order to save the classloader context resolution the implementation
> of
> > > the resolver may have to save the package name of the given object and
> > some
> > > extra info therefore the content package name will be serialized two
> > times.
> > > Ok, it's not a big deal.
> > > But now if the resolver identify that it can reuse the same class
> loader
> > > for a couple of classes. It will be interesting for it to have a
> > > serialization context in order to save, let say, classloader/id mapping
> > in
> > > order to save the id instead of a more longer representation.
> > > So I propose something like that:
> > > *public interface ClassResolver {*
> > > *    // This method is invoked on the sender side to *
> > > *    // marshal the information about the class.*
> > > *    // where the context is a map style object that is reset/new for
> > each
> > > object graph serialization.*
> > > *    public byte[] writeClass(Object o, Context context) throws
> > > IgniteException;*
> > >
> > > *    // This method is invoked on the receiving side to*
> > > *    // retrieve the class based on provided information.*
> > > *    // where the context is a map style object that is reset/new for
> > each
> > > object graph serialization.*
> > > *    public Class<?> readClass(byte[], Context context) throws
> > > IgniteException;*
> > > *}*
> > >
> > > I think your proposal will solve our issue and maybe also open a door
> for
> > > the osgi development.
> > > Let me know what do you think about me proposal? :)
> > >
> > > Thanks,
> > >
> > > Romain
> > >
> > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <[hidden email]>
> a
> > > écrit :
> > >
> > > > Romain,
> > > >
> > > > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > > > earlier? Will it solve your problem?
> > > >
> > > > D.
> > > >
> > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Hi Raul,
> > > > >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes
> > how?
> > > > >  - I like the point 2. Maybe for a graph of object that come from
> > > > different
> > > > > packages / bundles you may have to recursively capture the package
> > > > version
> > > > > of the individual element of your graph.
> > > > >  - For the point 3 I wonder if it will not over-complexify the
> > solution.
> > > > As
> > > > > a developer you will have to think about it. And it is not obvious
> > in the
> > > > > code where things are serialized or not. You may use lambda in your
> > > > > application code therefore the current package become what you call
> > a DTO
> > > > > package. The current state of ignite does not enforce you to
> specify
> > it
> > > > for
> > > > > "classical" classloading application. If you introduce this extra
> > step
> > > > for
> > > > > OSGi ready application you will maybe discourage people to use
> OSGi.
> > > > >
> > > > > To comeback to our solution. We start we a strong assumption: our
> > cluster
> > > > > is homogeneous in term of code so, of course, it simplify our life
> > :).
> > > > >
> > > > > BTW if you can introduce an extension point in the class resolution
> > > > > mechanism it can be interesting for end user in order to allow them
> > to
> > > > > optimize it based on there specific use cases.
> > > > >
> > > > > Romain.
> > > > >
> > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a
> > écrit :
> > > > >
> > > > > > Hi Andrey,
> > > > > >
> > > > > > Thanks for the participation in this topic.
> > > > > >
> > > > > > I don't like the solution to incorporate the bundle symbolic name
> > in
> > > > the
> > > > > > serialised form. Nothing guarantees that the classdef will be
> > located
> > > > > under
> > > > > > the same bundle in both source and target machines. We also have
> to
> > > > take
> > > > > > into account that OSGi allows for multiple versions of the same
> > > > > > bundle/packages to co-exist in the same  container. So it becomes
> > more
> > > > > > complex.
> > > > > >
> > > > > > Using the TCCL should work when serialising, but it'll probably
> be
> > of
> > > > no
> > > > > > use when deserialising on the other end.
> > > > > >
> > > > > > I need to enhance Ignite to:
> > > > > >
> > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > 2. Incorporate the package version of the class in the serialised
> > form
> > > > if
> > > > > > Ignite is running in an OSGi environment.
> > > > > > 3. On the receiver end, discover cache entities / DTOs in all
> > bundles
> > > > > > through a custom OSGi manifest header or the like, as I explained
> > > > before.
> > > > > > Package version must be taken into account.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Raúl.
> > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]>
> > wrote:
> > > > > >
> > > > > > > Raul,
> > > > > > >
> > > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > > OSGi-enabled
> > > > > > is
> > > > > > > the marshalling. More specifically the issue is with
> > deserialization
> > > > of
> > > > > > the
> > > > > > > classes that are provided by the bundles other than the Ignite
> > bundle
> > > > > > > itself.
> > > > > > >
> > > > > > > When the Ignite transport layer receives a message it needs to
> > figure
> > > > > out
> > > > > > > how to deserialize the bytes and for that it needs to know the
> > bundle
> > > > > > that
> > > > > > > provides the class to be deserailized. At this point TCCL is of
> > no
> > > > use.
> > > > > > To
> > > > > > > make things more complex, the class may contain other classes
> > that
> > > > come
> > > > > > > from other bundles, and so on recursively. This means that each
> > > > object
> > > > > in
> > > > > > > the hierarchy must be serialized with its bundle name (or
> bundle
> > id),
> > > > > so
> > > > > > > that the deserializer will then be able to correctly resolve
> the
> > > > class
> > > > > > > while traversing the object hierarchy during deserialization.
> > > > > > >
> > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> > ability to
> > > > > > plug
> > > > > > > in a custom class resolver. Romain's solution was to use Kryo
> > that
> > > > does
> > > > > > > provide a way to customize class resolution. It has solved the
> > > > problem
> > > > > of
> > > > > > > capturing the bundle info and he was able to successfully run
> > Ignite
> > > > > as a
> > > > > > > bundle in an OSGi container (after some repackaging and
> > inclusion of
> > > > > the
> > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > complexity
> > > > to
> > > > > > the
> > > > > > > code and incorrect use of Kryo's numerous serializers caused
> some
> > > > weird
> > > > > > > hard-to-debug issues in the Ignite core (like duplicate cache
> > entries
> > > > > due
> > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class --
> > go
> > > > > > > figure!). Overall the Kryo-based solution is brittle and hard
> to
> > > > > > maintain.
> > > > > > >
> > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > resolution.
> > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to
> > the
> > > > > > > original one) to be used in OSGi environment.
> > > > > > >
> > > > > > > Regards
> > > > > > > Andrey
> > > > > > >
> > > > > > > > From: [hidden email]
> > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > > To: [hidden email]
> > > > > > > >
> > > > > > > > Hi Romain,
> > > > > > > >
> > > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate
> > your
> > > > > > input.
> > > > > > > >
> > > > > > > > I'm thinking about the situation you describe and I wonder if
> > > > you're
> > > > > > > > exporting Ignite as an OSGi service which is then consumed
> from
> > > > other
> > > > > > > > bundles. Under this situation, it would be quite easy to
> > reproduce
> > > > > the
> > > > > > > > behaviour you describe if Ignite is not resolving classes via
> > the
> > > > > TCCL.
> > > > > > > > Need to dig deeper into that.
> > > > > > > >
> > > > > > > > Off the top of my head, there are two alternatives to solve
> it:
> > > > > > > >
> > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > > > used) –
> > > > > > we
> > > > > > > > gotta be wary of possible regressions.
> > > > > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> > > > bundles
> > > > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > *Raúl Kripalani*
> > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> > Big
> > > > Data
> > > > > > and
> > > > > > > > Messaging Engineer
> > > > > > > > http://about.me/raulkripalani |
> > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > >
> > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'm really interested in this issue:
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We
> some
> > > > stuff
> > > > > to
> > > > > > > make
> > > > > > > > > it work in our osgi environment. The main issue for us now
> > is the
> > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > OptimizedMarshaller
> > > > > > > > > or any other marshaller that works with object that come
> from
> > > > > outside
> > > > > > > your
> > > > > > > > > class space.
> > > > > > > > >
> > > > > > > > > We have try kryo that works. Kryo provide an extension
> point
> > in
> > > > > order
> > > > > > > to
> > > > > > > > > resolve the classes:
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > . With this extension we are able to solve the problem of
> > > > external
> > > > > > > classes.
> > > > > > > > > The only issue with kryo is that some classes need a
> certain
> > care
> > > > > in
> > > > > > > the
> > > > > > > > > serialization process and therefore a specialized
> serializer.
> > > > > > > > >
> > > > > > > > > So I would like to know from the community what do think of
> > > > > changing
> > > > > > > the
> > > > > > > > > way the optimized marshaller works or introducing the
> > support of
> > > > > yet
> > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks in advance,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Romain.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > "Misys" is the trade name of the Misys group of companies.
> > This
> > > > > email
> > > > > > > and
> > > > > > > > > any attachments have been scanned for known viruses using
> > > > multiple
> > > > > > > > > scanners. This email message is intended for the named
> > recipient
> > > > > > only.
> > > > > > > It
> > > > > > > > > may be privileged and/or confidential. If you are not the
> > named
> > > > > > > recipient
> > > > > > > > > of this email please notify us immediately and do not copy
> > it or
> > > > > use
> > > > > > > it for
> > > > > > > > > any purpose, nor disclose its contents to any other person.
> > This
> > > > > > email
> > > > > > > does
> > > > > > > > > not constitute the commencement of legal relations between
> > you
> > > > and
> > > > > > > Misys.
> > > > > > > > > Please refer to the executed contract between you and the
> > > > relevant
> > > > > > > member
> > > > > > > > > of the Misys group for the identity of the contracting
> party
> > with
> > > > > > > which you
> > > > > > > > > are dealing.
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>

[Andrey Kornev]

Dmitriy,

I think your approach will work, but I let Romain respond.

Also, in terms of the implementation, please keep in mind that the resolver must be called for each non-JDK and non-Ignite core class (it would probably make sense to eschew storing class loaders for such classes in favor of compactness of the serialized representation -- see below). Also, it's worth keeping a cache of already resolved class loaders per marshaller invocation (this is probably the context that Romain has mentioned in his previous posting) to minimize the number of the resolver calls.

In terms of the resolver's implementation, the simplest way to serialize the class loader would be by capturing two pieces of information (both strings): the bundle symbolic name and the bundle version. This approach however may result in bloating of the serialized representation: I'd roughly estimate the overhead per element to be at least 20-30 bytes (the length of the symbolic name string, plus the length of the version string). There are way to reduce the overhead (like serializing the hash code of the bundle id string rather than the string itself, and then come up with a clever way of resolving the hash collisions), but they all come at cost of increased complexity...

An alternative approach would be rely on the special bundle id which is an integer and is generated by the OSGi container. But in this case, all nodes must ensure that all the bundles have consistent ids (bundle A with id 42 on node N1, has the same id on every other node) which is not easy -- while not entirely impossible -- to guarantee. As long as the nodes are homogeneous (have the same set of bundles deployed) the OSGi container is guaranteed to assign to the bundles the same ids.

Thanks
Andrey

> From: [hidden email]
> Date: Tue, 3 Nov 2015 16:29:41 -0800
> Subject: Re: OSGi migration may require a special marshaller
> To: [hidden email]
>
> Romain,
>
> In the upcoming release we will be deprecating the OptimizedMarshaller and
> will be switching to a default internal marshaller (which is based on the
> new PortableMarshaller donated by GridGain).
>
> Having said that, we may be able to pass BinaryWriter and BinaryReader
> instead of byte arrays. This will be pretty close to passing the stream, as
> suggested by Andrey.
>
> Also, I still think that we should only resolve class loaders and not the
> class itself. The main reason is that Ignite will encode class names into
> an integer hash code and will store the integer->class-fqn mapping in
> internal replicated cache. I doubt users will get more efficient than an
> integer (4 bytes) for a class name.
>
> On the receiving side, once we are able to get the right class loader, we
> can easily get the proper class by calling ClassLoader.findClass(class-fqn).
>
> Thoughts?
>
> D.
>
> On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <[hidden email]>
> wrote:
>
> > Hi,
> > Maybe a missing point. I think but I'm not sure that in the
> > OptimizedMarshaller there is a caching of already serialized classes. Due
> > to the dynamic nature of OSGi this may lead to memory leak. In fact if a
> > bundle is refreshed, it will produce a new BundleRevision and therefore a
> > new classloader. And if you don't release the class from the previous
> > BundleRevision then you endup with memory leak. So maybe the Marshaller
> > interface or somewhere should provide a way to free those classes /
> > classloaders.
> >
> > Regards,
> >
> > Romain
> >
> > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <[hidden email]> a
> > écrit :
> >
> > > Romain/Dmitriy,
> > >
> > > I prefer Romain's approach, but just curious, in the API you guys are
> > > proposing why use a byte[] rather than OutputStream/InputStream? With a
> > > byte[], one would inevitably end up wrapping it into a byte stream class.
> > > Also, the stream-based interface would be more in line with the
> > Marshaller
> > > API.
> > >
> > > Also for symmetry with the readClass() method, I suggest the writeClass()
> > > take a Class<?> rather than an object.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: [hidden email]
> > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: [hidden email]
> > > >
> > > > Hi Dmitriy,
> > > > I think your solution is good. Maybe I will change it a little bit...
> > :P
> > > > I think you should delegate the Class resolution to the resolver.
> > Because
> > > > for example with your current solution the marshaller may before (or
> > > after)
> > > > store the fqn of the class (maybe only the first time it encounter it)
> > > but
> > > > in order to save the classloader context resolution the implementation
> > of
> > > > the resolver may have to save the package name of the given object and
> > > some
> > > > extra info therefore the content package name will be serialized two
> > > times.
> > > > Ok, it's not a big deal.
> > > > But now if the resolver identify that it can reuse the same class
> > loader
> > > > for a couple of classes. It will be interesting for it to have a
> > > > serialization context in order to save, let say, classloader/id mapping
> > > in
> > > > order to save the id instead of a more longer representation.
> > > > So I propose something like that:
> > > > *public interface ClassResolver {*
> > > > *    // This method is invoked on the sender side to *
> > > > *    // marshal the information about the class.*
> > > > *    // where the context is a map style object that is reset/new for
> > > each
> > > > object graph serialization.*
> > > > *    public byte[] writeClass(Object o, Context context) throws
> > > > IgniteException;*
> > > >
> > > > *    // This method is invoked on the receiving side to*
> > > > *    // retrieve the class based on provided information.*
> > > > *    // where the context is a map style object that is reset/new for
> > > each
> > > > object graph serialization.*
> > > > *    public Class<?> readClass(byte[], Context context) throws
> > > > IgniteException;*
> > > > *}*
> > > >
> > > > I think your proposal will solve our issue and maybe also open a door
> > for
> > > > the osgi development.
> > > > Let me know what do you think about me proposal? :)
> > > >
> > > > Thanks,
> > > >
> > > > Romain
> > > >
> > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <[hidden email]>
> > a
> > > > écrit :
> > > >
> > > > > Romain,
> > > > >
> > > > > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > > > > earlier? Will it solve your problem?
> > > > >
> > > > > D.
> > > > >
> > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Raul,
> > > > > >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes
> > > how?
> > > > > >  - I like the point 2. Maybe for a graph of object that come from
> > > > > different
> > > > > > packages / bundles you may have to recursively capture the package
> > > > > version
> > > > > > of the individual element of your graph.
> > > > > >  - For the point 3 I wonder if it will not over-complexify the
> > > solution.
> > > > > As
> > > > > > a developer you will have to think about it. And it is not obvious
> > > in the
> > > > > > code where things are serialized or not. You may use lambda in your
> > > > > > application code therefore the current package become what you call
> > > a DTO
> > > > > > package. The current state of ignite does not enforce you to
> > specify
> > > it
> > > > > for
> > > > > > "classical" classloading application. If you introduce this extra
> > > step
> > > > > for
> > > > > > OSGi ready application you will maybe discourage people to use
> > OSGi.
> > > > > >
> > > > > > To comeback to our solution. We start we a strong assumption: our
> > > cluster
> > > > > > is homogeneous in term of code so, of course, it simplify our life
> > > :).
> > > > > >
> > > > > > BTW if you can introduce an extension point in the class resolution
> > > > > > mechanism it can be interesting for end user in order to allow them
> > > to
> > > > > > optimize it based on there specific use cases.
> > > > > >
> > > > > > Romain.
> > > > > >
> > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]> a
> > > écrit :
> > > > > >
> > > > > > > Hi Andrey,
> > > > > > >
> > > > > > > Thanks for the participation in this topic.
> > > > > > >
> > > > > > > I don't like the solution to incorporate the bundle symbolic name
> > > in
> > > > > the
> > > > > > > serialised form. Nothing guarantees that the classdef will be
> > > located
> > > > > > under
> > > > > > > the same bundle in both source and target machines. We also have
> > to
> > > > > take
> > > > > > > into account that OSGi allows for multiple versions of the same
> > > > > > > bundle/packages to co-exist in the same  container. So it becomes
> > > more
> > > > > > > complex.
> > > > > > >
> > > > > > > Using the TCCL should work when serialising, but it'll probably
> > be
> > > of
> > > > > no
> > > > > > > use when deserialising on the other end.
> > > > > > >
> > > > > > > I need to enhance Ignite to:
> > > > > > >
> > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > 2. Incorporate the package version of the class in the serialised
> > > form
> > > > > if
> > > > > > > Ignite is running in an OSGi environment.
> > > > > > > 3. On the receiver end, discover cache entities / DTOs in all
> > > bundles
> > > > > > > through a custom OSGi manifest header or the like, as I explained
> > > > > before.
> > > > > > > Package version must be taken into account.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Raúl.
> > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <[hidden email]>
> > > wrote:
> > > > > > >
> > > > > > > > Raul,
> > > > > > > >
> > > > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > > > OSGi-enabled
> > > > > > > is
> > > > > > > > the marshalling. More specifically the issue is with
> > > deserialization
> > > > > of
> > > > > > > the
> > > > > > > > classes that are provided by the bundles other than the Ignite
> > > bundle
> > > > > > > > itself.
> > > > > > > >
> > > > > > > > When the Ignite transport layer receives a message it needs to
> > > figure
> > > > > > out
> > > > > > > > how to deserialize the bytes and for that it needs to know the
> > > bundle
> > > > > > > that
> > > > > > > > provides the class to be deserailized. At this point TCCL is of
> > > no
> > > > > use.
> > > > > > > To
> > > > > > > > make things more complex, the class may contain other classes
> > > that
> > > > > come
> > > > > > > > from other bundles, and so on recursively. This means that each
> > > > > object
> > > > > > in
> > > > > > > > the hierarchy must be serialized with its bundle name (or
> > bundle
> > > id),
> > > > > > so
> > > > > > > > that the deserializer will then be able to correctly resolve
> > the
> > > > > class
> > > > > > > > while traversing the object hierarchy during deserialization.
> > > > > > > >
> > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> > > ability to
> > > > > > > plug
> > > > > > > > in a custom class resolver. Romain's solution was to use Kryo
> > > that
> > > > > does
> > > > > > > > provide a way to customize class resolution. It has solved the
> > > > > problem
> > > > > > of
> > > > > > > > capturing the bundle info and he was able to successfully run
> > > Ignite
> > > > > > as a
> > > > > > > > bundle in an OSGi container (after some repackaging and
> > > inclusion of
> > > > > > the
> > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > complexity
> > > > > to
> > > > > > > the
> > > > > > > > code and incorrect use of Kryo's numerous serializers caused
> > some
> > > > > weird
> > > > > > > > hard-to-debug issues in the Ignite core (like duplicate cache
> > > entries
> > > > > > due
> > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class --
> > > go
> > > > > > > > figure!). Overall the Kryo-based solution is brittle and hard
> > to
> > > > > > > maintain.
> > > > > > > >
> > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > resolution.
> > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to
> > > the
> > > > > > > > original one) to be used in OSGi environment.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Andrey
> > > > > > > >
> > > > > > > > > From: [hidden email]
> > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > > > To: [hidden email]
> > > > > > > > >
> > > > > > > > > Hi Romain,
> > > > > > > > >
> > > > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate
> > > your
> > > > > > > input.
> > > > > > > > >
> > > > > > > > > I'm thinking about the situation you describe and I wonder if
> > > > > you're
> > > > > > > > > exporting Ignite as an OSGi service which is then consumed
> > from
> > > > > other
> > > > > > > > > bundles. Under this situation, it would be quite easy to
> > > reproduce
> > > > > > the
> > > > > > > > > behaviour you describe if Ignite is not resolving classes via
> > > the
> > > > > > TCCL.
> > > > > > > > > Need to dig deeper into that.
> > > > > > > > >
> > > > > > > > > Off the top of my head, there are two alternatives to solve
> > it:
> > > > > > > > >
> > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > > > > used) –
> > > > > > > we
> > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> > > > > bundles
> > > > > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > *Raúl Kripalani*
> > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> > > Big
> > > > > Data
> > > > > > > and
> > > > > > > > > Messaging Engineer
> > > > > > > > > http://about.me/raulkripalani |
> > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > >
> > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We
> > some
> > > > > stuff
> > > > > > to
> > > > > > > > make
> > > > > > > > > > it work in our osgi environment. The main issue for us now
> > > is the
> > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > OptimizedMarshaller
> > > > > > > > > > or any other marshaller that works with object that come
> > from
> > > > > > outside
> > > > > > > > your
> > > > > > > > > > class space.
> > > > > > > > > >
> > > > > > > > > > We have try kryo that works. Kryo provide an extension
> > point
> > > in
> > > > > > order
> > > > > > > > to
> > > > > > > > > > resolve the classes:
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > . With this extension we are able to solve the problem of
> > > > > external
> > > > > > > > classes.
> > > > > > > > > > The only issue with kryo is that some classes need a
> > certain
> > > care
> > > > > > in
> > > > > > > > the
> > > > > > > > > > serialization process and therefore a specialized
> > serializer.
> > > > > > > > > >
> > > > > > > > > > So I would like to know from the community what do think of
> > > > > > changing
> > > > > > > > the
> > > > > > > > > > way the optimized marshaller works or introducing the
> > > support of
> > > > > > yet
> > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks in advance,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Romain.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > "Misys" is the trade name of the Misys group of companies.
> > > This
> > > > > > email
> > > > > > > > and
> > > > > > > > > > any attachments have been scanned for known viruses using
> > > > > multiple
> > > > > > > > > > scanners. This email message is intended for the named
> > > recipient
> > > > > > > only.
> > > > > > > > It
> > > > > > > > > > may be privileged and/or confidential. If you are not the
> > > named
> > > > > > > > recipient
> > > > > > > > > > of this email please notify us immediately and do not copy
> > > it or
> > > > > > use
> > > > > > > > it for
> > > > > > > > > > any purpose, nor disclose its contents to any other person.
> > > This
> > > > > > > email
> > > > > > > > does
> > > > > > > > > > not constitute the commencement of legal relations between
> > > you
> > > > > and
> > > > > > > > Misys.
> > > > > > > > > > Please refer to the executed contract between you and the
> > > > > relevant
> > > > > > > > member
> > > > > > > > > > of the Misys group for the identity of the contracting
> > party
> > > with
> > > > > > > > which you
> > > > > > > > > > are dealing.
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >

     

[dsetrakyan]

Andrey, et al,

Can you provide a way on how to get a required Bundle object based on, say,
bundle symbolic name and version?

I have reached the end of the internet trying to find a way in OSGI to look
up a Bundle based on something other than an actual Class belonging to that
bundle, but no luck.

D.

On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <[hidden email]>
wrote:

> Dmitriy,
>
> I think your approach will work, but I let Romain respond.
>
> Also, in terms of the implementation, please keep in mind that the
> resolver must be called for each non-JDK and non-Ignite core class (it
> would probably make sense to eschew storing class loaders for such classes
> in favor of compactness of the serialized representation -- see below).
> Also, it's worth keeping a cache of already resolved class loaders per
> marshaller invocation (this is probably the context that Romain has
> mentioned in his previous posting) to minimize the number of the resolver
> calls.
>
> In terms of the resolver's implementation, the simplest way to serialize
> the class loader would be by capturing two pieces of information (both
> strings): the bundle symbolic name and the bundle version. This approach
> however may result in bloating of the serialized representation: I'd
> roughly estimate the overhead per element to be at least 20-30 bytes (the
> length of the symbolic name string, plus the length of the version string).
> There are way to reduce the overhead (like serializing the hash code of the
> bundle id string rather than the string itself, and then come up with a
> clever way of resolving the hash collisions), but they all come at cost of
> increased complexity...
>
> An alternative approach would be rely on the special bundle id which is an
> integer and is generated by the OSGi container. But in this case, all nodes
> must ensure that all the bundles have consistent ids (bundle A with id 42
> on node N1, has the same id on every other node) which is not easy -- while
> not entirely impossible -- to guarantee. As long as the nodes are
> homogeneous (have the same set of bundles deployed) the OSGi container is
> guaranteed to assign to the bundles the same ids.
>
> Thanks
> Andrey
>
> > From: [hidden email]
> > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > Subject: Re: OSGi migration may require a special marshaller
> > To: [hidden email]
> >
> > Romain,
> >
> > In the upcoming release we will be deprecating the OptimizedMarshaller
> and
> > will be switching to a default internal marshaller (which is based on the
> > new PortableMarshaller donated by GridGain).
> >
> > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > instead of byte arrays. This will be pretty close to passing the stream,
> as
> > suggested by Andrey.
> >
> > Also, I still think that we should only resolve class loaders and not the
> > class itself. The main reason is that Ignite will encode class names into
> > an integer hash code and will store the integer->class-fqn mapping in
> > internal replicated cache. I doubt users will get more efficient than an
> > integer (4 bytes) for a class name.
> >
> > On the receiving side, once we are able to get the right class loader, we
> > can easily get the proper class by calling
> ClassLoader.findClass(class-fqn).
> >
> > Thoughts?
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <[hidden email]>
> > wrote:
> >
> > > Hi,
> > > Maybe a missing point. I think but I'm not sure that in the
> > > OptimizedMarshaller there is a caching of already serialized classes.
> Due
> > > to the dynamic nature of OSGi this may lead to memory leak. In fact if
> a
> > > bundle is refreshed, it will produce a new BundleRevision and
> therefore a
> > > new classloader. And if you don't release the class from the previous
> > > BundleRevision then you endup with memory leak. So maybe the Marshaller
> > > interface or somewhere should provide a way to free those classes /
> > > classloaders.
> > >
> > > Regards,
> > >
> > > Romain
> > >
> > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <[hidden email]>
> a
> > > écrit :
> > >
> > > > Romain/Dmitriy,
> > > >
> > > > I prefer Romain's approach, but just curious, in the API you guys are
> > > > proposing why use a byte[] rather than OutputStream/InputStream?
> With a
> > > > byte[], one would inevitably end up wrapping it into a byte stream
> class.
> > > > Also, the stream-based interface would be more in line with the
> > > Marshaller
> > > > API.
> > > >
> > > > Also for symmetry with the readClass() method, I suggest the
> writeClass()
> > > > take a Class<?> rather than an object.
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > From: [hidden email]
> > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: [hidden email]
> > > > >
> > > > > Hi Dmitriy,
> > > > > I think your solution is good. Maybe I will change it a little
> bit...
> > > :P
> > > > > I think you should delegate the Class resolution to the resolver.
> > > Because
> > > > > for example with your current solution the marshaller may before
> (or
> > > > after)
> > > > > store the fqn of the class (maybe only the first time it encounter
> it)
> > > > but
> > > > > in order to save the classloader context resolution the
> implementation
> > > of
> > > > > the resolver may have to save the package name of the given object
> and
> > > > some
> > > > > extra info therefore the content package name will be serialized
> two
> > > > times.
> > > > > Ok, it's not a big deal.
> > > > > But now if the resolver identify that it can reuse the same class
> > > loader
> > > > > for a couple of classes. It will be interesting for it to have a
> > > > > serialization context in order to save, let say, classloader/id
> mapping
> > > > in
> > > > > order to save the id instead of a more longer representation.
> > > > > So I propose something like that:
> > > > > *public interface ClassResolver {*
> > > > > *    // This method is invoked on the sender side to *
> > > > > *    // marshal the information about the class.*
> > > > > *    // where the context is a map style object that is reset/new
> for
> > > > each
> > > > > object graph serialization.*
> > > > > *    public byte[] writeClass(Object o, Context context) throws
> > > > > IgniteException;*
> > > > >
> > > > > *    // This method is invoked on the receiving side to*
> > > > > *    // retrieve the class based on provided information.*
> > > > > *    // where the context is a map style object that is reset/new
> for
> > > > each
> > > > > object graph serialization.*
> > > > > *    public Class<?> readClass(byte[], Context context) throws
> > > > > IgniteException;*
> > > > > *}*
> > > > >
> > > > > I think your proposal will solve our issue and maybe also open a
> door
> > > for
> > > > > the osgi development.
> > > > > Let me know what do you think about me proposal? :)
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Romain
> > > > >
> > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> [hidden email]>
> > > a
> > > > > écrit :
> > > > >
> > > > > > Romain,
> > > > > >
> > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> proposed
> > > > > > earlier? Will it solve your problem?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Raul,
> > > > > > >  - Do you plan to use the TCCL when marshalling in OSGi env?
> If yes
> > > > how?
> > > > > > >  - I like the point 2. Maybe for a graph of object that come
> from
> > > > > > different
> > > > > > > packages / bundles you may have to recursively capture the
> package
> > > > > > version
> > > > > > > of the individual element of your graph.
> > > > > > >  - For the point 3 I wonder if it will not over-complexify the
> > > > solution.
> > > > > > As
> > > > > > > a developer you will have to think about it. And it is not
> obvious
> > > > in the
> > > > > > > code where things are serialized or not. You may use lambda in
> your
> > > > > > > application code therefore the current package become what you
> call
> > > > a DTO
> > > > > > > package. The current state of ignite does not enforce you to
> > > specify
> > > > it
> > > > > > for
> > > > > > > "classical" classloading application. If you introduce this
> extra
> > > > step
> > > > > > for
> > > > > > > OSGi ready application you will maybe discourage people to use
> > > OSGi.
> > > > > > >
> > > > > > > To comeback to our solution. We start we a strong assumption:
> our
> > > > cluster
> > > > > > > is homogeneous in term of code so, of course, it simplify our
> life
> > > > :).
> > > > > > >
> > > > > > > BTW if you can introduce an extension point in the class
> resolution
> > > > > > > mechanism it can be interesting for end user in order to allow
> them
> > > > to
> > > > > > > optimize it based on there specific use cases.
> > > > > > >
> > > > > > > Romain.
> > > > > > >
> > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[hidden email]>
> a
> > > > écrit :
> > > > > > >
> > > > > > > > Hi Andrey,
> > > > > > > >
> > > > > > > > Thanks for the participation in this topic.
> > > > > > > >
> > > > > > > > I don't like the solution to incorporate the bundle symbolic
> name
> > > > in
> > > > > > the
> > > > > > > > serialised form. Nothing guarantees that the classdef will be
> > > > located
> > > > > > > under
> > > > > > > > the same bundle in both source and target machines. We also
> have
> > > to
> > > > > > take
> > > > > > > > into account that OSGi allows for multiple versions of the
> same
> > > > > > > > bundle/packages to co-exist in the same  container. So it
> becomes
> > > > more
> > > > > > > > complex.
> > > > > > > >
> > > > > > > > Using the TCCL should work when serialising, but it'll
> probably
> > > be
> > > > of
> > > > > > no
> > > > > > > > use when deserialising on the other end.
> > > > > > > >
> > > > > > > > I need to enhance Ignite to:
> > > > > > > >
> > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > 2. Incorporate the package version of the class in the
> serialised
> > > > form
> > > > > > if
> > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > 3. On the receiver end, discover cache entities / DTOs in all
> > > > bundles
> > > > > > > > through a custom OSGi manifest header or the like, as I
> explained
> > > > > > before.
> > > > > > > > Package version must be taken into account.
> > > > > > > >
> > > > > > > > What do you think?
> > > > > > > >
> > > > > > > > Raúl.
> > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> [hidden email]>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Raul,
> > > > > > > > >
> > > > > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > > > > OSGi-enabled
> > > > > > > > is
> > > > > > > > > the marshalling. More specifically the issue is with
> > > > deserialization
> > > > > > of
> > > > > > > > the
> > > > > > > > > classes that are provided by the bundles other than the
> Ignite
> > > > bundle
> > > > > > > > > itself.
> > > > > > > > >
> > > > > > > > > When the Ignite transport layer receives a message it
> needs to
> > > > figure
> > > > > > > out
> > > > > > > > > how to deserialize the bytes and for that it needs to know
> the
> > > > bundle
> > > > > > > > that
> > > > > > > > > provides the class to be deserailized. At this point TCCL
> is of
> > > > no
> > > > > > use.
> > > > > > > > To
> > > > > > > > > make things more complex, the class may contain other
> classes
> > > > that
> > > > > > come
> > > > > > > > > from other bundles, and so on recursively. This means that
> each
> > > > > > object
> > > > > > > in
> > > > > > > > > the hierarchy must be serialized with its bundle name (or
> > > bundle
> > > > id),
> > > > > > > so
> > > > > > > > > that the deserializer will then be able to correctly
> resolve
> > > the
> > > > > > class
> > > > > > > > > while traversing the object hierarchy during
> deserialization.
> > > > > > > > >
> > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> > > > ability to
> > > > > > > > plug
> > > > > > > > > in a custom class resolver. Romain's solution was to use
> Kryo
> > > > that
> > > > > > does
> > > > > > > > > provide a way to customize class resolution. It has solved
> the
> > > > > > problem
> > > > > > > of
> > > > > > > > > capturing the bundle info and he was able to successfully
> run
> > > > Ignite
> > > > > > > as a
> > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > inclusion of
> > > > > > > the
> > > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > > complexity
> > > > > > to
> > > > > > > > the
> > > > > > > > > code and incorrect use of Kryo's numerous serializers
> caused
> > > some
> > > > > > weird
> > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> cache
> > > > entries
> > > > > > > due
> > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> class --
> > > > go
> > > > > > > > > figure!). Overall the Kryo-based solution is brittle and
> hard
> > > to
> > > > > > > > maintain.
> > > > > > > > >
> > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > > resolution.
> > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> addition to
> > > > the
> > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Andrey
> > > > > > > > >
> > > > > > > > > > From: [hidden email]
> > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > Subject: Re: OSGi migration may require a special
> marshaller
> > > > > > > > > > To: [hidden email]
> > > > > > > > > >
> > > > > > > > > > Hi Romain,
> > > > > > > > > >
> > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> appreciate
> > > > your
> > > > > > > > input.
> > > > > > > > > >
> > > > > > > > > > I'm thinking about the situation you describe and I
> wonder if
> > > > > > you're
> > > > > > > > > > exporting Ignite as an OSGi service which is then
> consumed
> > > from
> > > > > > other
> > > > > > > > > > bundles. Under this situation, it would be quite easy to
> > > > reproduce
> > > > > > > the
> > > > > > > > > > behaviour you describe if Ignite is not resolving
> classes via
> > > > the
> > > > > > > TCCL.
> > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > >
> > > > > > > > > > Off the top of my head, there are two alternatives to
> solve
> > > it:
> > > > > > > > > >
> > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> already
> > > > > > used) –
> > > > > > > > we
> > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package so
> that
> > > > > > bundles
> > > > > > > > > > containing DTOs can expose packages to Ignite's
> marshallers.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> Integration,
> > > > Big
> > > > > > Data
> > > > > > > > and
> > > > > > > > > > Messaging Engineer
> > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We
> > > some
> > > > > > stuff
> > > > > > > to
> > > > > > > > > make
> > > > > > > > > > > it work in our osgi environment. The main issue for us
> now
> > > > is the
> > > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > or any other marshaller that works with object that
> come
> > > from
> > > > > > > outside
> > > > > > > > > your
> > > > > > > > > > > class space.
> > > > > > > > > > >
> > > > > > > > > > > We have try kryo that works. Kryo provide an extension
> > > point
> > > > in
> > > > > > > order
> > > > > > > > > to
> > > > > > > > > > > resolve the classes:
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > . With this extension we are able to solve the problem
> of
> > > > > > external
> > > > > > > > > classes.
> > > > > > > > > > > The only issue with kryo is that some classes need a
> > > certain
> > > > care
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > serialization process and therefore a specialized
> > > serializer.
> > > > > > > > > > >
> > > > > > > > > > > So I would like to know from the community what do
> think of
> > > > > > > changing
> > > > > > > > > the
> > > > > > > > > > > way the optimized marshaller works or introducing the
> > > > support of
> > > > > > > yet
> > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks in advance,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Romain.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > "Misys" is the trade name of the Misys group of
> companies.
> > > > This
> > > > > > > email
> > > > > > > > > and
> > > > > > > > > > > any attachments have been scanned for known viruses
> using
> > > > > > multiple
> > > > > > > > > > > scanners. This email message is intended for the named
> > > > recipient
> > > > > > > > only.
> > > > > > > > > It
> > > > > > > > > > > may be privileged and/or confidential. If you are not
> the
> > > > named
> > > > > > > > > recipient
> > > > > > > > > > > of this email please notify us immediately and do not
> copy
> > > > it or
> > > > > > > use
> > > > > > > > > it for
> > > > > > > > > > > any purpose, nor disclose its contents to any other
> person.
> > > > This
> > > > > > > > email
> > > > > > > > > does
> > > > > > > > > > > not constitute the commencement of legal relations
> between
> > > > you
> > > > > > and
> > > > > > > > > Misys.
> > > > > > > > > > > Please refer to the executed contract between you and
> the
> > > > > > relevant
> > > > > > > > > member
> > > > > > > > > > > of the Misys group for the identity of the contracting
> > > party
> > > > with
> > > > > > > > > which you
> > > > > > > > > > > are dealing.
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
>
>

[Raul Kripalani-2]

Could we rewind a little bit please? What is the point of using the bundle
symbolic name in the first place?

In OSGi, it is bad practice to refer to bundles directly. A bundle is a
packaging unit, that's all, but it should not have any further
consideration.

It is perfectly feasible that a given package is exported by bundle A in a
container, and by bundle B in another. One should not make any assumptions
in that regard. A simple example of this is an enterprise-wide deployment
of Ignite, where multiple apps in different OSGi containers share cache
entities. Why would one expect these entities to be hosted in the same
bundle across all apps?

Referring to a bundle in OSGi world is like referring to a JAR in a normal
Java environment. One would not always expect to resolve class A.B.C from
bundle xyz-1.2.3.jar, right? You would resolve from the classpath.

There are other mechanisms to reach into the OSGi environment and resolve a
class arbitrarily from another bundle. Some of which I've pointed out. But
relying on a bundle symbolic name receives an -1 from me.

Can't elaborate now, I'm travelling.

Will get back to my office tomorrow and I'll dig deeper into possibilities.

Regards,
Raúl.
On 4 Nov 2015 03:07, "Dmitriy Setrakyan" <[hidden email]> wrote:

> Andrey, et al,
>
> Can you provide a way on how to get a required Bundle object based on, say,
> bundle symbolic name and version?
>
> I have reached the end of the internet trying to find a way in OSGI to look
> up a Bundle based on something other than an actual Class belonging to that
> bundle, but no luck.
>
> D.
>
> On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <[hidden email]>
> wrote:
>
> > Dmitriy,
> >
> > I think your approach will work, but I let Romain respond.
> >
> > Also, in terms of the implementation, please keep in mind that the
> > resolver must be called for each non-JDK and non-Ignite core class (it
> > would probably make sense to eschew storing class loaders for such
> classes
> > in favor of compactness of the serialized representation -- see below).
> > Also, it's worth keeping a cache of already resolved class loaders per
> > marshaller invocation (this is probably the context that Romain has
> > mentioned in his previous posting) to minimize the number of the resolver
> > calls.
> >
> > In terms of the resolver's implementation, the simplest way to serialize
> > the class loader would be by capturing two pieces of information (both
> > strings): the bundle symbolic name and the bundle version. This approach
> > however may result in bloating of the serialized representation: I'd
> > roughly estimate the overhead per element to be at least 20-30 bytes (the
> > length of the symbolic name string, plus the length of the version
> string).
> > There are way to reduce the overhead (like serializing the hash code of
> the
> > bundle id string rather than the string itself, and then come up with a
> > clever way of resolving the hash collisions), but they all come at cost
> of
> > increased complexity...
> >
> > An alternative approach would be rely on the special bundle id which is
> an
> > integer and is generated by the OSGi container. But in this case, all
> nodes
> > must ensure that all the bundles have consistent ids (bundle A with id 42
> > on node N1, has the same id on every other node) which is not easy --
> while
> > not entirely impossible -- to guarantee. As long as the nodes are
> > homogeneous (have the same set of bundles deployed) the OSGi container is
> > guaranteed to assign to the bundles the same ids.
> >
> > Thanks
> > Andrey
> >
> > > From: [hidden email]
> > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: [hidden email]
> > >
> > > Romain,
> > >
> > > In the upcoming release we will be deprecating the OptimizedMarshaller
> > and
> > > will be switching to a default internal marshaller (which is based on
> the
> > > new PortableMarshaller donated by GridGain).
> > >
> > > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > > instead of byte arrays. This will be pretty close to passing the
> stream,
> > as
> > > suggested by Andrey.
> > >
> > > Also, I still think that we should only resolve class loaders and not
> the
> > > class itself. The main reason is that Ignite will encode class names
> into
> > > an integer hash code and will store the integer->class-fqn mapping in
> > > internal replicated cache. I doubt users will get more efficient than
> an
> > > integer (4 bytes) for a class name.
> > >
> > > On the receiving side, once we are able to get the right class loader,
> we
> > > can easily get the proper class by calling
> > ClassLoader.findClass(class-fqn).
> > >
> > > Thoughts?
> > >
> > > D.
> > >
> > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <[hidden email]
> >
> > > wrote:
> > >
> > > > Hi,
> > > > Maybe a missing point. I think but I'm not sure that in the
> > > > OptimizedMarshaller there is a caching of already serialized classes.
> > Due
> > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> if
> > a
> > > > bundle is refreshed, it will produce a new BundleRevision and
> > therefore a
> > > > new classloader. And if you don't release the class from the previous
> > > > BundleRevision then you endup with memory leak. So maybe the
> Marshaller
> > > > interface or somewhere should provide a way to free those classes /
> > > > classloaders.
> > > >
> > > > Regards,
> > > >
> > > > Romain
> > > >
> > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <[hidden email]
> >
> > a
> > > > écrit :
> > > >
> > > > > Romain/Dmitriy,
> > > > >
> > > > > I prefer Romain's approach, but just curious, in the API you guys
> are
> > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > With a
> > > > > byte[], one would inevitably end up wrapping it into a byte stream
> > class.
> > > > > Also, the stream-based interface would be more in line with the
> > > > Marshaller
> > > > > API.
> > > > >
> > > > > Also for symmetry with the readClass() method, I suggest the
> > writeClass()
> > > > > take a Class<?> rather than an object.
> > > > >
> > > > > Regards
> > > > > Andrey
> > > > >
> > > > > > From: [hidden email]
> > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > To: [hidden email]
> > > > > >
> > > > > > Hi Dmitriy,
> > > > > > I think your solution is good. Maybe I will change it a little
> > bit...
> > > > :P
> > > > > > I think you should delegate the Class resolution to the resolver.
> > > > Because
> > > > > > for example with your current solution the marshaller may before
> > (or
> > > > > after)
> > > > > > store the fqn of the class (maybe only the first time it
> encounter
> > it)
> > > > > but
> > > > > > in order to save the classloader context resolution the
> > implementation
> > > > of
> > > > > > the resolver may have to save the package name of the given
> object
> > and
> > > > > some
> > > > > > extra info therefore the content package name will be serialized
> > two
> > > > > times.
> > > > > > Ok, it's not a big deal.
> > > > > > But now if the resolver identify that it can reuse the same class
> > > > loader
> > > > > > for a couple of classes. It will be interesting for it to have a
> > > > > > serialization context in order to save, let say, classloader/id
> > mapping
> > > > > in
> > > > > > order to save the id instead of a more longer representation.
> > > > > > So I propose something like that:
> > > > > > *public interface ClassResolver {*
> > > > > > *    // This method is invoked on the sender side to *
> > > > > > *    // marshal the information about the class.*
> > > > > > *    // where the context is a map style object that is reset/new
> > for
> > > > > each
> > > > > > object graph serialization.*
> > > > > > *    public byte[] writeClass(Object o, Context context) throws
> > > > > > IgniteException;*
> > > > > >
> > > > > > *    // This method is invoked on the receiving side to*
> > > > > > *    // retrieve the class based on provided information.*
> > > > > > *    // where the context is a map style object that is reset/new
> > for
> > > > > each
> > > > > > object graph serialization.*
> > > > > > *    public Class<?> readClass(byte[], Context context) throws
> > > > > > IgniteException;*
> > > > > > *}*
> > > > > >
> > > > > > I think your proposal will solve our issue and maybe also open a
> > door
> > > > for
> > > > > > the osgi development.
> > > > > > Let me know what do you think about me proposal? :)
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Romain
> > > > > >
> > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > [hidden email]>
> > > > a
> > > > > > écrit :
> > > > > >
> > > > > > > Romain,
> > > > > > >
> > > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> > proposed
> > > > > > > earlier? Will it solve your problem?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > [hidden email]
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Raul,
> > > > > > > >  - Do you plan to use the TCCL when marshalling in OSGi env?
> > If yes
> > > > > how?
> > > > > > > >  - I like the point 2. Maybe for a graph of object that come
> > from
> > > > > > > different
> > > > > > > > packages / bundles you may have to recursively capture the
> > package
> > > > > > > version
> > > > > > > > of the individual element of your graph.
> > > > > > > >  - For the point 3 I wonder if it will not over-complexify
> the
> > > > > solution.
> > > > > > > As
> > > > > > > > a developer you will have to think about it. And it is not
> > obvious
> > > > > in the
> > > > > > > > code where things are serialized or not. You may use lambda
> in
> > your
> > > > > > > > application code therefore the current package become what
> you
> > call
> > > > > a DTO
> > > > > > > > package. The current state of ignite does not enforce you to
> > > > specify
> > > > > it
> > > > > > > for
> > > > > > > > "classical" classloading application. If you introduce this
> > extra
> > > > > step
> > > > > > > for
> > > > > > > > OSGi ready application you will maybe discourage people to
> use
> > > > OSGi.
> > > > > > > >
> > > > > > > > To comeback to our solution. We start we a strong assumption:
> > our
> > > > > cluster
> > > > > > > > is homogeneous in term of code so, of course, it simplify our
> > life
> > > > > :).
> > > > > > > >
> > > > > > > > BTW if you can introduce an extension point in the class
> > resolution
> > > > > > > > mechanism it can be interesting for end user in order to
> allow
> > them
> > > > > to
> > > > > > > > optimize it based on there specific use cases.
> > > > > > > >
> > > > > > > > Romain.
> > > > > > > >
> > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <
> [hidden email]>
> > a
> > > > > écrit :
> > > > > > > >
> > > > > > > > > Hi Andrey,
> > > > > > > > >
> > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > >
> > > > > > > > > I don't like the solution to incorporate the bundle
> symbolic
> > name
> > > > > in
> > > > > > > the
> > > > > > > > > serialised form. Nothing guarantees that the classdef will
> be
> > > > > located
> > > > > > > > under
> > > > > > > > > the same bundle in both source and target machines. We also
> > have
> > > > to
> > > > > > > take
> > > > > > > > > into account that OSGi allows for multiple versions of the
> > same
> > > > > > > > > bundle/packages to co-exist in the same  container. So it
> > becomes
> > > > > more
> > > > > > > > > complex.
> > > > > > > > >
> > > > > > > > > Using the TCCL should work when serialising, but it'll
> > probably
> > > > be
> > > > > of
> > > > > > > no
> > > > > > > > > use when deserialising on the other end.
> > > > > > > > >
> > > > > > > > > I need to enhance Ignite to:
> > > > > > > > >
> > > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > > 2. Incorporate the package version of the class in the
> > serialised
> > > > > form
> > > > > > > if
> > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in
> all
> > > > > bundles
> > > > > > > > > through a custom OSGi manifest header or the like, as I
> > explained
> > > > > > > before.
> > > > > > > > > Package version must be taken into account.
> > > > > > > > >
> > > > > > > > > What do you think?
> > > > > > > > >
> > > > > > > > > Raúl.
> > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > [hidden email]>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Raul,
> > > > > > > > > >
> > > > > > > > > > The fundamental hurdle we need to jump over to make
> Ignite
> > > > > > > OSGi-enabled
> > > > > > > > > is
> > > > > > > > > > the marshalling. More specifically the issue is with
> > > > > deserialization
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > classes that are provided by the bundles other than the
> > Ignite
> > > > > bundle
> > > > > > > > > > itself.
> > > > > > > > > >
> > > > > > > > > > When the Ignite transport layer receives a message it
> > needs to
> > > > > figure
> > > > > > > > out
> > > > > > > > > > how to deserialize the bytes and for that it needs to
> know
> > the
> > > > > bundle
> > > > > > > > > that
> > > > > > > > > > provides the class to be deserailized. At this point TCCL
> > is of
> > > > > no
> > > > > > > use.
> > > > > > > > > To
> > > > > > > > > > make things more complex, the class may contain other
> > classes
> > > > > that
> > > > > > > come
> > > > > > > > > > from other bundles, and so on recursively. This means
> that
> > each
> > > > > > > object
> > > > > > > > in
> > > > > > > > > > the hierarchy must be serialized with its bundle name (or
> > > > bundle
> > > > > id),
> > > > > > > > so
> > > > > > > > > > that the deserializer will then be able to correctly
> > resolve
> > > > the
> > > > > > > class
> > > > > > > > > > while traversing the object hierarchy during
> > deserialization.
> > > > > > > > > >
> > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking
> the
> > > > > ability to
> > > > > > > > > plug
> > > > > > > > > > in a custom class resolver. Romain's solution was to use
> > Kryo
> > > > > that
> > > > > > > does
> > > > > > > > > > provide a way to customize class resolution. It has
> solved
> > the
> > > > > > > problem
> > > > > > > > of
> > > > > > > > > > capturing the bundle info and he was able to successfully
> > run
> > > > > Ignite
> > > > > > > > as a
> > > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > > inclusion of
> > > > > > > > the
> > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > > > complexity
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > code and incorrect use of Kryo's numerous serializers
> > caused
> > > > some
> > > > > > > weird
> > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> > cache
> > > > > entries
> > > > > > > > due
> > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > class --
> > > > > go
> > > > > > > > > > figure!). Overall the Kryo-based solution is brittle and
> > hard
> > > > to
> > > > > > > > > maintain.
> > > > > > > > > >
> > > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > > > resolution.
> > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> > addition to
> > > > > the
> > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Andrey
> > > > > > > > > >
> > > > > > > > > > > From: [hidden email]
> > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > > Subject: Re: OSGi migration may require a special
> > marshaller
> > > > > > > > > > > To: [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > Hi Romain,
> > > > > > > > > > >
> > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> > appreciate
> > > > > your
> > > > > > > > > input.
> > > > > > > > > > >
> > > > > > > > > > > I'm thinking about the situation you describe and I
> > wonder if
> > > > > > > you're
> > > > > > > > > > > exporting Ignite as an OSGi service which is then
> > consumed
> > > > from
> > > > > > > other
> > > > > > > > > > > bundles. Under this situation, it would be quite easy
> to
> > > > > reproduce
> > > > > > > > the
> > > > > > > > > > > behaviour you describe if Ignite is not resolving
> > classes via
> > > > > the
> > > > > > > > TCCL.
> > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > >
> > > > > > > > > > > Off the top of my head, there are two alternatives to
> > solve
> > > > it:
> > > > > > > > > > >
> > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> > already
> > > > > > > used) –
> > > > > > > > > we
> > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package
> so
> > that
> > > > > > > bundles
> > > > > > > > > > > containing DTOs can expose packages to Ignite's
> > marshallers.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> > Integration,
> > > > > Big
> > > > > > > Data
> > > > > > > > > and
> > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 .
> We
> > > > some
> > > > > > > stuff
> > > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > > it work in our osgi environment. The main issue for
> us
> > now
> > > > > is the
> > > > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > or any other marshaller that works with object that
> > come
> > > > from
> > > > > > > > outside
> > > > > > > > > > your
> > > > > > > > > > > > class space.
> > > > > > > > > > > >
> > > > > > > > > > > > We have try kryo that works. Kryo provide an
> extension
> > > > point
> > > > > in
> > > > > > > > order
> > > > > > > > > > to
> > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > . With this extension we are able to solve the
> problem
> > of
> > > > > > > external
> > > > > > > > > > classes.
> > > > > > > > > > > > The only issue with kryo is that some classes need a
> > > > certain
> > > > > care
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > serialization process and therefore a specialized
> > > > serializer.
> > > > > > > > > > > >
> > > > > > > > > > > > So I would like to know from the community what do
> > think of
> > > > > > > > changing
> > > > > > > > > > the
> > > > > > > > > > > > way the optimized marshaller works or introducing the
> > > > > support of
> > > > > > > > yet
> > > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Romain.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > > "Misys" is the trade name of the Misys group of
> > companies.
> > > > > This
> > > > > > > > email
> > > > > > > > > > and
> > > > > > > > > > > > any attachments have been scanned for known viruses
> > using
> > > > > > > multiple
> > > > > > > > > > > > scanners. This email message is intended for the
> named
> > > > > recipient
> > > > > > > > > only.
> > > > > > > > > > It
> > > > > > > > > > > > may be privileged and/or confidential. If you are not
> > the
> > > > > named
> > > > > > > > > > recipient
> > > > > > > > > > > > of this email please notify us immediately and do not
> > copy
> > > > > it or
> > > > > > > > use
> > > > > > > > > > it for
> > > > > > > > > > > > any purpose, nor disclose its contents to any other
> > person.
> > > > > This
> > > > > > > > > email
> > > > > > > > > > does
> > > > > > > > > > > > not constitute the commencement of legal relations
> > between
> > > > > you
> > > > > > > and
> > > > > > > > > > Misys.
> > > > > > > > > > > > Please refer to the executed contract between you and
> > the
> > > > > > > relevant
> > > > > > > > > > member
> > > > > > > > > > > > of the Misys group for the identity of the
> contracting
> > > > party
> > > > > with
> > > > > > > > > > which you
> > > > > > > > > > > > are dealing.
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> >
>

[Andrey Kornev]

Raul,

While we're waiting for the specific proposal(s) from you, I'd like to point out that OSGi bundle is much more than just a jar or a packaging unit. First and foremost an OSGi bundle is also a class loader unlike the jars in non-OSGi environment. We're after the class loader hence the proposal to use the bundle symbolic name and bundle version for serialization of the *class loader*.

Have a safe trip and please, do get back to us at your earliest.

Regards
Andrey

> Date: Wed, 4 Nov 2015 10:56:32 +0000
> Subject: Re: OSGi migration may require a special marshaller
> From: [hidden email]
> To: [hidden email]
>
> Could we rewind a little bit please? What is the point of using the bundle
> symbolic name in the first place?
>
> In OSGi, it is bad practice to refer to bundles directly. A bundle is a
> packaging unit, that's all, but it should not have any further
> consideration.
>
> It is perfectly feasible that a given package is exported by bundle A in a
> container, and by bundle B in another. One should not make any assumptions
> in that regard. A simple example of this is an enterprise-wide deployment
> of Ignite, where multiple apps in different OSGi containers share cache
> entities. Why would one expect these entities to be hosted in the same
> bundle across all apps?
>
> Referring to a bundle in OSGi world is like referring to a JAR in a normal
> Java environment. One would not always expect to resolve class A.B.C from
> bundle xyz-1.2.3.jar, right? You would resolve from the classpath.
>
> There are other mechanisms to reach into the OSGi environment and resolve a
> class arbitrarily from another bundle. Some of which I've pointed out. But
> relying on a bundle symbolic name receives an -1 from me.
>
> Can't elaborate now, I'm travelling.
>
> Will get back to my office tomorrow and I'll dig deeper into possibilities.
>
> Regards,
> Raúl.
> On 4 Nov 2015 03:07, "Dmitriy Setrakyan" <[hidden email]> wrote:
>
> > Andrey, et al,
> >
> > Can you provide a way on how to get a required Bundle object based on, say,
> > bundle symbolic name and version?
> >
> > I have reached the end of the internet trying to find a way in OSGI to look
> > up a Bundle based on something other than an actual Class belonging to that
> > bundle, but no luck.
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <[hidden email]>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > I think your approach will work, but I let Romain respond.
> > >
> > > Also, in terms of the implementation, please keep in mind that the
> > > resolver must be called for each non-JDK and non-Ignite core class (it
> > > would probably make sense to eschew storing class loaders for such
> > classes
> > > in favor of compactness of the serialized representation -- see below).
> > > Also, it's worth keeping a cache of already resolved class loaders per
> > > marshaller invocation (this is probably the context that Romain has
> > > mentioned in his previous posting) to minimize the number of the resolver
> > > calls.
> > >
> > > In terms of the resolver's implementation, the simplest way to serialize
> > > the class loader would be by capturing two pieces of information (both
> > > strings): the bundle symbolic name and the bundle version. This approach
> > > however may result in bloating of the serialized representation: I'd
> > > roughly estimate the overhead per element to be at least 20-30 bytes (the
> > > length of the symbolic name string, plus the length of the version
> > string).
> > > There are way to reduce the overhead (like serializing the hash code of
> > the
> > > bundle id string rather than the string itself, and then come up with a
> > > clever way of resolving the hash collisions), but they all come at cost
> > of
> > > increased complexity...
> > >
> > > An alternative approach would be rely on the special bundle id which is
> > an
> > > integer and is generated by the OSGi container. But in this case, all
> > nodes
> > > must ensure that all the bundles have consistent ids (bundle A with id 42
> > > on node N1, has the same id on every other node) which is not easy --
> > while
> > > not entirely impossible -- to guarantee. As long as the nodes are
> > > homogeneous (have the same set of bundles deployed) the OSGi container is
> > > guaranteed to assign to the bundles the same ids.
> > >
> > > Thanks
> > > Andrey
> > >
> > > > From: [hidden email]
> > > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: [hidden email]
> > > >
> > > > Romain,
> > > >
> > > > In the upcoming release we will be deprecating the OptimizedMarshaller
> > > and
> > > > will be switching to a default internal marshaller (which is based on
> > the
> > > > new PortableMarshaller donated by GridGain).
> > > >
> > > > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > > > instead of byte arrays. This will be pretty close to passing the
> > stream,
> > > as
> > > > suggested by Andrey.
> > > >
> > > > Also, I still think that we should only resolve class loaders and not
> > the
> > > > class itself. The main reason is that Ignite will encode class names
> > into
> > > > an integer hash code and will store the integer->class-fqn mapping in
> > > > internal replicated cache. I doubt users will get more efficient than
> > an
> > > > integer (4 bytes) for a class name.
> > > >
> > > > On the receiving side, once we are able to get the right class loader,
> > we
> > > > can easily get the proper class by calling
> > > ClassLoader.findClass(class-fqn).
> > > >
> > > > Thoughts?
> > > >
> > > > D.
> > > >
> > > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <[hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Maybe a missing point. I think but I'm not sure that in the
> > > > > OptimizedMarshaller there is a caching of already serialized classes.
> > > Due
> > > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> > if
> > > a
> > > > > bundle is refreshed, it will produce a new BundleRevision and
> > > therefore a
> > > > > new classloader. And if you don't release the class from the previous
> > > > > BundleRevision then you endup with memory leak. So maybe the
> > Marshaller
> > > > > interface or somewhere should provide a way to free those classes /
> > > > > classloaders.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Romain
> > > > >
> > > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <[hidden email]
> > >
> > > a
> > > > > écrit :
> > > > >
> > > > > > Romain/Dmitriy,
> > > > > >
> > > > > > I prefer Romain's approach, but just curious, in the API you guys
> > are
> > > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > > With a
> > > > > > byte[], one would inevitably end up wrapping it into a byte stream
> > > class.
> > > > > > Also, the stream-based interface would be more in line with the
> > > > > Marshaller
> > > > > > API.
> > > > > >
> > > > > > Also for symmetry with the readClass() method, I suggest the
> > > writeClass()
> > > > > > take a Class<?> rather than an object.
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > > From: [hidden email]
> > > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: [hidden email]
> > > > > > >
> > > > > > > Hi Dmitriy,
> > > > > > > I think your solution is good. Maybe I will change it a little
> > > bit...
> > > > > :P
> > > > > > > I think you should delegate the Class resolution to the resolver.
> > > > > Because
> > > > > > > for example with your current solution the marshaller may before
> > > (or
> > > > > > after)
> > > > > > > store the fqn of the class (maybe only the first time it
> > encounter
> > > it)
> > > > > > but
> > > > > > > in order to save the classloader context resolution the
> > > implementation
> > > > > of
> > > > > > > the resolver may have to save the package name of the given
> > object
> > > and
> > > > > > some
> > > > > > > extra info therefore the content package name will be serialized
> > > two
> > > > > > times.
> > > > > > > Ok, it's not a big deal.
> > > > > > > But now if the resolver identify that it can reuse the same class
> > > > > loader
> > > > > > > for a couple of classes. It will be interesting for it to have a
> > > > > > > serialization context in order to save, let say, classloader/id
> > > mapping
> > > > > > in
> > > > > > > order to save the id instead of a more longer representation.
> > > > > > > So I propose something like that:
> > > > > > > *public interface ClassResolver {*
> > > > > > > *    // This method is invoked on the sender side to *
> > > > > > > *    // marshal the information about the class.*
> > > > > > > *    // where the context is a map style object that is reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > *    public byte[] writeClass(Object o, Context context) throws
> > > > > > > IgniteException;*
> > > > > > >
> > > > > > > *    // This method is invoked on the receiving side to*
> > > > > > > *    // retrieve the class based on provided information.*
> > > > > > > *    // where the context is a map style object that is reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > *    public Class<?> readClass(byte[], Context context) throws
> > > > > > > IgniteException;*
> > > > > > > *}*
> > > > > > >
> > > > > > > I think your proposal will solve our issue and maybe also open a
> > > door
> > > > > for
> > > > > > > the osgi development.
> > > > > > > Let me know what do you think about me proposal? :)
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Romain
> > > > > > >
> > > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > > [hidden email]>
> > > > > a
> > > > > > > écrit :
> > > > > > >
> > > > > > > > Romain,
> > > > > > > >
> > > > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> > > proposed
> > > > > > > > earlier? Will it solve your problem?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Raul,
> > > > > > > > >  - Do you plan to use the TCCL when marshalling in OSGi env?
> > > If yes
> > > > > > how?
> > > > > > > > >  - I like the point 2. Maybe for a graph of object that come
> > > from
> > > > > > > > different
> > > > > > > > > packages / bundles you may have to recursively capture the
> > > package
> > > > > > > > version
> > > > > > > > > of the individual element of your graph.
> > > > > > > > >  - For the point 3 I wonder if it will not over-complexify
> > the
> > > > > > solution.
> > > > > > > > As
> > > > > > > > > a developer you will have to think about it. And it is not
> > > obvious
> > > > > > in the
> > > > > > > > > code where things are serialized or not. You may use lambda
> > in
> > > your
> > > > > > > > > application code therefore the current package become what
> > you
> > > call
> > > > > > a DTO
> > > > > > > > > package. The current state of ignite does not enforce you to
> > > > > specify
> > > > > > it
> > > > > > > > for
> > > > > > > > > "classical" classloading application. If you introduce this
> > > extra
> > > > > > step
> > > > > > > > for
> > > > > > > > > OSGi ready application you will maybe discourage people to
> > use
> > > > > OSGi.
> > > > > > > > >
> > > > > > > > > To comeback to our solution. We start we a strong assumption:
> > > our
> > > > > > cluster
> > > > > > > > > is homogeneous in term of code so, of course, it simplify our
> > > life
> > > > > > :).
> > > > > > > > >
> > > > > > > > > BTW if you can introduce an extension point in the class
> > > resolution
> > > > > > > > > mechanism it can be interesting for end user in order to
> > allow
> > > them
> > > > > > to
> > > > > > > > > optimize it based on there specific use cases.
> > > > > > > > >
> > > > > > > > > Romain.
> > > > > > > > >
> > > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <
> > [hidden email]>
> > > a
> > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > Hi Andrey,
> > > > > > > > > >
> > > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > > >
> > > > > > > > > > I don't like the solution to incorporate the bundle
> > symbolic
> > > name
> > > > > > in
> > > > > > > > the
> > > > > > > > > > serialised form. Nothing guarantees that the classdef will
> > be
> > > > > > located
> > > > > > > > > under
> > > > > > > > > > the same bundle in both source and target machines. We also
> > > have
> > > > > to
> > > > > > > > take
> > > > > > > > > > into account that OSGi allows for multiple versions of the
> > > same
> > > > > > > > > > bundle/packages to co-exist in the same  container. So it
> > > becomes
> > > > > > more
> > > > > > > > > > complex.
> > > > > > > > > >
> > > > > > > > > > Using the TCCL should work when serialising, but it'll
> > > probably
> > > > > be
> > > > > > of
> > > > > > > > no
> > > > > > > > > > use when deserialising on the other end.
> > > > > > > > > >
> > > > > > > > > > I need to enhance Ignite to:
> > > > > > > > > >
> > > > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > > > 2. Incorporate the package version of the class in the
> > > serialised
> > > > > > form
> > > > > > > > if
> > > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in
> > all
> > > > > > bundles
> > > > > > > > > > through a custom OSGi manifest header or the like, as I
> > > explained
> > > > > > > > before.
> > > > > > > > > > Package version must be taken into account.
> > > > > > > > > >
> > > > > > > > > > What do you think?
> > > > > > > > > >
> > > > > > > > > > Raúl.
> > > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > > [hidden email]>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Raul,
> > > > > > > > > > >
> > > > > > > > > > > The fundamental hurdle we need to jump over to make
> > Ignite
> > > > > > > > OSGi-enabled
> > > > > > > > > > is
> > > > > > > > > > > the marshalling. More specifically the issue is with
> > > > > > deserialization
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > classes that are provided by the bundles other than the
> > > Ignite
> > > > > > bundle
> > > > > > > > > > > itself.
> > > > > > > > > > >
> > > > > > > > > > > When the Ignite transport layer receives a message it
> > > needs to
> > > > > > figure
> > > > > > > > > out
> > > > > > > > > > > how to deserialize the bytes and for that it needs to
> > know
> > > the
> > > > > > bundle
> > > > > > > > > > that
> > > > > > > > > > > provides the class to be deserailized. At this point TCCL
> > > is of
> > > > > > no
> > > > > > > > use.
> > > > > > > > > > To
> > > > > > > > > > > make things more complex, the class may contain other
> > > classes
> > > > > > that
> > > > > > > > come
> > > > > > > > > > > from other bundles, and so on recursively. This means
> > that
> > > each
> > > > > > > > object
> > > > > > > > > in
> > > > > > > > > > > the hierarchy must be serialized with its bundle name (or
> > > > > bundle
> > > > > > id),
> > > > > > > > > so
> > > > > > > > > > > that the deserializer will then be able to correctly
> > > resolve
> > > > > the
> > > > > > > > class
> > > > > > > > > > > while traversing the object hierarchy during
> > > deserialization.
> > > > > > > > > > >
> > > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking
> > the
> > > > > > ability to
> > > > > > > > > > plug
> > > > > > > > > > > in a custom class resolver. Romain's solution was to use
> > > Kryo
> > > > > > that
> > > > > > > > does
> > > > > > > > > > > provide a way to customize class resolution. It has
> > solved
> > > the
> > > > > > > > problem
> > > > > > > > > of
> > > > > > > > > > > capturing the bundle info and he was able to successfully
> > > run
> > > > > > Ignite
> > > > > > > > > as a
> > > > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > > > inclusion of
> > > > > > > > > the
> > > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > > > > complexity
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > code and incorrect use of Kryo's numerous serializers
> > > caused
> > > > > some
> > > > > > > > weird
> > > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> > > cache
> > > > > > entries
> > > > > > > > > due
> > > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > > class --
> > > > > > go
> > > > > > > > > > > figure!). Overall the Kryo-based solution is brittle and
> > > hard
> > > > > to
> > > > > > > > > > maintain.
> > > > > > > > > > >
> > > > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > > > > resolution.
> > > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> > > addition to
> > > > > > the
> > > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Andrey
> > > > > > > > > > >
> > > > > > > > > > > > From: [hidden email]
> > > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > > > Subject: Re: OSGi migration may require a special
> > > marshaller
> > > > > > > > > > > > To: [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Romain,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> > > appreciate
> > > > > > your
> > > > > > > > > > input.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm thinking about the situation you describe and I
> > > wonder if
> > > > > > > > you're
> > > > > > > > > > > > exporting Ignite as an OSGi service which is then
> > > consumed
> > > > > from
> > > > > > > > other
> > > > > > > > > > > > bundles. Under this situation, it would be quite easy
> > to
> > > > > > reproduce
> > > > > > > > > the
> > > > > > > > > > > > behaviour you describe if Ignite is not resolving
> > > classes via
> > > > > > the
> > > > > > > > > TCCL.
> > > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > > >
> > > > > > > > > > > > Off the top of my head, there are two alternatives to
> > > solve
> > > > > it:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> > > already
> > > > > > > > used) –
> > > > > > > > > > we
> > > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package
> > so
> > > that
> > > > > > > > bundles
> > > > > > > > > > > > containing DTOs can expose packages to Ignite's
> > > marshallers.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> > > Integration,
> > > > > > Big
> > > > > > > > Data
> > > > > > > > > > and
> > > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 .
> > We
> > > > > some
> > > > > > > > stuff
> > > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > > > it work in our osgi environment. The main issue for
> > us
> > > now
> > > > > > is the
> > > > > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > > or any other marshaller that works with object that
> > > come
> > > > > from
> > > > > > > > > outside
> > > > > > > > > > > your
> > > > > > > > > > > > > class space.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have try kryo that works. Kryo provide an
> > extension
> > > > > point
> > > > > > in
> > > > > > > > > order
> > > > > > > > > > > to
> > > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > > . With this extension we are able to solve the
> > problem
> > > of
> > > > > > > > external
> > > > > > > > > > > classes.
> > > > > > > > > > > > > The only issue with kryo is that some classes need a
> > > > > certain
> > > > > > care
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > serialization process and therefore a specialized
> > > > > serializer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So I would like to know from the community what do
> > > think of
> > > > > > > > > changing
> > > > > > > > > > > the
> > > > > > > > > > > > > way the optimized marshaller works or introducing the
> > > > > > support of
> > > > > > > > > yet
> > > > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Romain.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > > > "Misys" is the trade name of the Misys group of
> > > companies.
> > > > > > This
> > > > > > > > > email
> > > > > > > > > > > and
> > > > > > > > > > > > > any attachments have been scanned for known viruses
> > > using
> > > > > > > > multiple
> > > > > > > > > > > > > scanners. This email message is intended for the
> > named
> > > > > > recipient
> > > > > > > > > > only.
> > > > > > > > > > > It
> > > > > > > > > > > > > may be privileged and/or confidential. If you are not
> > > the
> > > > > > named
> > > > > > > > > > > recipient
> > > > > > > > > > > > > of this email please notify us immediately and do not
> > > copy
> > > > > > it or
> > > > > > > > > use
> > > > > > > > > > > it for
> > > > > > > > > > > > > any purpose, nor disclose its contents to any other
> > > person.
> > > > > > This
> > > > > > > > > > email
> > > > > > > > > > > does
> > > > > > > > > > > > > not constitute the commencement of legal relations
> > > between
> > > > > > you
> > > > > > > > and
> > > > > > > > > > > Misys.
> > > > > > > > > > > > > Please refer to the executed contract between you and
> > > the
> > > > > > > > relevant
> > > > > > > > > > > member
> > > > > > > > > > > > > of the Misys group for the identity of the
> > contracting
> > > > > party
> > > > > > with
> > > > > > > > > > > which you
> > > > > > > > > > > > > are dealing.
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >