Apache Ignite Developers - Legacy Mail Archive

Ignite modularisation: client/server separation

Classic

List

Threaded

11 messages Options

Raul Kripalani

Ignite modularisation: client/server separation

Hello guys,

Spoke to Dmitriy over the weekend about the modularisation of Ignite.

At Apache Camel we strive to make all our components (including the future
camel-ignite) OSGi compatible because have a significant user base
deploying Camel apps on Apache Karaf.

I see some groundwork before we can aspire to make OSGi components
communicate with Ignite:

* There seems to be no concept of a client. Client-side and server-side
coexist in ignite-core. There's no code separation, so a client wanting to
communicate with an Ignite topology will end up importing the server-side
implementation too.

* ... and the server-side implementation uses classloading constructs
(e.g. Zero Deployment) which may prove hard to engineer for compatibility
with OSGi. This re-enginering is a waste of time IMHO because there's no
interest in running Ignite servers on top of OSGi - only clients.

* The ignite-core JAR (6.7mb) is too heavy for a component that's
nothing but a client. Some average Java client API sizes for reference:
hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).

* Possible dependency leak. Currently not a problem as we lack 3rd party
deps in ignite-core. But if we introduce any, we'll impose our dep versions
on the client creating possible classpath hell. Or seen from another
perspective: I don't know the reason we have no 3rd party deps, and I'm
sure we've had to reinvent the wheel at some point... Perhaps we avoided
them because we knew we would be imposing them on the client? If that's the
case, separating server and client will bring lots of flexibility.

* This architecture/design entails a lack of separation of concerns,
IMO. While it's true that a client can start an embedded Ignite instance
(and for that it'll need the server-side code), not all clients will do. In
my opinion, ignite-java-client (hypothetical name) should discover
ignite-core (just the server-side impl) in the classpath if starting an
embedded Ignite is requested.

What's your opinion? Obviously such a refactoring is a large undertaking
and not for immediate action. If the community shares this vision, I would
start thinking about this for 2.0.

P.S.: With regards to Camel, a warning on the camel-ignite doc page for not
being OSGi-compliant will suffice.

Regards,

*Raúl Kripalani*
Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

Sergi

Re: Ignite modularisation: client/server separation

Raul,

As far as I understand your main concern is that Ignite is incompatible
with OSGi (other points seem to be more about your personal feelings about
"right engineering practices" than about real problems). I believe that we
can achieve this compatibility without such a big refactoring of
everything. But do we really need that if it is not a requirement for
Camel?

BTW client/server API of Ignite is intentionally unified to make it easier
to use (simply switch one flag in config instead of rewriting your code to
completely different API). In this case unification works better than
separation of concerns.

Sergi

2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:

> Hello guys,
>
> Spoke to Dmitriy over the weekend about the modularisation of Ignite.
>
> At Apache Camel we strive to make all our components (including the future
> camel-ignite) OSGi compatible because have a significant user base
> deploying Camel apps on Apache Karaf.
>
> I see some groundwork before we can aspire to make OSGi components
> communicate with Ignite:
>
> * There seems to be no concept of a client. Client-side and server-side
> coexist in ignite-core. There's no code separation, so a client wanting to
> communicate with an Ignite topology will end up importing the server-side
> implementation too.
>
> * ... and the server-side implementation uses classloading constructs
> (e.g. Zero Deployment) which may prove hard to engineer for compatibility
> with OSGi. This re-enginering is a waste of time IMHO because there's no
> interest in running Ignite servers on top of OSGi - only clients.
>
> * The ignite-core JAR (6.7mb) is too heavy for a component that's
> nothing but a client. Some average Java client API sizes for reference:
> hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
>
> * Possible dependency leak. Currently not a problem as we lack 3rd party
> deps in ignite-core. But if we introduce any, we'll impose our dep versions
> on the client creating possible classpath hell. Or seen from another
> perspective: I don't know the reason we have no 3rd party deps, and I'm
> sure we've had to reinvent the wheel at some point... Perhaps we avoided
> them because we knew we would be imposing them on the client? If that's the
> case, separating server and client will bring lots of flexibility.
>
> * This architecture/design entails a lack of separation of concerns,
> IMO. While it's true that a client can start an embedded Ignite instance
> (and for that it'll need the server-side code), not all clients will do. In
> my opinion, ignite-java-client (hypothetical name) should discover
> ignite-core (just the server-side impl) in the classpath if starting an
> embedded Ignite is requested.
>
> What's your opinion? Obviously such a refactoring is a large undertaking
> and not for immediate action. If the community shares this vision, I would
> start thinking about this for 2.0.
>
> P.S.: With regards to Camel, a warning on the camel-ignite doc page for not
> being OSGi-compliant will suffice.
>
> Regards,
>
> *Raúl Kripalani*
> Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
> Integration specialist
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>

Gianfranco Murador

Re: Ignite modularisation: client/server separation

Hello everyone,
I have a question about this topic :
the unification between client and server is also due to the fact that I
can do computations on the client nodes ?
For computation I mean something like this:

ClusterGroup clientGroup ignite.cluster().forClients ( ) ;
IgniteCompute clientCompute = ignite.compute ( clientGroup ) ;
// Execute computation on the client nodes .
clientCompute.broadcast ( ( ) - > System.out.println ( " Hello Client " ) )
;

Thanks, Regards, Gianfranco

2015-08-03 15:23 GMT+02:00 Sergi Vladykin <[hidden email]>:

> Raul,
>
> As far as I understand your main concern is that Ignite is incompatible
> with OSGi (other points seem to be more about your personal feelings about
> "right engineering practices" than about real problems). I believe that we
> can achieve this compatibility without such a big refactoring of
> everything. But do we really need that if it is not a requirement for
> Camel?
>
> BTW client/server API of Ignite is intentionally unified to make it easier
> to use (simply switch one flag in config instead of rewriting your code to
> completely different API). In this case unification works better than
> separation of concerns.
>
> Sergi
>
> 2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:
>
> > Hello guys,
> >
> > Spoke to Dmitriy over the weekend about the modularisation of Ignite.
> >
> > At Apache Camel we strive to make all our components (including the
> future
> > camel-ignite) OSGi compatible because have a significant user base
> > deploying Camel apps on Apache Karaf.
> >
> > I see some groundwork before we can aspire to make OSGi components
> > communicate with Ignite:
> >
> > * There seems to be no concept of a client. Client-side and
> server-side
> > coexist in ignite-core. There's no code separation, so a client wanting
> to
> > communicate with an Ignite topology will end up importing the server-side
> > implementation too.
> >
> > * ... and the server-side implementation uses classloading constructs
> > (e.g. Zero Deployment) which may prove hard to engineer for compatibility
> > with OSGi. This re-enginering is a waste of time IMHO because there's no
> > interest in running Ignite servers on top of OSGi - only clients.
> >
> > * The ignite-core JAR (6.7mb) is too heavy for a component that's
> > nothing but a client. Some average Java client API sizes for reference:
> > hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
> >
> > * Possible dependency leak. Currently not a problem as we lack 3rd
> party
> > deps in ignite-core. But if we introduce any, we'll impose our dep
> versions
> > on the client creating possible classpath hell. Or seen from another
> > perspective: I don't know the reason we have no 3rd party deps, and I'm
> > sure we've had to reinvent the wheel at some point... Perhaps we avoided
> > them because we knew we would be imposing them on the client? If that's
> the
> > case, separating server and client will bring lots of flexibility.
> >
> > * This architecture/design entails a lack of separation of concerns,
> > IMO. While it's true that a client can start an embedded Ignite instance
> > (and for that it'll need the server-side code), not all clients will do.
> In
> > my opinion, ignite-java-client (hypothetical name) should discover
> > ignite-core (just the server-side impl) in the classpath if starting an
> > embedded Ignite is requested.
> >
> > What's your opinion? Obviously such a refactoring is a large undertaking
> > and not for immediate action. If the community shares this vision, I
> would
> > start thinking about this for 2.0.
> >
> > P.S.: With regards to Camel, a warning on the camel-ignite doc page for
> not
> > being OSGi-compliant will suffice.
> >
> > Regards,
> >
> > *Raúl Kripalani*
> > Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
> > Integration specialist
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
>

Raul Kripalani

Re: Ignite modularisation: client/server separation

In reply to this post by Sergi

Hi Sergey,

That's great if you think that the OSGi enablement can be achieved without
separating client from server. My concern with OSGifying all of current's
ignite-core is that we would be incurring the cost of OSGi-enabling the
server side when, quite honestly, there's probably little interest in that.
Folks will be interested in interacting as clients with Ignite servers from
OSGi, not in running Ignite servers within OSGi containers.

Out of over 200 Camel components hosted at the ASF, there are only around
12 that are *not* OSGi-friendly. When we create a component that leverages
a library/framework that's not OSGi-compatible, we try to engage with the
relevant communities to enhance the library in that direction. Or if the
communities are not in a position to collaborate and the OSGi enablement
can be achieved without structural changes (sometimes it's just a matter of
adding the right manifest headers – but I get the impression Ignite's
server-side will be more complex), we create a wrapper in the ServiceMix
Bundles community and publish it to Maven Central.

The ~ 12 components which are not OSGi-friendly are young, mostly
introduced in Camel >= 2.15. So it's ok to create a camel-ignite component
without OSGi support for now, but we should be having a roadmap to not
frustrate our OSGi users (as I said, we have a significant base) and for
the component to be accepted into Camel without objections.

With regards to the separation itself, I still think a JAR of nearly 7mb is
too big a burden for all clients. This is not an opinion about "right
engineering practices", but my practical opinion as an Ignite user. I get
excited with the applications of Ignite in the IoT – for pure caching, one
can use the Memcached client, but the compute capability, service registry,
messaging, etc. make it really appealing in this context. And given the
resource scarcity inherent in IoT devices, I don't see why a mobile client
or an embedded device should spare 7mb for the library when it won't even
use 20% of its capability given that it'll never act as a server.

Regards,

*Raúl Kripalani*
Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Mon, Aug 3, 2015 at 2:23 PM, Sergi Vladykin <[hidden email]>
wrote:

Sergi

Re: Ignite modularisation: client/server separation

In reply to this post by Gianfranco Murador

Gianfranco,

Honestly, I don't like that we allowing to run computations on clients,
because anyways in most cases it is meaningless, but now we must keep this
for backward compatibility. And for example if we would decide to implement
separate client with the same API as suggests Raul, this will be an
obstacle.

Sergi

2015-08-03 16:48 GMT+03:00 Gianfranco Murador <[hidden email]>
:

> Hello everyone,
> I have a question about this topic :
> the unification between client and server is also due to the fact that I
> can do computations on the client nodes ?
> For computation I mean something like this:
>
> ClusterGroup clientGroup ignite.cluster().forClients ( ) ;
> IgniteCompute clientCompute = ignite.compute ( clientGroup ) ;
> // Execute computation on the client nodes .
> clientCompute.broadcast ( ( ) - > System.out.println ( " Hello Client " ) )
> ;
>
> Thanks, Regards, Gianfranco
>
>
> 2015-08-03 15:23 GMT+02:00 Sergi Vladykin <[hidden email]>:
>
> > Raul,
> >
> > As far as I understand your main concern is that Ignite is incompatible
> > with OSGi (other points seem to be more about your personal feelings
> about
> > "right engineering practices" than about real problems). I believe that
> we
> > can achieve this compatibility without such a big refactoring of
> > everything. But do we really need that if it is not a requirement for
> > Camel?
> >
> > BTW client/server API of Ignite is intentionally unified to make it
> easier
> > to use (simply switch one flag in config instead of rewriting your code
> to
> > completely different API). In this case unification works better than
> > separation of concerns.
> >
> > Sergi
> >
> > 2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:
> >
> > > Hello guys,
> > >
> > > Spoke to Dmitriy over the weekend about the modularisation of Ignite.
> > >
> > > At Apache Camel we strive to make all our components (including the
> > future
> > > camel-ignite) OSGi compatible because have a significant user base
> > > deploying Camel apps on Apache Karaf.
> > >
> > > I see some groundwork before we can aspire to make OSGi components
> > > communicate with Ignite:
> > >
> > > * There seems to be no concept of a client. Client-side and
> > server-side
> > > coexist in ignite-core. There's no code separation, so a client wanting
> > to
> > > communicate with an Ignite topology will end up importing the
> server-side
> > > implementation too.
> > >
> > > * ... and the server-side implementation uses classloading
> constructs
> > > (e.g. Zero Deployment) which may prove hard to engineer for
> compatibility
> > > with OSGi. This re-enginering is a waste of time IMHO because there's
> no
> > > interest in running Ignite servers on top of OSGi - only clients.
> > >
> > > * The ignite-core JAR (6.7mb) is too heavy for a component that's
> > > nothing but a client. Some average Java client API sizes for reference:
> > > hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
> > >
> > > * Possible dependency leak. Currently not a problem as we lack 3rd
> > party
> > > deps in ignite-core. But if we introduce any, we'll impose our dep
> > versions
> > > on the client creating possible classpath hell. Or seen from another
> > > perspective: I don't know the reason we have no 3rd party deps, and I'm
> > > sure we've had to reinvent the wheel at some point... Perhaps we
> avoided
> > > them because we knew we would be imposing them on the client? If that's
> > the
> > > case, separating server and client will bring lots of flexibility.
> > >
> > > * This architecture/design entails a lack of separation of concerns,
> > > IMO. While it's true that a client can start an embedded Ignite
> instance
> > > (and for that it'll need the server-side code), not all clients will
> do.
> > In
> > > my opinion, ignite-java-client (hypothetical name) should discover
> > > ignite-core (just the server-side impl) in the classpath if starting an
> > > embedded Ignite is requested.
> > >
> > > What's your opinion? Obviously such a refactoring is a large
> undertaking
> > > and not for immediate action. If the community shares this vision, I
> > would
> > > start thinking about this for 2.0.
> > >
> > > P.S.: With regards to Camel, a warning on the camel-ignite doc page for
> > not
> > > being OSGi-compliant will suffice.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
> > > Integration specialist
> > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> >
>

Sergi

Re: Ignite modularisation: client/server separation

Raul,

Again, for the sake of unification I don't see why client will be OSGi
compliant but server will not. It will only confuse users. Having said
that, if we are going to add OSGi support IMO we should try to do that for
both client and server. And I'm sure this still will be cheaper than
separating client and server.

As for jar size, I can burn a big server out of resources with a single
Java class less than 1kb in size =) So this is still a wrong metric to me.
If I was IoT stuff developer I would not trust any third party dependencies
at all until I know that they were *initially designed* for IoT and
restricted devices. And what is needed here is exactly separation of
concerns: tiny, as optimized as possible for each particular IoT device
client + powerful Ignite cluster bundled with application logic on the back
end.

Time ago at GridGain we were experimenting with iOS and Android clients,
but this did not actually pay off..

Sergi

2015-08-03 18:40 GMT+03:00 Sergi Vladykin <[hidden email]>:

> Gianfranco,
>
> Honestly, I don't like that we allowing to run computations on clients,
> because anyways in most cases it is meaningless, but now we must keep this
> for backward compatibility. And for example if we would decide to implement
> separate client with the same API as suggests Raul, this will be an
> obstacle.
>
> Sergi
>
> 2015-08-03 16:48 GMT+03:00 Gianfranco Murador <
> [hidden email]>:
>
>> Hello everyone,
>> I have a question about this topic :
>> the unification between client and server is also due to the fact that I
>> can do computations on the client nodes ?
>> For computation I mean something like this:
>>
>> ClusterGroup clientGroup ignite.cluster().forClients ( ) ;
>> IgniteCompute clientCompute = ignite.compute ( clientGroup ) ;
>> // Execute computation on the client nodes .
>> clientCompute.broadcast ( ( ) - > System.out.println ( " Hello Client " )
>> )
>> ;
>>
>> Thanks, Regards, Gianfranco
>>
>>
>> 2015-08-03 15:23 GMT+02:00 Sergi Vladykin <[hidden email]>:
>>
>> > Raul,
>> >
>> > As far as I understand your main concern is that Ignite is incompatible
>> > with OSGi (other points seem to be more about your personal feelings
>> about
>> > "right engineering practices" than about real problems). I believe that
>> we
>> > can achieve this compatibility without such a big refactoring of
>> > everything. But do we really need that if it is not a requirement for
>> > Camel?
>> >
>> > BTW client/server API of Ignite is intentionally unified to make it
>> easier
>> > to use (simply switch one flag in config instead of rewriting your code
>> to
>> > completely different API). In this case unification works better than
>> > separation of concerns.
>> >
>> > Sergi
>> >
>> > 2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:
>> >
>> > > Hello guys,
>> > >
>> > > Spoke to Dmitriy over the weekend about the modularisation of Ignite.
>> > >
>> > > At Apache Camel we strive to make all our components (including the
>> > future
>> > > camel-ignite) OSGi compatible because have a significant user base
>> > > deploying Camel apps on Apache Karaf.
>> > >
>> > > I see some groundwork before we can aspire to make OSGi components
>> > > communicate with Ignite:
>> > >
>> > > * There seems to be no concept of a client. Client-side and
>> > server-side
>> > > coexist in ignite-core. There's no code separation, so a client
>> wanting
>> > to
>> > > communicate with an Ignite topology will end up importing the
>> server-side
>> > > implementation too.
>> > >
>> > > * ... and the server-side implementation uses classloading
>> constructs
>> > > (e.g. Zero Deployment) which may prove hard to engineer for
>> compatibility
>> > > with OSGi. This re-enginering is a waste of time IMHO because there's
>> no
>> > > interest in running Ignite servers on top of OSGi - only clients.
>> > >
>> > > * The ignite-core JAR (6.7mb) is too heavy for a component that's
>> > > nothing but a client. Some average Java client API sizes for
>> reference:
>> > > hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
>> > >
>> > > * Possible dependency leak. Currently not a problem as we lack 3rd
>> > party
>> > > deps in ignite-core. But if we introduce any, we'll impose our dep
>> > versions
>> > > on the client creating possible classpath hell. Or seen from another
>> > > perspective: I don't know the reason we have no 3rd party deps, and
>> I'm
>> > > sure we've had to reinvent the wheel at some point... Perhaps we
>> avoided
>> > > them because we knew we would be imposing them on the client? If
>> that's
>> > the
>> > > case, separating server and client will bring lots of flexibility.
>> > >
>> > > * This architecture/design entails a lack of separation of
>> concerns,
>> > > IMO. While it's true that a client can start an embedded Ignite
>> instance
>> > > (and for that it'll need the server-side code), not all clients will
>> do.
>> > In
>> > > my opinion, ignite-java-client (hypothetical name) should discover
>> > > ignite-core (just the server-side impl) in the classpath if starting
>> an
>> > > embedded Ignite is requested.
>> > >
>> > > What's your opinion? Obviously such a refactoring is a large
>> undertaking
>> > > and not for immediate action. If the community shares this vision, I
>> > would
>> > > start thinking about this for 2.0.
>> > >
>> > > P.S.: With regards to Camel, a warning on the camel-ignite doc page
>> for
>> > not
>> > > being OSGi-compliant will suffice.
>> > >
>> > > Regards,
>> > >
>> > > *Raúl Kripalani*
>> > > Apache Camel PMC Member & Committer | Enterprise Architect, Open
>> Source
>> > > Integration specialist
>> > > http://about.me/raulkripalani |
>> http://www.linkedin.com/in/raulkripalani
>> > > http://blog.raulkr.net | twitter: @raulvk
>> > >
>> >
>>
>
>

Andrey Kornev

RE: Ignite modularisation: client/server separation

In reply to this post by Raul Kripalani

Hi Raul,

At my company we run Ignite in an OSGi container. If I remember correctly we only had to provide our own (OSGI-aware) implementation of the Marshaller SPI and add the bundle manifest.

As for the "client" bundle, wouldn't the Ignite REST API be sufficient for what you have in mind? It's quite easy to write a very "thin" client with no 3d party dependencies (except maybe the REST client) on top of the REST API.

Regards
Andrey

> From: [hidden email]
> Date: Mon, 3 Aug 2015 16:01:19 +0100
> Subject: Re: Ignite modularisation: client/server separation
> To: [hidden email]
>
> Hi Sergey,
>
> That's great if you think that the OSGi enablement can be achieved without
> separating client from server. My concern with OSGifying all of current's
> ignite-core is that we would be incurring the cost of OSGi-enabling the
> server side when, quite honestly, there's probably little interest in that.
> Folks will be interested in interacting as clients with Ignite servers from
> OSGi, not in running Ignite servers within OSGi containers.
>
> Out of over 200 Camel components hosted at the ASF, there are only around
> 12 that are *not* OSGi-friendly. When we create a component that leverages
> a library/framework that's not OSGi-compatible, we try to engage with the
> relevant communities to enhance the library in that direction. Or if the
> communities are not in a position to collaborate and the OSGi enablement
> can be achieved without structural changes (sometimes it's just a matter of
> adding the right manifest headers – but I get the impression Ignite's
> server-side will be more complex), we create a wrapper in the ServiceMix
> Bundles community and publish it to Maven Central.
>
> The ~ 12 components which are not OSGi-friendly are young, mostly
> introduced in Camel >= 2.15. So it's ok to create a camel-ignite component
> without OSGi support for now, but we should be having a roadmap to not
> frustrate our OSGi users (as I said, we have a significant base) and for
> the component to be accepted into Camel without objections.
>
> With regards to the separation itself, I still think a JAR of nearly 7mb is
> too big a burden for all clients. This is not an opinion about "right
> engineering practices", but my practical opinion as an Ignite user. I get
> excited with the applications of Ignite in the IoT – for pure caching, one
> can use the Memcached client, but the compute capability, service registry,
> messaging, etc. make it really appealing in this context. And given the
> resource scarcity inherent in IoT devices, I don't see why a mobile client
> or an embedded device should spare 7mb for the library when it won't even
> use 20% of its capability given that it'll never act as a server.
>
> Regards,
>
> *Raúl Kripalani*
> Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
> Integration specialist
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Mon, Aug 3, 2015 at 2:23 PM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > Raul,
> >
> > As far as I understand your main concern is that Ignite is incompatible
> > with OSGi (other points seem to be more about your personal feelings about
> > "right engineering practices" than about real problems). I believe that we
> > can achieve this compatibility without such a big refactoring of
> > everything. But do we really need that if it is not a requirement for
> > Camel?
> >
> > BTW client/server API of Ignite is intentionally unified to make it easier
> > to use (simply switch one flag in config instead of rewriting your code to
> > completely different API). In this case unification works better than
> > separation of concerns.
> >
> > Sergi
> >
> > 2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:
> >
> > > Hello guys,
> > >
> > > Spoke to Dmitriy over the weekend about the modularisation of Ignite.
> > >
> > > At Apache Camel we strive to make all our components (including the
> > future
> > > camel-ignite) OSGi compatible because have a significant user base
> > > deploying Camel apps on Apache Karaf.
> > >
> > > I see some groundwork before we can aspire to make OSGi components
> > > communicate with Ignite:
> > >
> > > * There seems to be no concept of a client. Client-side and
> > server-side
> > > coexist in ignite-core. There's no code separation, so a client wanting
> > to
> > > communicate with an Ignite topology will end up importing the server-side
> > > implementation too.
> > >
> > > * ... and the server-side implementation uses classloading constructs
> > > (e.g. Zero Deployment) which may prove hard to engineer for compatibility
> > > with OSGi. This re-enginering is a waste of time IMHO because there's no
> > > interest in running Ignite servers on top of OSGi - only clients.
> > >
> > > * The ignite-core JAR (6.7mb) is too heavy for a component that's
> > > nothing but a client. Some average Java client API sizes for reference:
> > > hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
> > >
> > > * Possible dependency leak. Currently not a problem as we lack 3rd
> > party
> > > deps in ignite-core. But if we introduce any, we'll impose our dep
> > versions
> > > on the client creating possible classpath hell. Or seen from another
> > > perspective: I don't know the reason we have no 3rd party deps, and I'm
> > > sure we've had to reinvent the wheel at some point... Perhaps we avoided
> > > them because we knew we would be imposing them on the client? If that's
> > the
> > > case, separating server and client will bring lots of flexibility.
> > >
> > > * This architecture/design entails a lack of separation of concerns,
> > > IMO. While it's true that a client can start an embedded Ignite instance
> > > (and for that it'll need the server-side code), not all clients will do.
> > In
> > > my opinion, ignite-java-client (hypothetical name) should discover
> > > ignite-core (just the server-side impl) in the classpath if starting an
> > > embedded Ignite is requested.
> > >
> > > What's your opinion? Obviously such a refactoring is a large undertaking
> > > and not for immediate action. If the community shares this vision, I
> > would
> > > start thinking about this for 2.0.
> > >
> > > P.S.: With regards to Camel, a warning on the camel-ignite doc page for
> > not
> > > being OSGi-compliant will suffice.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
> > > Integration specialist
> > > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> >

Raul Kripalani

Re: Ignite modularisation: client/server separation

In reply to this post by Sergi

Inline.

On Mon, Aug 3, 2015 at 5:14 PM, Sergi Vladykin <[hidden email]>
wrote:

> Raul,
>
> Again, for the sake of unification I don't see why client will be OSGi
> compliant but server will not. It will only confuse users.

Not really. Client and server-side don't need to be built around the same
technology, let alone written in the same language. There are countless
technologies whose server-side is not OSGi (ehcache, Hazelcast, Cassandra)
nor even Java (Redis, MongoDB), yet they have an OSGi-compatible Java
client/driver.

> Having said that, if we are going to add OSGi support IMO we should try to
> do that for
> both client and server. And I'm sure this still will be cheaper than
> separating client and server.
>

Yep, I can help with that. But to me, OSGi support and Ignite
modularisation are different yet related topics.

As for jar size, I can burn a big server out of resources with a single
> Java class less than 1kb in size =) So this is still a wrong metric to me.

That wasn't my point.

Many developers (at least the good ones!) will tell you that they don't
like blowing up the size of their WARs, JARs, EARs, etc. unnecessarily and
shipping out unused binary. And while this will always happen to some
extent (you can't really piece apart the JDK or other 3rd party libraries –
at least until Project Jigsaw is a reality ;-)), offering a lightweight
client is essential.

Imagine that being a client of Kafka meant dragging along the entire server
implementation, or ActiveMQ, or Cassandra, etc. Does that make sense?

Clearly there are some use cases in the world of Ignite that justify
shipping the server along, but not all use cases require the ability to
instantly turn a client into a server and back.

In a nutshell, my proposal is to *not* touch the current ignite-core, but
instead create a *lightweight ignite-java-client*, while keeping
ignite-core intact. The former would be used by Java clients who do not
intend on becoming servers; it'd be a simplified version of ignite-core
honouring the API but providing only the plumbing to communicate with the
servers. As such, with a swap of ignite-java-client with ignite-core (and
passing the parameters), that client would instantly be able to act as a
server.

Does that sound better and less traumatic?

> If I was IoT stuff developer I would not trust any third party dependencies
> at all until I know that they were *initially designed* for IoT and
> restricted devices. And what is needed here is exactly separation of
> concerns: tiny, as optimized as possible for each particular IoT device
> client + powerful Ignite cluster bundled with application logic on the back
> end.
>

That's an opinionated statement! I'm more the kind of guy that prefers to
offer options and let people choose what best adapts to them.

> Time ago at GridGain we were experimenting with iOS and Android clients,
> but this did not actually pay off..
>

You were closed-source back then. When you become Open Source, people start
using your stack in ways you hadn't imagined... ;-)

>
> Sergi
>
>
>
> 2015-08-03 18:40 GMT+03:00 Sergi Vladykin <[hidden email]>:
>
> > Gianfranco,
> >
> > Honestly, I don't like that we allowing to run computations on clients,
> > because anyways in most cases it is meaningless, but now we must keep
> this
> > for backward compatibility. And for example if we would decide to
> implement
> > separate client with the same API as suggests Raul, this will be an
> > obstacle.
> >
> > Sergi
> >
> > 2015-08-03 16:48 GMT+03:00 Gianfranco Murador <
> > [hidden email]>:
> >
> >> Hello everyone,
> >> I have a question about this topic :
> >> the unification between client and server is also due to the fact that I
> >> can do computations on the client nodes ?
> >> For computation I mean something like this:
> >>
> >> ClusterGroup clientGroup ignite.cluster().forClients ( ) ;
> >> IgniteCompute clientCompute = ignite.compute ( clientGroup ) ;
> >> // Execute computation on the client nodes .
> >> clientCompute.broadcast ( ( ) - > System.out.println ( " Hello Client "
> )
> >> )
> >> ;
> >>
> >> Thanks, Regards, Gianfranco
> >>
> >>
> >> 2015-08-03 15:23 GMT+02:00 Sergi Vladykin <[hidden email]>:
> >>
> >> > Raul,
> >> >
> >> > As far as I understand your main concern is that Ignite is
> incompatible
> >> > with OSGi (other points seem to be more about your personal feelings
> >> about
> >> > "right engineering practices" than about real problems). I believe
> that
> >> we
> >> > can achieve this compatibility without such a big refactoring of
> >> > everything. But do we really need that if it is not a requirement for
> >> > Camel?
> >> >
> >> > BTW client/server API of Ignite is intentionally unified to make it
> >> easier
> >> > to use (simply switch one flag in config instead of rewriting your
> code
> >> to
> >> > completely different API). In this case unification works better than
> >> > separation of concerns.
> >> >
> >> > Sergi
> >> >
> >> > 2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:
> >> >
> >> > > Hello guys,
> >> > >
> >> > > Spoke to Dmitriy over the weekend about the modularisation of
> Ignite.
> >> > >
> >> > > At Apache Camel we strive to make all our components (including the
> >> > future
> >> > > camel-ignite) OSGi compatible because have a significant user base
> >> > > deploying Camel apps on Apache Karaf.
> >> > >
> >> > > I see some groundwork before we can aspire to make OSGi components
> >> > > communicate with Ignite:
> >> > >
> >> > > * There seems to be no concept of a client. Client-side and
> >> > server-side
> >> > > coexist in ignite-core. There's no code separation, so a client
> >> wanting
> >> > to
> >> > > communicate with an Ignite topology will end up importing the
> >> server-side
> >> > > implementation too.
> >> > >
> >> > > * ... and the server-side implementation uses classloading
> >> constructs
> >> > > (e.g. Zero Deployment) which may prove hard to engineer for
> >> compatibility
> >> > > with OSGi. This re-enginering is a waste of time IMHO because
> there's
> >> no
> >> > > interest in running Ignite servers on top of OSGi - only clients.
> >> > >
> >> > > * The ignite-core JAR (6.7mb) is too heavy for a component that's
> >> > > nothing but a client. Some average Java client API sizes for
> >> reference:
> >> > > hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
> >> > >
> >> > > * Possible dependency leak. Currently not a problem as we lack
> 3rd
> >> > party
> >> > > deps in ignite-core. But if we introduce any, we'll impose our dep
> >> > versions
> >> > > on the client creating possible classpath hell. Or seen from another
> >> > > perspective: I don't know the reason we have no 3rd party deps, and
> >> I'm
> >> > > sure we've had to reinvent the wheel at some point... Perhaps we
> >> avoided
> >> > > them because we knew we would be imposing them on the client? If
> >> that's
> >> > the
> >> > > case, separating server and client will bring lots of flexibility.
> >> > >
> >> > > * This architecture/design entails a lack of separation of
> >> concerns,
> >> > > IMO. While it's true that a client can start an embedded Ignite
> >> instance
> >> > > (and for that it'll need the server-side code), not all clients will
> >> do.
> >> > In
> >> > > my opinion, ignite-java-client (hypothetical name) should discover
> >> > > ignite-core (just the server-side impl) in the classpath if starting
> >> an
> >> > > embedded Ignite is requested.
> >> > >
> >> > > What's your opinion? Obviously such a refactoring is a large
> >> undertaking
> >> > > and not for immediate action. If the community shares this vision, I
> >> > would
> >> > > start thinking about this for 2.0.
> >> > >
> >> > > P.S.: With regards to Camel, a warning on the camel-ignite doc page
> >> for
> >> > not
> >> > > being OSGi-compliant will suffice.
> >> > >
> >> > > Regards,
> >> > >
> >> > > *Raúl Kripalani*
> >> > > Apache Camel PMC Member & Committer | Enterprise Architect, Open
> >> Source
> >> > > Integration specialist
> >> > > http://about.me/raulkripalani |
> >> http://www.linkedin.com/in/raulkripalani
> >> > > http://blog.raulkr.net | twitter: @raulvk
> >> > >
> >> >
> >>
> >
> >
>

dsetrakyan

Re: Ignite modularisation: client/server separation

Raul,

Thanks for brining up good points.

In my view, most projects create client-only libraries simply because they
are not able to provide full API functionality on the client side. Ignite,
on the other hand, unlike other projects, can perform client-side caching,
transactions, queries, compute, services, etc. I personally view it as
advantage, not the other way around. Having said that, if the size of the
JAR file matters, our users can always use the Memcached or REST clients
provided by Ignite.

In my view, simply having a bigger size JAR file does not preclude Ignite
from becoming OSGI compatible, or from integrating with Camel. I also would
like to avoid postponing OSGI compatibility for the next year and becoming
1 of the 12 OSGI-non-compatible components in Camel.

My preference would be to wrap the current ignite-core into an OSGI bundle
and integrating with Camel cleanly. We can always wait and see if anyone
complains about the size of the JAR file or requests some enhancements
before going into deeper refactoring.

Thoughts?

D.

On Mon, Aug 3, 2015 at 10:54 AM, Raul Kripalani <[hidden email]> wrote:

> Inline.
>
> On Mon, Aug 3, 2015 at 5:14 PM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > Raul,
> >
> > Again, for the sake of unification I don't see why client will be OSGi
> > compliant but server will not. It will only confuse users.
>
>
> Not really. Client and server-side don't need to be built around the same
> technology, let alone written in the same language. There are countless
> technologies whose server-side is not OSGi (ehcache, Hazelcast, Cassandra)
> nor even Java (Redis, MongoDB), yet they have an OSGi-compatible Java
> client/driver.
>

>
> > Having said that, if we are going to add OSGi support IMO we should try
> to
> > do that for
> > both client and server. And I'm sure this still will be cheaper than
> > separating client and server.
> >
>
> Yep, I can help with that. But to me, OSGi support and Ignite
> modularisation are different yet related topics.
>
> As for jar size, I can burn a big server out of resources with a single
> > Java class less than 1kb in size =) So this is still a wrong metric to
> me.
>
>
> That wasn't my point.
>
> Many developers (at least the good ones!) will tell you that they don't
> like blowing up the size of their WARs, JARs, EARs, etc. unnecessarily and
> shipping out unused binary. And while this will always happen to some
> extent (you can't really piece apart the JDK or other 3rd party libraries –
> at least until Project Jigsaw is a reality ;-)), offering a lightweight
> client is essential.
>
> Imagine that being a client of Kafka meant dragging along the entire server
> implementation, or ActiveMQ, or Cassandra, etc. Does that make sense?
>
> Clearly there are some use cases in the world of Ignite that justify
> shipping the server along, but not all use cases require the ability to
> instantly turn a client into a server and back.
>
> In a nutshell, my proposal is to *not* touch the current ignite-core, but
> instead create a *lightweight ignite-java-client*, while keeping
> ignite-core intact. The former would be used by Java clients who do not
> intend on becoming servers; it'd be a simplified version of ignite-core
> honouring the API but providing only the plumbing to communicate with the
> servers. As such, with a swap of ignite-java-client with ignite-core (and
> passing the parameters), that client would instantly be able to act as a
> server.
>
> Does that sound better and less traumatic?
>
>
>
> > If I was IoT stuff developer I would not trust any third party
> dependencies
> > at all until I know that they were *initially designed* for IoT and
> > restricted devices. And what is needed here is exactly separation of
> > concerns: tiny, as optimized as possible for each particular IoT device
> > client + powerful Ignite cluster bundled with application logic on the
> back
> > end.
> >
>
> That's an opinionated statement! I'm more the kind of guy that prefers to
> offer options and let people choose what best adapts to them.
>
>
> > Time ago at GridGain we were experimenting with iOS and Android clients,
> > but this did not actually pay off..
> >
>
> You were closed-source back then. When you become Open Source, people start
> using your stack in ways you hadn't imagined... ;-)
>
>
> >
> > Sergi
> >
> >
> >
> > 2015-08-03 18:40 GMT+03:00 Sergi Vladykin <[hidden email]>:
> >
> > > Gianfranco,
> > >
> > > Honestly, I don't like that we allowing to run computations on clients,
> > > because anyways in most cases it is meaningless, but now we must keep
> > this
> > > for backward compatibility. And for example if we would decide to
> > implement
> > > separate client with the same API as suggests Raul, this will be an
> > > obstacle.
> > >
> > > Sergi
> > >
> > > 2015-08-03 16:48 GMT+03:00 Gianfranco Murador <
> > > [hidden email]>:
> > >
> > >> Hello everyone,
> > >> I have a question about this topic :
> > >> the unification between client and server is also due to the fact
> that I
> > >> can do computations on the client nodes ?
> > >> For computation I mean something like this:
> > >>
> > >> ClusterGroup clientGroup ignite.cluster().forClients ( ) ;
> > >> IgniteCompute clientCompute = ignite.compute ( clientGroup ) ;
> > >> // Execute computation on the client nodes .
> > >> clientCompute.broadcast ( ( ) - > System.out.println ( " Hello Client
> "
> > )
> > >> )
> > >> ;
> > >>
> > >> Thanks, Regards, Gianfranco
> > >>
> > >>
> > >> 2015-08-03 15:23 GMT+02:00 Sergi Vladykin <[hidden email]>:
> > >>
> > >> > Raul,
> > >> >
> > >> > As far as I understand your main concern is that Ignite is
> > incompatible
> > >> > with OSGi (other points seem to be more about your personal feelings
> > >> about
> > >> > "right engineering practices" than about real problems). I believe
> > that
> > >> we
> > >> > can achieve this compatibility without such a big refactoring of
> > >> > everything. But do we really need that if it is not a requirement
> for
> > >> > Camel?
> > >> >
> > >> > BTW client/server API of Ignite is intentionally unified to make it
> > >> easier
> > >> > to use (simply switch one flag in config instead of rewriting your
> > code
> > >> to
> > >> > completely different API). In this case unification works better
> than
> > >> > separation of concerns.
> > >> >
> > >> > Sergi
> > >> >
> > >> > 2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:
> > >> >
> > >> > > Hello guys,
> > >> > >
> > >> > > Spoke to Dmitriy over the weekend about the modularisation of
> > Ignite.
> > >> > >
> > >> > > At Apache Camel we strive to make all our components (including
> the
> > >> > future
> > >> > > camel-ignite) OSGi compatible because have a significant user base
> > >> > > deploying Camel apps on Apache Karaf.
> > >> > >
> > >> > > I see some groundwork before we can aspire to make OSGi components
> > >> > > communicate with Ignite:
> > >> > >
> > >> > > * There seems to be no concept of a client. Client-side and
> > >> > server-side
> > >> > > coexist in ignite-core. There's no code separation, so a client
> > >> wanting
> > >> > to
> > >> > > communicate with an Ignite topology will end up importing the
> > >> server-side
> > >> > > implementation too.
> > >> > >
> > >> > > * ... and the server-side implementation uses classloading
> > >> constructs
> > >> > > (e.g. Zero Deployment) which may prove hard to engineer for
> > >> compatibility
> > >> > > with OSGi. This re-enginering is a waste of time IMHO because
> > there's
> > >> no
> > >> > > interest in running Ignite servers on top of OSGi - only clients.
> > >> > >
> > >> > > * The ignite-core JAR (6.7mb) is too heavy for a component
> that's
> > >> > > nothing but a client. Some average Java client API sizes for
> > >> reference:
> > >> > > hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
> > >> > >
> > >> > > * Possible dependency leak. Currently not a problem as we lack
> > 3rd
> > >> > party
> > >> > > deps in ignite-core. But if we introduce any, we'll impose our dep
> > >> > versions
> > >> > > on the client creating possible classpath hell. Or seen from
> another
> > >> > > perspective: I don't know the reason we have no 3rd party deps,
> and
> > >> I'm
> > >> > > sure we've had to reinvent the wheel at some point... Perhaps we
> > >> avoided
> > >> > > them because we knew we would be imposing them on the client? If
> > >> that's
> > >> > the
> > >> > > case, separating server and client will bring lots of flexibility.
> > >> > >
> > >> > > * This architecture/design entails a lack of separation of
> > >> concerns,
> > >> > > IMO. While it's true that a client can start an embedded Ignite
> > >> instance
> > >> > > (and for that it'll need the server-side code), not all clients
> will
> > >> do.
> > >> > In
> > >> > > my opinion, ignite-java-client (hypothetical name) should discover
> > >> > > ignite-core (just the server-side impl) in the classpath if
> starting
> > >> an
> > >> > > embedded Ignite is requested.
> > >> > >
> > >> > > What's your opinion? Obviously such a refactoring is a large
> > >> undertaking
> > >> > > and not for immediate action. If the community shares this
> vision, I
> > >> > would
> > >> > > start thinking about this for 2.0.
> > >> > >
> > >> > > P.S.: With regards to Camel, a warning on the camel-ignite doc
> page
> > >> for
> > >> > not
> > >> > > being OSGi-compliant will suffice.
> > >> > >
> > >> > > Regards,
> > >> > >
> > >> > > *Raúl Kripalani*
> > >> > > Apache Camel PMC Member & Committer | Enterprise Architect, Open
> > >> Source
> > >> > > Integration specialist
> > >> > > http://about.me/raulkripalani |
> > >> http://www.linkedin.com/in/raulkripalani
> > >> > > http://blog.raulkr.net | twitter: @raulvk
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Sergi

Re: Ignite modularisation: client/server separation

I agree with Dmitriy and Andrey, REST client should suffice if you don't
like the full library, no reason to invent another client.

Andrey, could you share with us what will be enough to make marshaller OSGi
aware? Or may be you can publish source code of your marshaller? I think we
should implement this in our marshallers and make this work out of the box.

Sergi

2015-08-04 0:16 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Raul,
>
> Thanks for brining up good points.
>
> In my view, most projects create client-only libraries simply because they
> are not able to provide full API functionality on the client side. Ignite,
> on the other hand, unlike other projects, can perform client-side caching,
> transactions, queries, compute, services, etc. I personally view it as
> advantage, not the other way around. Having said that, if the size of the
> JAR file matters, our users can always use the Memcached or REST clients
> provided by Ignite.
>
> In my view, simply having a bigger size JAR file does not preclude Ignite
> from becoming OSGI compatible, or from integrating with Camel. I also would
> like to avoid postponing OSGI compatibility for the next year and becoming
> 1 of the 12 OSGI-non-compatible components in Camel.
>
> My preference would be to wrap the current ignite-core into an OSGI bundle
> and integrating with Camel cleanly. We can always wait and see if anyone
> complains about the size of the JAR file or requests some enhancements
> before going into deeper refactoring.
>
> Thoughts?
>
> D.
>
> On Mon, Aug 3, 2015 at 10:54 AM, Raul Kripalani <[hidden email]> wrote:
>
> > Inline.
> >
> > On Mon, Aug 3, 2015 at 5:14 PM, Sergi Vladykin <[hidden email]
> >
> > wrote:
> >
> > > Raul,
> > >
> > > Again, for the sake of unification I don't see why client will be OSGi
> > > compliant but server will not. It will only confuse users.
> >
> >
> > Not really. Client and server-side don't need to be built around the same
> > technology, let alone written in the same language. There are countless
> > technologies whose server-side is not OSGi (ehcache, Hazelcast,
> Cassandra)
> > nor even Java (Redis, MongoDB), yet they have an OSGi-compatible Java
> > client/driver.
> >
>
> >
> > > Having said that, if we are going to add OSGi support IMO we should try
> > to
> > > do that for
> > > both client and server. And I'm sure this still will be cheaper than
> > > separating client and server.
> > >
> >
> > Yep, I can help with that. But to me, OSGi support and Ignite
> > modularisation are different yet related topics.
> >
> > As for jar size, I can burn a big server out of resources with a single
> > > Java class less than 1kb in size =) So this is still a wrong metric to
> > me.
> >
> >
> > That wasn't my point.
> >
> > Many developers (at least the good ones!) will tell you that they don't
> > like blowing up the size of their WARs, JARs, EARs, etc. unnecessarily
> and
> > shipping out unused binary. And while this will always happen to some
> > extent (you can't really piece apart the JDK or other 3rd party
> libraries –
> > at least until Project Jigsaw is a reality ;-)), offering a lightweight
> > client is essential.
> >
> > Imagine that being a client of Kafka meant dragging along the entire
> server
> > implementation, or ActiveMQ, or Cassandra, etc. Does that make sense?
> >
> > Clearly there are some use cases in the world of Ignite that justify
> > shipping the server along, but not all use cases require the ability to
> > instantly turn a client into a server and back.
> >
> > In a nutshell, my proposal is to *not* touch the current ignite-core, but
> > instead create a *lightweight ignite-java-client*, while keeping
> > ignite-core intact. The former would be used by Java clients who do not
> > intend on becoming servers; it'd be a simplified version of ignite-core
> > honouring the API but providing only the plumbing to communicate with the
> > servers. As such, with a swap of ignite-java-client with ignite-core (and
> > passing the parameters), that client would instantly be able to act as a
> > server.
> >
> > Does that sound better and less traumatic?
> >
> >
> >
> > > If I was IoT stuff developer I would not trust any third party
> > dependencies
> > > at all until I know that they were *initially designed* for IoT and
> > > restricted devices. And what is needed here is exactly separation of
> > > concerns: tiny, as optimized as possible for each particular IoT device
> > > client + powerful Ignite cluster bundled with application logic on the
> > back
> > > end.
> > >
> >
> > That's an opinionated statement! I'm more the kind of guy that prefers to
> > offer options and let people choose what best adapts to them.
> >
> >
> > > Time ago at GridGain we were experimenting with iOS and Android
> clients,
> > > but this did not actually pay off..
> > >
> >
> > You were closed-source back then. When you become Open Source, people
> start
> > using your stack in ways you hadn't imagined... ;-)
> >
> >
> > >
> > > Sergi
> > >
> > >
> > >
> > > 2015-08-03 18:40 GMT+03:00 Sergi Vladykin <[hidden email]>:
> > >
> > > > Gianfranco,
> > > >
> > > > Honestly, I don't like that we allowing to run computations on
> clients,
> > > > because anyways in most cases it is meaningless, but now we must keep
> > > this
> > > > for backward compatibility. And for example if we would decide to
> > > implement
> > > > separate client with the same API as suggests Raul, this will be an
> > > > obstacle.
> > > >
> > > > Sergi
> > > >
> > > > 2015-08-03 16:48 GMT+03:00 Gianfranco Murador <
> > > > [hidden email]>:
> > > >
> > > >> Hello everyone,
> > > >> I have a question about this topic :
> > > >> the unification between client and server is also due to the fact
> > that I
> > > >> can do computations on the client nodes ?
> > > >> For computation I mean something like this:
> > > >>
> > > >> ClusterGroup clientGroup ignite.cluster().forClients ( ) ;
> > > >> IgniteCompute clientCompute = ignite.compute ( clientGroup ) ;
> > > >> // Execute computation on the client nodes .
> > > >> clientCompute.broadcast ( ( ) - > System.out.println ( " Hello
> Client
> > "
> > > )
> > > >> )
> > > >> ;
> > > >>
> > > >> Thanks, Regards, Gianfranco
> > > >>
> > > >>
> > > >> 2015-08-03 15:23 GMT+02:00 Sergi Vladykin <[hidden email]
> >:
> > > >>
> > > >> > Raul,
> > > >> >
> > > >> > As far as I understand your main concern is that Ignite is
> > > incompatible
> > > >> > with OSGi (other points seem to be more about your personal
> feelings
> > > >> about
> > > >> > "right engineering practices" than about real problems). I believe
> > > that
> > > >> we
> > > >> > can achieve this compatibility without such a big refactoring of
> > > >> > everything. But do we really need that if it is not a requirement
> > for
> > > >> > Camel?
> > > >> >
> > > >> > BTW client/server API of Ignite is intentionally unified to make
> it
> > > >> easier
> > > >> > to use (simply switch one flag in config instead of rewriting your
> > > code
> > > >> to
> > > >> > completely different API). In this case unification works better
> > than
> > > >> > separation of concerns.
> > > >> >
> > > >> > Sergi
> > > >> >
> > > >> > 2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:
> > > >> >
> > > >> > > Hello guys,
> > > >> > >
> > > >> > > Spoke to Dmitriy over the weekend about the modularisation of
> > > Ignite.
> > > >> > >
> > > >> > > At Apache Camel we strive to make all our components (including
> > the
> > > >> > future
> > > >> > > camel-ignite) OSGi compatible because have a significant user
> base
> > > >> > > deploying Camel apps on Apache Karaf.
> > > >> > >
> > > >> > > I see some groundwork before we can aspire to make OSGi
> components
> > > >> > > communicate with Ignite:
> > > >> > >
> > > >> > > * There seems to be no concept of a client. Client-side and
> > > >> > server-side
> > > >> > > coexist in ignite-core. There's no code separation, so a client
> > > >> wanting
> > > >> > to
> > > >> > > communicate with an Ignite topology will end up importing the
> > > >> server-side
> > > >> > > implementation too.
> > > >> > >
> > > >> > > * ... and the server-side implementation uses classloading
> > > >> constructs
> > > >> > > (e.g. Zero Deployment) which may prove hard to engineer for
> > > >> compatibility
> > > >> > > with OSGi. This re-enginering is a waste of time IMHO because
> > > there's
> > > >> no
> > > >> > > interest in running Ignite servers on top of OSGi - only
> clients.
> > > >> > >
> > > >> > > * The ignite-core JAR (6.7mb) is too heavy for a component
> > that's
> > > >> > > nothing but a client. Some average Java client API sizes for
> > > >> reference:
> > > >> > > hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
> > > >> > >
> > > >> > > * Possible dependency leak. Currently not a problem as we
> lack
> > > 3rd
> > > >> > party
> > > >> > > deps in ignite-core. But if we introduce any, we'll impose our
> dep
> > > >> > versions
> > > >> > > on the client creating possible classpath hell. Or seen from
> > another
> > > >> > > perspective: I don't know the reason we have no 3rd party deps,
> > and
> > > >> I'm
> > > >> > > sure we've had to reinvent the wheel at some point... Perhaps we
> > > >> avoided
> > > >> > > them because we knew we would be imposing them on the client? If
> > > >> that's
> > > >> > the
> > > >> > > case, separating server and client will bring lots of
> flexibility.
> > > >> > >
> > > >> > > * This architecture/design entails a lack of separation of
> > > >> concerns,
> > > >> > > IMO. While it's true that a client can start an embedded Ignite
> > > >> instance
> > > >> > > (and for that it'll need the server-side code), not all clients
> > will
> > > >> do.
> > > >> > In
> > > >> > > my opinion, ignite-java-client (hypothetical name) should
> discover
> > > >> > > ignite-core (just the server-side impl) in the classpath if
> > starting
> > > >> an
> > > >> > > embedded Ignite is requested.
> > > >> > >
> > > >> > > What's your opinion? Obviously such a refactoring is a large
> > > >> undertaking
> > > >> > > and not for immediate action. If the community shares this
> > vision, I
> > > >> > would
> > > >> > > start thinking about this for 2.0.
> > > >> > >
> > > >> > > P.S.: With regards to Camel, a warning on the camel-ignite doc
> > page
> > > >> for
> > > >> > not
> > > >> > > being OSGi-compliant will suffice.
> > > >> > >
> > > >> > > Regards,
> > > >> > >
> > > >> > > *Raúl Kripalani*
> > > >> > > Apache Camel PMC Member & Committer | Enterprise Architect, Open
> > > >> Source
> > > >> > > Integration specialist
> > > >> > > http://about.me/raulkripalani |
> > > >> http://www.linkedin.com/in/raulkripalani
> > > >> > > http://blog.raulkr.net | twitter: @raulvk
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Andrey Kornev

RE: Ignite modularisation: client/server separation

Sergi,

Our implementation is very specific to our internal requirements and is of little value as a general purpose marshaller implementation. I'll check with the developer who originally worked on it if he'd be interested in contributing to Ignite.

On a high level, in OSGi environment one must take special care of the class loading issues. Specifically, the FQN of a class is no longer sufficient to be able to create an instance of the class. The name of the bundle where the class is defined is also required. The name will be used during deserialization to first look up the bundle's class loader which is then used to load the class from its FQN.

We ended up using Kryo since it supports pluggable class resolver implementation (the thing that deals with the bundle names and class loaders). Our marshaller implementation delegates most of the work to Kryo and we only make sure that the bundle name is written/read to/from the stream. Also, we had to jump thru a few extra hoops making sure Ignite's internal classes get serialized/deserialized correctly.

Cheers
Andrey

> From: [hidden email]
> Date: Tue, 4 Aug 2015 10:53:20 +0300
> Subject: Re: Ignite modularisation: client/server separation
> To: [hidden email]
>
> I agree with Dmitriy and Andrey, REST client should suffice if you don't
> like the full library, no reason to invent another client.
>
> Andrey, could you share with us what will be enough to make marshaller OSGi
> aware? Or may be you can publish source code of your marshaller? I think we
> should implement this in our marshallers and make this work out of the box.
>
> Sergi
>
> 2015-08-04 0:16 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Raul,
> >
> > Thanks for brining up good points.
> >
> > In my view, most projects create client-only libraries simply because they
> > are not able to provide full API functionality on the client side. Ignite,
> > on the other hand, unlike other projects, can perform client-side caching,
> > transactions, queries, compute, services, etc. I personally view it as
> > advantage, not the other way around. Having said that, if the size of the
> > JAR file matters, our users can always use the Memcached or REST clients
> > provided by Ignite.
> >
> > In my view, simply having a bigger size JAR file does not preclude Ignite
> > from becoming OSGI compatible, or from integrating with Camel. I also would
> > like to avoid postponing OSGI compatibility for the next year and becoming
> > 1 of the 12 OSGI-non-compatible components in Camel.
> >
> > My preference would be to wrap the current ignite-core into an OSGI bundle
> > and integrating with Camel cleanly. We can always wait and see if anyone
> > complains about the size of the JAR file or requests some enhancements
> > before going into deeper refactoring.
> >
> > Thoughts?
> >
> > D.
> >
> > On Mon, Aug 3, 2015 at 10:54 AM, Raul Kripalani <[hidden email]> wrote:
> >
> > > Inline.
> > >
> > > On Mon, Aug 3, 2015 at 5:14 PM, Sergi Vladykin <[hidden email]
> > >
> > > wrote:
> > >
> > > > Raul,
> > > >
> > > > Again, for the sake of unification I don't see why client will be OSGi
> > > > compliant but server will not. It will only confuse users.
> > >
> > >
> > > Not really. Client and server-side don't need to be built around the same
> > > technology, let alone written in the same language. There are countless
> > > technologies whose server-side is not OSGi (ehcache, Hazelcast,
> > Cassandra)
> > > nor even Java (Redis, MongoDB), yet they have an OSGi-compatible Java
> > > client/driver.
> > >
> >
> > >
> > > > Having said that, if we are going to add OSGi support IMO we should try
> > > to
> > > > do that for
> > > > both client and server. And I'm sure this still will be cheaper than
> > > > separating client and server.
> > > >
> > >
> > > Yep, I can help with that. But to me, OSGi support and Ignite
> > > modularisation are different yet related topics.
> > >
> > > As for jar size, I can burn a big server out of resources with a single
> > > > Java class less than 1kb in size =) So this is still a wrong metric to
> > > me.
> > >
> > >
> > > That wasn't my point.
> > >
> > > Many developers (at least the good ones!) will tell you that they don't
> > > like blowing up the size of their WARs, JARs, EARs, etc. unnecessarily
> > and
> > > shipping out unused binary. And while this will always happen to some
> > > extent (you can't really piece apart the JDK or other 3rd party
> > libraries –
> > > at least until Project Jigsaw is a reality ;-)), offering a lightweight
> > > client is essential.
> > >
> > > Imagine that being a client of Kafka meant dragging along the entire
> > server
> > > implementation, or ActiveMQ, or Cassandra, etc. Does that make sense?
> > >
> > > Clearly there are some use cases in the world of Ignite that justify
> > > shipping the server along, but not all use cases require the ability to
> > > instantly turn a client into a server and back.
> > >
> > > In a nutshell, my proposal is to *not* touch the current ignite-core, but
> > > instead create a *lightweight ignite-java-client*, while keeping
> > > ignite-core intact. The former would be used by Java clients who do not
> > > intend on becoming servers; it'd be a simplified version of ignite-core
> > > honouring the API but providing only the plumbing to communicate with the
> > > servers. As such, with a swap of ignite-java-client with ignite-core (and
> > > passing the parameters), that client would instantly be able to act as a
> > > server.
> > >
> > > Does that sound better and less traumatic?
> > >
> > >
> > >
> > > > If I was IoT stuff developer I would not trust any third party
> > > dependencies
> > > > at all until I know that they were *initially designed* for IoT and
> > > > restricted devices. And what is needed here is exactly separation of
> > > > concerns: tiny, as optimized as possible for each particular IoT device
> > > > client + powerful Ignite cluster bundled with application logic on the
> > > back
> > > > end.
> > > >
> > >
> > > That's an opinionated statement! I'm more the kind of guy that prefers to
> > > offer options and let people choose what best adapts to them.
> > >
> > >
> > > > Time ago at GridGain we were experimenting with iOS and Android
> > clients,
> > > > but this did not actually pay off..
> > > >
> > >
> > > You were closed-source back then. When you become Open Source, people
> > start
> > > using your stack in ways you hadn't imagined... ;-)
> > >
> > >
> > > >
> > > > Sergi
> > > >
> > > >
> > > >
> > > > 2015-08-03 18:40 GMT+03:00 Sergi Vladykin <[hidden email]>:
> > > >
> > > > > Gianfranco,
> > > > >
> > > > > Honestly, I don't like that we allowing to run computations on
> > clients,
> > > > > because anyways in most cases it is meaningless, but now we must keep
> > > > this
> > > > > for backward compatibility. And for example if we would decide to
> > > > implement
> > > > > separate client with the same API as suggests Raul, this will be an
> > > > > obstacle.
> > > > >
> > > > > Sergi
> > > > >
> > > > > 2015-08-03 16:48 GMT+03:00 Gianfranco Murador <
> > > > > [hidden email]>:
> > > > >
> > > > >> Hello everyone,
> > > > >> I have a question about this topic :
> > > > >> the unification between client and server is also due to the fact
> > > that I
> > > > >> can do computations on the client nodes ?
> > > > >> For computation I mean something like this:
> > > > >>
> > > > >> ClusterGroup clientGroup ignite.cluster().forClients ( ) ;
> > > > >> IgniteCompute clientCompute = ignite.compute ( clientGroup ) ;
> > > > >> // Execute computation on the client nodes .
> > > > >> clientCompute.broadcast ( ( ) - > System.out.println ( " Hello
> > Client
> > > "
> > > > )
> > > > >> )
> > > > >> ;
> > > > >>
> > > > >> Thanks, Regards, Gianfranco
> > > > >>
> > > > >>
> > > > >> 2015-08-03 15:23 GMT+02:00 Sergi Vladykin <[hidden email]
> > >:
> > > > >>
> > > > >> > Raul,
> > > > >> >
> > > > >> > As far as I understand your main concern is that Ignite is
> > > > incompatible
> > > > >> > with OSGi (other points seem to be more about your personal
> > feelings
> > > > >> about
> > > > >> > "right engineering practices" than about real problems). I believe
> > > > that
> > > > >> we
> > > > >> > can achieve this compatibility without such a big refactoring of
> > > > >> > everything. But do we really need that if it is not a requirement
> > > for
> > > > >> > Camel?
> > > > >> >
> > > > >> > BTW client/server API of Ignite is intentionally unified to make
> > it
> > > > >> easier
> > > > >> > to use (simply switch one flag in config instead of rewriting your
> > > > code
> > > > >> to
> > > > >> > completely different API). In this case unification works better
> > > than
> > > > >> > separation of concerns.
> > > > >> >
> > > > >> > Sergi
> > > > >> >
> > > > >> > 2015-08-03 14:31 GMT+03:00 Raul Kripalani <[hidden email]>:
> > > > >> >
> > > > >> > > Hello guys,
> > > > >> > >
> > > > >> > > Spoke to Dmitriy over the weekend about the modularisation of
> > > > Ignite.
> > > > >> > >
> > > > >> > > At Apache Camel we strive to make all our components (including
> > > the
> > > > >> > future
> > > > >> > > camel-ignite) OSGi compatible because have a significant user
> > base
> > > > >> > > deploying Camel apps on Apache Karaf.
> > > > >> > >
> > > > >> > > I see some groundwork before we can aspire to make OSGi
> > components
> > > > >> > > communicate with Ignite:
> > > > >> > >
> > > > >> > > * There seems to be no concept of a client. Client-side and
> > > > >> > server-side
> > > > >> > > coexist in ignite-core. There's no code separation, so a client
> > > > >> wanting
> > > > >> > to
> > > > >> > > communicate with an Ignite topology will end up importing the
> > > > >> server-side
> > > > >> > > implementation too.
> > > > >> > >
> > > > >> > > * ... and the server-side implementation uses classloading
> > > > >> constructs
> > > > >> > > (e.g. Zero Deployment) which may prove hard to engineer for
> > > > >> compatibility
> > > > >> > > with OSGi. This re-enginering is a waste of time IMHO because
> > > > there's
> > > > >> no
> > > > >> > > interest in running Ignite servers on top of OSGi - only
> > clients.
> > > > >> > >
> > > > >> > > * The ignite-core JAR (6.7mb) is too heavy for a component
> > > that's
> > > > >> > > nothing but a client. Some average Java client API sizes for
> > > > >> reference:
> > > > >> > > hazelcast (400kb), activemq-client (1.2 mb), kafka (300kb).
> > > > >> > >
> > > > >> > > * Possible dependency leak. Currently not a problem as we
> > lack
> > > > 3rd
> > > > >> > party
> > > > >> > > deps in ignite-core. But if we introduce any, we'll impose our
> > dep
> > > > >> > versions
> > > > >> > > on the client creating possible classpath hell. Or seen from
> > > another
> > > > >> > > perspective: I don't know the reason we have no 3rd party deps,
> > > and
> > > > >> I'm
> > > > >> > > sure we've had to reinvent the wheel at some point... Perhaps we
> > > > >> avoided
> > > > >> > > them because we knew we would be imposing them on the client? If
> > > > >> that's
> > > > >> > the
> > > > >> > > case, separating server and client will bring lots of
> > flexibility.
> > > > >> > >
> > > > >> > > * This architecture/design entails a lack of separation of
> > > > >> concerns,
> > > > >> > > IMO. While it's true that a client can start an embedded Ignite
> > > > >> instance
> > > > >> > > (and for that it'll need the server-side code), not all clients
> > > will
> > > > >> do.
> > > > >> > In
> > > > >> > > my opinion, ignite-java-client (hypothetical name) should
> > discover
> > > > >> > > ignite-core (just the server-side impl) in the classpath if
> > > starting
> > > > >> an
> > > > >> > > embedded Ignite is requested.
> > > > >> > >
> > > > >> > > What's your opinion? Obviously such a refactoring is a large
> > > > >> undertaking
> > > > >> > > and not for immediate action. If the community shares this
> > > vision, I
> > > > >> > would
> > > > >> > > start thinking about this for 2.0.
> > > > >> > >
> > > > >> > > P.S.: With regards to Camel, a warning on the camel-ignite doc
> > > page
> > > > >> for
> > > > >> > not
> > > > >> > > being OSGi-compliant will suffice.
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > >
> > > > >> > > *Raúl Kripalani*
> > > > >> > > Apache Camel PMC Member & Committer | Enterprise Architect, Open
> > > > >> Source
> > > > >> > > Integration specialist
> > > > >> > > http://about.me/raulkripalani |
> > > > >> http://www.linkedin.com/in/raulkripalani
> > > > >> > > http://blog.raulkr.net | twitter: @raulvk
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >