Deploying code on ignite clusters

[Ognen Duzlevski]

I have a 5 machine ignite grid deployed on EC2 (still very much in testing
but would like to move to becoming more serious about using it in research
at least and eventually in production). In order to be able to cache
various Scala/java classes, I created a fat jar of my Scala app and put it
in the ignite/libs/ subdirectory. Then I start ignite on each node by
running the ignite.sh script.

When I add new classes, I recompile the app and create a new fat jar. How
does one deploy this new code so that the already running ignite becomes
aware of it? I am not much of a Java programmer (jumped straight to Scala)
- are there any ways Java allows for loading new classes/jars "on the fly"
into running JVMs? If so, does Ignite support them? If not, what do people
do to deploy new jars so they can become usable to an already running cache
grid?

Thanks!
Ognen

[dsetrakyan]

Ognen,

What kind of behavior do you need for the deployment? For example, if your
caches already have data with older classes, what should happen to that
data?

Of course, the easiest way to accomplish what you are doing is to copy the
new jars everywhere and restart the cluster. If this approach is
acceptable, I would just go with it.

D.

On Sun, Apr 26, 2015 at 7:29 PM, Ognen Duzlevski <[hidden email]>
wrote:

> I have a 5 machine ignite grid deployed on EC2 (still very much in testing
> but would like to move to becoming more serious about using it in research
> at least and eventually in production). In order to be able to cache
> various Scala/java classes, I created a fat jar of my Scala app and put it
> in the ignite/libs/ subdirectory. Then I start ignite on each node by
> running the ignite.sh script.
>
> When I add new classes, I recompile the app and create a new fat jar. How
> does one deploy this new code so that the already running ignite becomes
> aware of it? I am not much of a Java programmer (jumped straight to Scala)
> - are there any ways Java allows for loading new classes/jars "on the fly"
> into running JVMs? If so, does Ignite support them? If not, what do people
> do to deploy new jars so they can become usable to an already running cache
> grid?
>
> Thanks!
> Ognen
>

[Ognen Duzlevski]

Dmitriy,

I suppose I could do that (restart the cluster) but I suppose in that case
I have to have a strategy to dump the cache contents somewhere and
reconstruct it after startup. It's not always about changes to existing
classes being cached, it is also about adding classes to cache - in this
scenario I decide to cache more data and for that too I would have to
restart the cluster after copying the jars around.

What does "zero deployment" actually mean (I saw this on the GridGain
pages)? Does it also apply to Ignite?

Thanks!


On Mon, Apr 27, 2015 at 10:57 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Ognen,
>
> What kind of behavior do you need for the deployment? For example, if your
> caches already have data with older classes, what should happen to that
> data?
>
> Of course, the easiest way to accomplish what you are doing is to copy the
> new jars everywhere and restart the cluster. If this approach is
> acceptable, I would just go with it.
>
> D.
>
> On Sun, Apr 26, 2015 at 7:29 PM, Ognen Duzlevski <
> [hidden email]>
> wrote:
>
> > I have a 5 machine ignite grid deployed on EC2 (still very much in
> testing
> > but would like to move to becoming more serious about using it in
> research
> > at least and eventually in production). In order to be able to cache
> > various Scala/java classes, I created a fat jar of my Scala app and put
> it
> > in the ignite/libs/ subdirectory. Then I start ignite on each node by
> > running the ignite.sh script.
> >
> > When I add new classes, I recompile the app and create a new fat jar. How
> > does one deploy this new code so that the already running ignite becomes
> > aware of it? I am not much of a Java programmer (jumped straight to
> Scala)
> > - are there any ways Java allows for loading new classes/jars "on the
> fly"
> > into running JVMs? If so, does Ignite support them? If not, what do
> people
> > do to deploy new jars so they can become usable to an already running
> cache
> > grid?
> >
> > Thanks!
> > Ognen
> >
>

[Konstantin Boudnik-2]

I think Ognen has raised a very valid concern in [1]: what to do when you have
a huge cluster? Esp. on multi-tenant systems you can not just bounce the whole
thing on every application's DO classes change. Is it possible to do rolling
restart of the cluster nodes, where some of the nodes will still be on the old
version of the classes and some on the new one?

Sorry for using the nabble link, but
mail-archives.apache.org/mod_mbox/ignite-dev takes forever to get updated.

Cos

[1] http://apache-ignite-developers.2346864.n4.nabble.com/New-contributor-tp114p123.html

On Mon, Apr 27, 2015 at 08:57AM, Dmitriy Setrakyan wrote:

> Ognen,
>
> What kind of behavior do you need for the deployment? For example, if your
> caches already have data with older classes, what should happen to that
> data?
>
> Of course, the easiest way to accomplish what you are doing is to copy the
> new jars everywhere and restart the cluster. If this approach is
> acceptable, I would just go with it.
>
> D.
>
> On Sun, Apr 26, 2015 at 7:29 PM, Ognen Duzlevski <[hidden email]>
> wrote:
>
> > I have a 5 machine ignite grid deployed on EC2 (still very much in testing
> > but would like to move to becoming more serious about using it in research
> > at least and eventually in production). In order to be able to cache
> > various Scala/java classes, I created a fat jar of my Scala app and put it
> > in the ignite/libs/ subdirectory. Then I start ignite on each node by
> > running the ignite.sh script.
> >
> > When I add new classes, I recompile the app and create a new fat jar. How
> > does one deploy this new code so that the already running ignite becomes
> > aware of it? I am not much of a Java programmer (jumped straight to Scala)
> > - are there any ways Java allows for loading new classes/jars "on the fly"
> > into running JVMs? If so, does Ignite support them? If not, what do people
> > do to deploy new jars so they can become usable to an already running cache
> > grid?
> >
> > Thanks!
> > Ognen
> >

[Ognen Duzlevski]

Cos, thanks. If anyone has any suggestions, I would be glad to hear them :-)

On Tue, Apr 28, 2015 at 12:59 AM, Konstantin Boudnik <[hidden email]> wrote:

> I think Ognen has raised a very valid concern in [1]: what to do when you
> have
> a huge cluster? Esp. on multi-tenant systems you can not just bounce the
> whole
> thing on every application's DO classes change. Is it possible to do
> rolling
> restart of the cluster nodes, where some of the nodes will still be on the
> old
> version of the classes and some on the new one?
>
> Sorry for using the nabble link, but
> mail-archives.apache.org/mod_mbox/ignite-dev takes forever to get updated.
>
> Cos
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/New-contributor-tp114p123.html
>
> On Mon, Apr 27, 2015 at 08:57AM, Dmitriy Setrakyan wrote:
> > Ognen,
> >
> > What kind of behavior do you need for the deployment? For example, if
> your
> > caches already have data with older classes, what should happen to that
> > data?
> >
> > Of course, the easiest way to accomplish what you are doing is to copy
> the
> > new jars everywhere and restart the cluster. If this approach is
> > acceptable, I would just go with it.
> >
> > D.
> >
> > On Sun, Apr 26, 2015 at 7:29 PM, Ognen Duzlevski <
> [hidden email]>
> > wrote:
> >
> > > I have a 5 machine ignite grid deployed on EC2 (still very much in
> testing
> > > but would like to move to becoming more serious about using it in
> research
> > > at least and eventually in production). In order to be able to cache
> > > various Scala/java classes, I created a fat jar of my Scala app and
> put it
> > > in the ignite/libs/ subdirectory. Then I start ignite on each node by
> > > running the ignite.sh script.
> > >
> > > When I add new classes, I recompile the app and create a new fat jar.
> How
> > > does one deploy this new code so that the already running ignite
> becomes
> > > aware of it? I am not much of a Java programmer (jumped straight to
> Scala)
> > > - are there any ways Java allows for loading new classes/jars "on the
> fly"
> > > into running JVMs? If so, does Ignite support them? If not, what do
> people
> > > do to deploy new jars so they can become usable to an already running
> cache
> > > grid?
> > >
> > > Thanks!
> > > Ognen
> > >
>

[dsetrakyan]

I agree, the concern is valid.

Ognen, Cos, what do you think should be the right behavior? Should we allow
different versions of the objects live in the same cache? GridGain
enterprise edition, by the way, allows it with Portable Objects.

If you would like to clear the older object versions and deploy new ones
automatically, Ignite supports it already. Just enable peerClassLoading on
Ignite configuration.

D.

On Tue, Apr 28, 2015 at 8:43 AM, Ognen Duzlevski <[hidden email]>
wrote:

> Cos, thanks. If anyone has any suggestions, I would be glad to hear them
> :-)
>
> On Tue, Apr 28, 2015 at 12:59 AM, Konstantin Boudnik <[hidden email]>
> wrote:
>
> > I think Ognen has raised a very valid concern in [1]: what to do when you
> > have
> > a huge cluster? Esp. on multi-tenant systems you can not just bounce the
> > whole
> > thing on every application's DO classes change. Is it possible to do
> > rolling
> > restart of the cluster nodes, where some of the nodes will still be on
> the
> > old
> > version of the classes and some on the new one?
> >
> > Sorry for using the nabble link, but
> > mail-archives.apache.org/mod_mbox/ignite-dev takes forever to get
> updated.
> >
> > Cos
> >
> > [1]
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/New-contributor-tp114p123.html
> >
> > On Mon, Apr 27, 2015 at 08:57AM, Dmitriy Setrakyan wrote:
> > > Ognen,
> > >
> > > What kind of behavior do you need for the deployment? For example, if
> > your
> > > caches already have data with older classes, what should happen to that
> > > data?
> > >
> > > Of course, the easiest way to accomplish what you are doing is to copy
> > the
> > > new jars everywhere and restart the cluster. If this approach is
> > > acceptable, I would just go with it.
> > >
> > > D.
> > >
> > > On Sun, Apr 26, 2015 at 7:29 PM, Ognen Duzlevski <
> > [hidden email]>
> > > wrote:
> > >
> > > > I have a 5 machine ignite grid deployed on EC2 (still very much in
> > testing
> > > > but would like to move to becoming more serious about using it in
> > research
> > > > at least and eventually in production). In order to be able to cache
> > > > various Scala/java classes, I created a fat jar of my Scala app and
> > put it
> > > > in the ignite/libs/ subdirectory. Then I start ignite on each node by
> > > > running the ignite.sh script.
> > > >
> > > > When I add new classes, I recompile the app and create a new fat jar.
> > How
> > > > does one deploy this new code so that the already running ignite
> > becomes
> > > > aware of it? I am not much of a Java programmer (jumped straight to
> > Scala)
> > > > - are there any ways Java allows for loading new classes/jars "on the
> > fly"
> > > > into running JVMs? If so, does Ignite support them? If not, what do
> > people
> > > > do to deploy new jars so they can become usable to an already running
> > cache
> > > > grid?
> > > >
> > > > Thanks!
> > > > Ognen
> > > >
> >
>

[Konstantin Boudnik-2]

On Tue, Apr 28, 2015 at 06:02PM, Dmitriy Setrakyan wrote:
> I agree, the concern is valid.
>
> Ognen, Cos, what do you think should be the right behavior? Should we allow
> different versions of the objects live in the same cache? GridGain

Being able to have different versions of the objects might be a bliss (I'm not
sure if that would fit Ognen usecase). But it might come with the funny
situations such as when an older revision never got purged off a cache,
causing potential data-conflicts, etc. To me peerClassLoading seems to fine,
but users need to be aware about its effects.

Just thinking out loud, really.
  Cos

> enterprise edition, by the way, allows it with Portable Objects.
>
> If you would like to clear the older object versions and deploy new ones
> automatically, Ignite supports it already. Just enable peerClassLoading on
> Ignite configuration.
>
> D.
>
> On Tue, Apr 28, 2015 at 8:43 AM, Ognen Duzlevski <[hidden email]>
> wrote:
>
> > Cos, thanks. If anyone has any suggestions, I would be glad to hear them
> > :-)
> >
> > On Tue, Apr 28, 2015 at 12:59 AM, Konstantin Boudnik <[hidden email]>
> > wrote:
> >
> > > I think Ognen has raised a very valid concern in [1]: what to do when you
> > > have
> > > a huge cluster? Esp. on multi-tenant systems you can not just bounce the
> > > whole
> > > thing on every application's DO classes change. Is it possible to do
> > > rolling
> > > restart of the cluster nodes, where some of the nodes will still be on
> > the
> > > old
> > > version of the classes and some on the new one?
> > >
> > > Sorry for using the nabble link, but
> > > mail-archives.apache.org/mod_mbox/ignite-dev takes forever to get
> > updated.
> > >
> > > Cos
> > >
> > > [1]
> > >
> > http://apache-ignite-developers.2346864.n4.nabble.com/New-contributor-tp114p123.html
> > >
> > > On Mon, Apr 27, 2015 at 08:57AM, Dmitriy Setrakyan wrote:
> > > > Ognen,
> > > >
> > > > What kind of behavior do you need for the deployment? For example, if
> > > your
> > > > caches already have data with older classes, what should happen to that
> > > > data?
> > > >
> > > > Of course, the easiest way to accomplish what you are doing is to copy
> > > the
> > > > new jars everywhere and restart the cluster. If this approach is
> > > > acceptable, I would just go with it.
> > > >
> > > > D.
> > > >
> > > > On Sun, Apr 26, 2015 at 7:29 PM, Ognen Duzlevski <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > I have a 5 machine ignite grid deployed on EC2 (still very much in
> > > testing
> > > > > but would like to move to becoming more serious about using it in
> > > research
> > > > > at least and eventually in production). In order to be able to cache
> > > > > various Scala/java classes, I created a fat jar of my Scala app and
> > > put it
> > > > > in the ignite/libs/ subdirectory. Then I start ignite on each node by
> > > > > running the ignite.sh script.
> > > > >
> > > > > When I add new classes, I recompile the app and create a new fat jar.
> > > How
> > > > > does one deploy this new code so that the already running ignite
> > > becomes
> > > > > aware of it? I am not much of a Java programmer (jumped straight to
> > > Scala)
> > > > > - are there any ways Java allows for loading new classes/jars "on the
> > > fly"
> > > > > into running JVMs? If so, does Ignite support them? If not, what do
> > > people
> > > > > do to deploy new jars so they can become usable to an already running
> > > cache
> > > > > grid?
> > > > >
> > > > > Thanks!
> > > > > Ognen
> > > > >
> > >
> >

[Ognen Duzlevski]

Cos,

I tend to agree with you. The more I think about it, having various
versions of classes in the same cache might prove to be a problem. With
that said, ignite seems to want a (fat) jar placed in libs/ on startup, in
order to become aware of all the external classes that could be cached at
some point down the road. If you produced a new version of your app, one
that adds more classes that could be cached - what is the current way to
make ignite aware of this new (fat) jar or set of classes? One obvious way
is to place the new jar into the libs/ directory on every node and restart
the ignite cluster. However, this means that all the "live" cache needs to
be abandoned (or at least persisted and re-loaded) - this seems expensive.
What else is there to address this use case scenario?

Thanks!

On Wed, Apr 29, 2015 at 1:06 AM, Konstantin Boudnik <[hidden email]> wrote:

> On Tue, Apr 28, 2015 at 06:02PM, Dmitriy Setrakyan wrote:
> > I agree, the concern is valid.
> >
> > Ognen, Cos, what do you think should be the right behavior? Should we
> allow
> > different versions of the objects live in the same cache? GridGain
>
> Being able to have different versions of the objects might be a bliss (I'm
> not
> sure if that would fit Ognen usecase). But it might come with the funny
> situations such as when an older revision never got purged off a cache,
> causing potential data-conflicts, etc. To me peerClassLoading seems to
> fine,
> but users need to be aware about its effects.
>
> Just thinking out loud, really.
>   Cos
>
> > enterprise edition, by the way, allows it with Portable Objects.
> >
> > If you would like to clear the older object versions and deploy new ones
> > automatically, Ignite supports it already. Just enable peerClassLoading
> on
> > Ignite configuration.
> >
> > D.
> >
> > On Tue, Apr 28, 2015 at 8:43 AM, Ognen Duzlevski <
> [hidden email]>
> > wrote:
> >
> > > Cos, thanks. If anyone has any suggestions, I would be glad to hear
> them
> > > :-)
> > >
> > > On Tue, Apr 28, 2015 at 12:59 AM, Konstantin Boudnik <[hidden email]>
> > > wrote:
> > >
> > > > I think Ognen has raised a very valid concern in [1]: what to do
> when you
> > > > have
> > > > a huge cluster? Esp. on multi-tenant systems you can not just bounce
> the
> > > > whole
> > > > thing on every application's DO classes change. Is it possible to do
> > > > rolling
> > > > restart of the cluster nodes, where some of the nodes will still be
> on
> > > the
> > > > old
> > > > version of the classes and some on the new one?
> > > >
> > > > Sorry for using the nabble link, but
> > > > mail-archives.apache.org/mod_mbox/ignite-dev takes forever to get
> > > updated.
> > > >
> > > > Cos
> > > >
> > > > [1]
> > > >
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/New-contributor-tp114p123.html
> > > >
> > > > On Mon, Apr 27, 2015 at 08:57AM, Dmitriy Setrakyan wrote:
> > > > > Ognen,
> > > > >
> > > > > What kind of behavior do you need for the deployment? For example,
> if
> > > > your
> > > > > caches already have data with older classes, what should happen to
> that
> > > > > data?
> > > > >
> > > > > Of course, the easiest way to accomplish what you are doing is to
> copy
> > > > the
> > > > > new jars everywhere and restart the cluster. If this approach is
> > > > > acceptable, I would just go with it.
> > > > >
> > > > > D.
> > > > >
> > > > > On Sun, Apr 26, 2015 at 7:29 PM, Ognen Duzlevski <
> > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > I have a 5 machine ignite grid deployed on EC2 (still very much
> in
> > > > testing
> > > > > > but would like to move to becoming more serious about using it in
> > > > research
> > > > > > at least and eventually in production). In order to be able to
> cache
> > > > > > various Scala/java classes, I created a fat jar of my Scala app
> and
> > > > put it
> > > > > > in the ignite/libs/ subdirectory. Then I start ignite on each
> node by
> > > > > > running the ignite.sh script.
> > > > > >
> > > > > > When I add new classes, I recompile the app and create a new fat
> jar.
> > > > How
> > > > > > does one deploy this new code so that the already running ignite
> > > > becomes
> > > > > > aware of it? I am not much of a Java programmer (jumped straight
> to
> > > > Scala)
> > > > > > - are there any ways Java allows for loading new classes/jars
> "on the
> > > > fly"
> > > > > > into running JVMs? If so, does Ignite support them? If not, what
> do
> > > > people
> > > > > > do to deploy new jars so they can become usable to an already
> running
> > > > cache
> > > > > > grid?
> > > > > >
> > > > > > Thanks!
> > > > > > Ognen
> > > > > >
> > > >
> > >
>

[Konstantin Boudnik-2]

I only can thing of rolling restart, eg where nodes are getting bounced one by
one. But that too might present an issue because different nodes will have
different object versions.

I'll resort to the expertise of the guys who've developed that - I don't
possess enough knowledge (yet?!) to weigh in properly. Sorry

Cos

On Wed, Apr 29, 2015 at 07:08AM, Ognen Duzlevski wrote:

> Cos,
>
> I tend to agree with you. The more I think about it, having various
> versions of classes in the same cache might prove to be a problem. With
> that said, ignite seems to want a (fat) jar placed in libs/ on startup, in
> order to become aware of all the external classes that could be cached at
> some point down the road. If you produced a new version of your app, one
> that adds more classes that could be cached - what is the current way to
> make ignite aware of this new (fat) jar or set of classes? One obvious way
> is to place the new jar into the libs/ directory on every node and restart
> the ignite cluster. However, this means that all the "live" cache needs to
> be abandoned (or at least persisted and re-loaded) - this seems expensive.
> What else is there to address this use case scenario?
>
> Thanks!
>
> On Wed, Apr 29, 2015 at 1:06 AM, Konstantin Boudnik <[hidden email]> wrote:
>
> > On Tue, Apr 28, 2015 at 06:02PM, Dmitriy Setrakyan wrote:
> > > I agree, the concern is valid.
> > >
> > > Ognen, Cos, what do you think should be the right behavior? Should we
> > allow
> > > different versions of the objects live in the same cache? GridGain
> >
> > Being able to have different versions of the objects might be a bliss (I'm
> > not
> > sure if that would fit Ognen usecase). But it might come with the funny
> > situations such as when an older revision never got purged off a cache,
> > causing potential data-conflicts, etc. To me peerClassLoading seems to
> > fine,
> > but users need to be aware about its effects.
> >
> > Just thinking out loud, really.
> >   Cos
> >
> > > enterprise edition, by the way, allows it with Portable Objects.
> > >
> > > If you would like to clear the older object versions and deploy new ones
> > > automatically, Ignite supports it already. Just enable peerClassLoading
> > on
> > > Ignite configuration.
> > >
> > > D.
> > >
> > > On Tue, Apr 28, 2015 at 8:43 AM, Ognen Duzlevski <
> > [hidden email]>
> > > wrote:
> > >
> > > > Cos, thanks. If anyone has any suggestions, I would be glad to hear
> > them
> > > > :-)
> > > >
> > > > On Tue, Apr 28, 2015 at 12:59 AM, Konstantin Boudnik <[hidden email]>
> > > > wrote:
> > > >
> > > > > I think Ognen has raised a very valid concern in [1]: what to do
> > when you
> > > > > have
> > > > > a huge cluster? Esp. on multi-tenant systems you can not just bounce
> > the
> > > > > whole
> > > > > thing on every application's DO classes change. Is it possible to do
> > > > > rolling
> > > > > restart of the cluster nodes, where some of the nodes will still be
> > on
> > > > the
> > > > > old
> > > > > version of the classes and some on the new one?
> > > > >
> > > > > Sorry for using the nabble link, but
> > > > > mail-archives.apache.org/mod_mbox/ignite-dev takes forever to get
> > > > updated.
> > > > >
> > > > > Cos
> > > > >
> > > > > [1]
> > > > >
> > > >
> > http://apache-ignite-developers.2346864.n4.nabble.com/New-contributor-tp114p123.html
> > > > >
> > > > > On Mon, Apr 27, 2015 at 08:57AM, Dmitriy Setrakyan wrote:
> > > > > > Ognen,
> > > > > >
> > > > > > What kind of behavior do you need for the deployment? For example,
> > if
> > > > > your
> > > > > > caches already have data with older classes, what should happen to
> > that
> > > > > > data?
> > > > > >
> > > > > > Of course, the easiest way to accomplish what you are doing is to
> > copy
> > > > > the
> > > > > > new jars everywhere and restart the cluster. If this approach is
> > > > > > acceptable, I would just go with it.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Sun, Apr 26, 2015 at 7:29 PM, Ognen Duzlevski <
> > > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > I have a 5 machine ignite grid deployed on EC2 (still very much
> > in
> > > > > testing
> > > > > > > but would like to move to becoming more serious about using it in
> > > > > research
> > > > > > > at least and eventually in production). In order to be able to
> > cache
> > > > > > > various Scala/java classes, I created a fat jar of my Scala app
> > and
> > > > > put it
> > > > > > > in the ignite/libs/ subdirectory. Then I start ignite on each
> > node by
> > > > > > > running the ignite.sh script.
> > > > > > >
> > > > > > > When I add new classes, I recompile the app and create a new fat
> > jar.
> > > > > How
> > > > > > > does one deploy this new code so that the already running ignite
> > > > > becomes
> > > > > > > aware of it? I am not much of a Java programmer (jumped straight
> > to
> > > > > Scala)
> > > > > > > - are there any ways Java allows for loading new classes/jars
> > "on the
> > > > > fly"
> > > > > > > into running JVMs? If so, does Ignite support them? If not, what
> > do
> > > > > people
> > > > > > > do to deploy new jars so they can become usable to an already
> > running
> > > > > cache
> > > > > > > grid?
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Ognen
> > > > > > >
> > > > >
> > > >
> >