Service grid redesign

[dmagda]

Denis,

I would encourage us to persist a service configuration in the meta store
and have this capability enabled by default. That's essential for services
started dynamically. Moreover, we support similar behavior for caches,
indexes, and other DDL changes happened at runtime.

--
Denis

On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <[hidden email]>
wrote:

> Another question, that I would like to discuss is whether services should
> be preserved on cluster restarts.
>
> Currently it depends on persistence configuration. If persistence for any
> data region is enabled, then services will be persisted as well. This is a
> pretty strange way of configuring this behaviour.
> I'm not sure, if anybody relies on this functionality right now. Should we
> support it at all? If yes, should we make it configurable?
>
> Denis
>
> пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <[hidden email]>:
>
> > Val,
> >
> > Sounds reasonable. I just think, that user should have some way to know,
> > that new version of a service class was deployed.
> > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm not sure,
> > whether it is triggered on class redeployment, though. If not, then
> another
> > event type should be added.
> >
> > I don't think, that a lot of people will implement their own
> > *DeploymentSpi*-s, so we should make work with *UriDeploymentSpi* as
> > comfortable as possible.
> >
> > Denis
> >
> > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > [hidden email]>:
> >
> >> Yes, the class deployment itself has to be explicit. I.e., there has to
> be
> >> a manual step where user updates the class, and the exact step required
> >> would depend on DeploymentSpi implementation. But then Ignite takes care
> >> of
> >> everything else - service redeployment and restart is automatic.
> >>
> >> Dmitriy Pavlov, all this is going to be disabled if DeploymentSpi is not
> >> configured. In this case service class definitions have to be deployed
> on
> >> local classpath and can't be updated in runtime. Just like it works
> right
> >> now.
> >>
> >> -Val
> >>
> >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> [hidden email]
> >> >
> >> wrote:
> >>
> >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <[hidden email]>
> >> > wrote:
> >> >
> >> > > Hi Igniters,
> >> > >
> >> > > I like automatic redeploy which can be disabled by config if user
> >> wants
> >> > to
> >> > > control this process. What do you think?
> >> > >
> >> >
> >> > I do not think we should have anything automatic when it comes to
> >> > deployment, everything should be explicit. However, if we use the
> >> > deployment SPI, then a user should be able to do "hot" redeploy,
> where a
> >> > new service will be deployed if the user drops an updated jar.
> >> >
> >> > We should not create anything new here. Ignite already has a
> deployment
> >> SPI
> >> > and it already works in a certain way. Let's not change it.
> >> >
> >> > D.
> >> >
> >>
> >
>

[Valentin Kulichenko]

Denis,

EVT_CLASS_DEPLOYED should be fired every time a class is deployed or
redeployed. If this doesn't happen in some cases, I believe this would be a
bug. I don't think we need to add any new events.

-Val

On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]> wrote:

> Denis,
>
> I would encourage us to persist a service configuration in the meta store
> and have this capability enabled by default. That's essential for services
> started dynamically. Moreover, we support similar behavior for caches,
> indexes, and other DDL changes happened at runtime.
>
> --
> Denis
>
> On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <[hidden email]>
> wrote:
>
> > Another question, that I would like to discuss is whether services should
> > be preserved on cluster restarts.
> >
> > Currently it depends on persistence configuration. If persistence for any
> > data region is enabled, then services will be persisted as well. This is
> a
> > pretty strange way of configuring this behaviour.
> > I'm not sure, if anybody relies on this functionality right now. Should
> we
> > support it at all? If yes, should we make it configurable?
> >
> > Denis
> >
> > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <[hidden email]>:
> >
> > > Val,
> > >
> > > Sounds reasonable. I just think, that user should have some way to
> know,
> > > that new version of a service class was deployed.
> > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm not sure,
> > > whether it is triggered on class redeployment, though. If not, then
> > another
> > > event type should be added.
> > >
> > > I don't think, that a lot of people will implement their own
> > > *DeploymentSpi*-s, so we should make work with *UriDeploymentSpi* as
> > > comfortable as possible.
> > >
> > > Denis
> > >
> > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > [hidden email]>:
> > >
> > >> Yes, the class deployment itself has to be explicit. I.e., there has
> to
> > be
> > >> a manual step where user updates the class, and the exact step
> required
> > >> would depend on DeploymentSpi implementation. But then Ignite takes
> care
> > >> of
> > >> everything else - service redeployment and restart is automatic.
> > >>
> > >> Dmitriy Pavlov, all this is going to be disabled if DeploymentSpi is
> not
> > >> configured. In this case service class definitions have to be deployed
> > on
> > >> local classpath and can't be updated in runtime. Just like it works
> > right
> > >> now.
> > >>
> > >> -Val
> > >>
> > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> > [hidden email]
> > >> >
> > >> wrote:
> > >>
> > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> [hidden email]>
> > >> > wrote:
> > >> >
> > >> > > Hi Igniters,
> > >> > >
> > >> > > I like automatic redeploy which can be disabled by config if user
> > >> wants
> > >> > to
> > >> > > control this process. What do you think?
> > >> > >
> > >> >
> > >> > I do not think we should have anything automatic when it comes to
> > >> > deployment, everything should be explicit. However, if we use the
> > >> > deployment SPI, then a user should be able to do "hot" redeploy,
> > where a
> > >> > new service will be deployed if the user drops an updated jar.
> > >> >
> > >> > We should not create anything new here. Ignite already has a
> > deployment
> > >> SPI
> > >> > and it already works in a certain way. Let's not change it.
> > >> >
> > >> > D.
> > >> >
> > >>
> > >
> >
>

[dmagda]

Val,

Guess we're talking about other situations. I'm bringing up the case when a
service was deployed dynamically and has to be brought up after a full
cluster restart w/o user intervention. To achieve this we need to persist
the service's configuration somewhere.

--
Denis

On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
[hidden email]> wrote:

> Denis,
>
> EVT_CLASS_DEPLOYED should be fired every time a class is deployed or
> redeployed. If this doesn't happen in some cases, I believe this would be a
> bug. I don't think we need to add any new events.
>
> -Val
>
> On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]> wrote:
>
> > Denis,
> >
> > I would encourage us to persist a service configuration in the meta store
> > and have this capability enabled by default. That's essential for
> services
> > started dynamically. Moreover, we support similar behavior for caches,
> > indexes, and other DDL changes happened at runtime.
> >
> > --
> > Denis
> >
> > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <[hidden email]>
> > wrote:
> >
> > > Another question, that I would like to discuss is whether services
> should
> > > be preserved on cluster restarts.
> > >
> > > Currently it depends on persistence configuration. If persistence for
> any
> > > data region is enabled, then services will be persisted as well. This
> is
> > a
> > > pretty strange way of configuring this behaviour.
> > > I'm not sure, if anybody relies on this functionality right now. Should
> > we
> > > support it at all? If yes, should we make it configurable?
> > >
> > > Denis
> > >
> > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <[hidden email]>:
> > >
> > > > Val,
> > > >
> > > > Sounds reasonable. I just think, that user should have some way to
> > know,
> > > > that new version of a service class was deployed.
> > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm not sure,
> > > > whether it is triggered on class redeployment, though. If not, then
> > > another
> > > > event type should be added.
> > > >
> > > > I don't think, that a lot of people will implement their own
> > > > *DeploymentSpi*-s, so we should make work with *UriDeploymentSpi* as
> > > > comfortable as possible.
> > > >
> > > > Denis
> > > >
> > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > > [hidden email]>:
> > > >
> > > >> Yes, the class deployment itself has to be explicit. I.e., there has
> > to
> > > be
> > > >> a manual step where user updates the class, and the exact step
> > required
> > > >> would depend on DeploymentSpi implementation. But then Ignite takes
> > care
> > > >> of
> > > >> everything else - service redeployment and restart is automatic.
> > > >>
> > > >> Dmitriy Pavlov, all this is going to be disabled if DeploymentSpi is
> > not
> > > >> configured. In this case service class definitions have to be
> deployed
> > > on
> > > >> local classpath and can't be updated in runtime. Just like it works
> > > right
> > > >> now.
> > > >>
> > > >> -Val
> > > >>
> > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> > > [hidden email]
> > > >> >
> > > >> wrote:
> > > >>
> > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > [hidden email]>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Igniters,
> > > >> > >
> > > >> > > I like automatic redeploy which can be disabled by config if
> user
> > > >> wants
> > > >> > to
> > > >> > > control this process. What do you think?
> > > >> > >
> > > >> >
> > > >> > I do not think we should have anything automatic when it comes to
> > > >> > deployment, everything should be explicit. However, if we use the
> > > >> > deployment SPI, then a user should be able to do "hot" redeploy,
> > > where a
> > > >> > new service will be deployed if the user drops an updated jar.
> > > >> >
> > > >> > We should not create anything new here. Ignite already has a
> > > deployment
> > > >> SPI
> > > >> > and it already works in a certain way. Let's not change it.
> > > >> >
> > > >> > D.
> > > >> >
> > > >>
> > > >
> > >
> >
>

[Valentin Kulichenko]

I was responding to another Denis :) Agree with you on your point though.

-Val

On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]> wrote:

> Val,
>
> Guess we're talking about other situations. I'm bringing up the case when a
> service was deployed dynamically and has to be brought up after a full
> cluster restart w/o user intervention. To achieve this we need to persist
> the service's configuration somewhere.
>
> --
> Denis
>
> On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> [hidden email]> wrote:
>
> > Denis,
> >
> > EVT_CLASS_DEPLOYED should be fired every time a class is deployed or
> > redeployed. If this doesn't happen in some cases, I believe this would
> be a
> > bug. I don't think we need to add any new events.
> >
> > -Val
> >
> > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]> wrote:
> >
> > > Denis,
> > >
> > > I would encourage us to persist a service configuration in the meta
> store
> > > and have this capability enabled by default. That's essential for
> > services
> > > started dynamically. Moreover, we support similar behavior for caches,
> > > indexes, and other DDL changes happened at runtime.
> > >
> > > --
> > > Denis
> > >
> > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> [hidden email]>
> > > wrote:
> > >
> > > > Another question, that I would like to discuss is whether services
> > should
> > > > be preserved on cluster restarts.
> > > >
> > > > Currently it depends on persistence configuration. If persistence for
> > any
> > > > data region is enabled, then services will be persisted as well. This
> > is
> > > a
> > > > pretty strange way of configuring this behaviour.
> > > > I'm not sure, if anybody relies on this functionality right now.
> Should
> > > we
> > > > support it at all? If yes, should we make it configurable?
> > > >
> > > > Denis
> > > >
> > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <[hidden email]
> >:
> > > >
> > > > > Val,
> > > > >
> > > > > Sounds reasonable. I just think, that user should have some way to
> > > know,
> > > > > that new version of a service class was deployed.
> > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm not
> sure,
> > > > > whether it is triggered on class redeployment, though. If not, then
> > > > another
> > > > > event type should be added.
> > > > >
> > > > > I don't think, that a lot of people will implement their own
> > > > > *DeploymentSpi*-s, so we should make work with *UriDeploymentSpi*
> as
> > > > > comfortable as possible.
> > > > >
> > > > > Denis
> > > > >
> > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > > > [hidden email]>:
> > > > >
> > > > >> Yes, the class deployment itself has to be explicit. I.e., there
> has
> > > to
> > > > be
> > > > >> a manual step where user updates the class, and the exact step
> > > required
> > > > >> would depend on DeploymentSpi implementation. But then Ignite
> takes
> > > care
> > > > >> of
> > > > >> everything else - service redeployment and restart is automatic.
> > > > >>
> > > > >> Dmitriy Pavlov, all this is going to be disabled if DeploymentSpi
> is
> > > not
> > > > >> configured. In this case service class definitions have to be
> > deployed
> > > > on
> > > > >> local classpath and can't be updated in runtime. Just like it
> works
> > > > right
> > > > >> now.
> > > > >>
> > > > >> -Val
> > > > >>
> > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > >> >
> > > > >> wrote:
> > > > >>
> > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > > [hidden email]>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Igniters,
> > > > >> > >
> > > > >> > > I like automatic redeploy which can be disabled by config if
> > user
> > > > >> wants
> > > > >> > to
> > > > >> > > control this process. What do you think?
> > > > >> > >
> > > > >> >
> > > > >> > I do not think we should have anything automatic when it comes
> to
> > > > >> > deployment, everything should be explicit. However, if we use
> the
> > > > >> > deployment SPI, then a user should be able to do "hot" redeploy,
> > > > where a
> > > > >> > new service will be deployed if the user drops an updated jar.
> > > > >> >
> > > > >> > We should not create anything new here. Ignite already has a
> > > > deployment
> > > > >> SPI
> > > > >> > and it already works in a certain way. Let's not change it.
> > > > >> >
> > > > >> > D.
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

[Denis Mekhanikov]

Denis,

Sounds reasonable. It's not clear, though, what should happen, if a joining
node has some services persisted, that are missing on other nodes.
Should we deploy them?
If we do so, it could lead to surprising behaviour. For example you could
kill a node, undeploy a service, then bring back an old node, and it would
make the service resurrect.
We could store some deployment counter along with the service
configurations on all nodes, that would show how many times the service
state has changed, i.e. it has been undeployed/redeployed. It should be
kept for undeployed services as well to avoid situations like I described.

But it still leaves a possibility of incorrect behaviour, if there was a
split-brain situation at some point. I don't think we should precess it
somehow, though. If we choose to tackle it, it will overcomplicate things
for a sake of a minor improvement.

Denis

вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
[hidden email]>:

> I was responding to another Denis :) Agree with you on your point though.
>
> -Val
>
> On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]> wrote:
>
> > Val,
> >
> > Guess we're talking about other situations. I'm bringing up the case
> when a
> > service was deployed dynamically and has to be brought up after a full
> > cluster restart w/o user intervention. To achieve this we need to persist
> > the service's configuration somewhere.
> >
> > --
> > Denis
> >
> > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > [hidden email]> wrote:
> >
> > > Denis,
> > >
> > > EVT_CLASS_DEPLOYED should be fired every time a class is deployed or
> > > redeployed. If this doesn't happen in some cases, I believe this would
> > be a
> > > bug. I don't think we need to add any new events.
> > >
> > > -Val
> > >
> > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]>
> wrote:
> > >
> > > > Denis,
> > > >
> > > > I would encourage us to persist a service configuration in the meta
> > store
> > > > and have this capability enabled by default. That's essential for
> > > services
> > > > started dynamically. Moreover, we support similar behavior for
> caches,
> > > > indexes, and other DDL changes happened at runtime.
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Another question, that I would like to discuss is whether services
> > > should
> > > > > be preserved on cluster restarts.
> > > > >
> > > > > Currently it depends on persistence configuration. If persistence
> for
> > > any
> > > > > data region is enabled, then services will be persisted as well.
> This
> > > is
> > > > a
> > > > > pretty strange way of configuring this behaviour.
> > > > > I'm not sure, if anybody relies on this functionality right now.
> > Should
> > > > we
> > > > > support it at all? If yes, should we make it configurable?
> > > > >
> > > > > Denis
> > > > >
> > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> [hidden email]
> > >:
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > Sounds reasonable. I just think, that user should have some way
> to
> > > > know,
> > > > > > that new version of a service class was deployed.
> > > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm not
> > sure,
> > > > > > whether it is triggered on class redeployment, though. If not,
> then
> > > > > another
> > > > > > event type should be added.
> > > > > >
> > > > > > I don't think, that a lot of people will implement their own
> > > > > > *DeploymentSpi*-s, so we should make work with *UriDeploymentSpi*
> > as
> > > > > > comfortable as possible.
> > > > > >
> > > > > > Denis
> > > > > >
> > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > > > > [hidden email]>:
> > > > > >
> > > > > >> Yes, the class deployment itself has to be explicit. I.e., there
> > has
> > > > to
> > > > > be
> > > > > >> a manual step where user updates the class, and the exact step
> > > > required
> > > > > >> would depend on DeploymentSpi implementation. But then Ignite
> > takes
> > > > care
> > > > > >> of
> > > > > >> everything else - service redeployment and restart is automatic.
> > > > > >>
> > > > > >> Dmitriy Pavlov, all this is going to be disabled if
> DeploymentSpi
> > is
> > > > not
> > > > > >> configured. In this case service class definitions have to be
> > > deployed
> > > > > on
> > > > > >> local classpath and can't be updated in runtime. Just like it
> > works
> > > > > right
> > > > > >> now.
> > > > > >>
> > > > > >> -Val
> > > > > >>
> > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> > > > > [hidden email]
> > > > > >> >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > > > [hidden email]>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi Igniters,
> > > > > >> > >
> > > > > >> > > I like automatic redeploy which can be disabled by config if
> > > user
> > > > > >> wants
> > > > > >> > to
> > > > > >> > > control this process. What do you think?
> > > > > >> > >
> > > > > >> >
> > > > > >> > I do not think we should have anything automatic when it comes
> > to
> > > > > >> > deployment, everything should be explicit. However, if we use
> > the
> > > > > >> > deployment SPI, then a user should be able to do "hot"
> redeploy,
> > > > > where a
> > > > > >> > new service will be deployed if the user drops an updated jar.
> > > > > >> >
> > > > > >> > We should not create anything new here. Ignite already has a
> > > > > deployment
> > > > > >> SPI
> > > > > >> > and it already works in a certain way. Let's not change it.
> > > > > >> >
> > > > > >> > D.
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Denis Mekhanikov]

Guys,

I'm also thinking, at which moment services should be deployed on the
joining nodes.
It would be good to have all services deployed by the moment, when a node
is accepted to the topology.
I think, it should work like this:

   1. connecting node sends a *TcpDiscoveryJoinRequestMessage *with
   persisted services configurations attached to it;
   2. coordinator recalculates service assignments and attaches them to the
   successive *TcpDiscoveryNodeAddedMessage*;
   3. connecting node receives the assignments, initialises all needed
   services and sends confirmation to the coordinator on completion;
   4. coordinator sends *TcpDiscoveryNodeAddFinishedMessage* only when it
   receives confirmation about deployed services from the joining node.

What do you think? Any pitfalls? Some discovery expert should look at this
procedure and tell, if it is viable.

Denis

ср, 11 апр. 2018 г. в 11:28, Denis Mekhanikov <[hidden email]>:

> Denis,
>
> Sounds reasonable. It's not clear, though, what should happen, if a
> joining node has some services persisted, that are missing on other nodes.
> Should we deploy them?
> If we do so, it could lead to surprising behaviour. For example you could
> kill a node, undeploy a service, then bring back an old node, and it would
> make the service resurrect.
> We could store some deployment counter along with the service
> configurations on all nodes, that would show how many times the service
> state has changed, i.e. it has been undeployed/redeployed. It should be
> kept for undeployed services as well to avoid situations like I described.
>
> But it still leaves a possibility of incorrect behaviour, if there was a
> split-brain situation at some point. I don't think we should precess it
> somehow, though. If we choose to tackle it, it will overcomplicate things
> for a sake of a minor improvement.
>
> Denis
>
> вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> [hidden email]>:
>
>> I was responding to another Denis :) Agree with you on your point though.
>>
>> -Val
>>
>> On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]> wrote:
>>
>> > Val,
>> >
>> > Guess we're talking about other situations. I'm bringing up the case
>> when a
>> > service was deployed dynamically and has to be brought up after a full
>> > cluster restart w/o user intervention. To achieve this we need to
>> persist
>> > the service's configuration somewhere.
>> >
>> > --
>> > Denis
>> >
>> > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
>> > [hidden email]> wrote:
>> >
>> > > Denis,
>> > >
>> > > EVT_CLASS_DEPLOYED should be fired every time a class is deployed or
>> > > redeployed. If this doesn't happen in some cases, I believe this would
>> > be a
>> > > bug. I don't think we need to add any new events.
>> > >
>> > > -Val
>> > >
>> > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]>
>> wrote:
>> > >
>> > > > Denis,
>> > > >
>> > > > I would encourage us to persist a service configuration in the meta
>> > store
>> > > > and have this capability enabled by default. That's essential for
>> > > services
>> > > > started dynamically. Moreover, we support similar behavior for
>> caches,
>> > > > indexes, and other DDL changes happened at runtime.
>> > > >
>> > > > --
>> > > > Denis
>> > > >
>> > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
>> > [hidden email]>
>> > > > wrote:
>> > > >
>> > > > > Another question, that I would like to discuss is whether services
>> > > should
>> > > > > be preserved on cluster restarts.
>> > > > >
>> > > > > Currently it depends on persistence configuration. If persistence
>> for
>> > > any
>> > > > > data region is enabled, then services will be persisted as well.
>> This
>> > > is
>> > > > a
>> > > > > pretty strange way of configuring this behaviour.
>> > > > > I'm not sure, if anybody relies on this functionality right now.
>> > Should
>> > > > we
>> > > > > support it at all? If yes, should we make it configurable?
>> > > > >
>> > > > > Denis
>> > > > >
>> > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
>> [hidden email]
>> > >:
>> > > > >
>> > > > > > Val,
>> > > > > >
>> > > > > > Sounds reasonable. I just think, that user should have some way
>> to
>> > > > know,
>> > > > > > that new version of a service class was deployed.
>> > > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm not
>> > sure,
>> > > > > > whether it is triggered on class redeployment, though. If not,
>> then
>> > > > > another
>> > > > > > event type should be added.
>> > > > > >
>> > > > > > I don't think, that a lot of people will implement their own
>> > > > > > *DeploymentSpi*-s, so we should make work with
>> *UriDeploymentSpi*
>> > as
>> > > > > > comfortable as possible.
>> > > > > >
>> > > > > > Denis
>> > > > > >
>> > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
>> > > > > > [hidden email]>:
>> > > > > >
>> > > > > >> Yes, the class deployment itself has to be explicit. I.e.,
>> there
>> > has
>> > > > to
>> > > > > be
>> > > > > >> a manual step where user updates the class, and the exact step
>> > > > required
>> > > > > >> would depend on DeploymentSpi implementation. But then Ignite
>> > takes
>> > > > care
>> > > > > >> of
>> > > > > >> everything else - service redeployment and restart is
>> automatic.
>> > > > > >>
>> > > > > >> Dmitriy Pavlov, all this is going to be disabled if
>> DeploymentSpi
>> > is
>> > > > not
>> > > > > >> configured. In this case service class definitions have to be
>> > > deployed
>> > > > > on
>> > > > > >> local classpath and can't be updated in runtime. Just like it
>> > works
>> > > > > right
>> > > > > >> now.
>> > > > > >>
>> > > > > >> -Val
>> > > > > >>
>> > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
>> > > > > [hidden email]
>> > > > > >> >
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
>> > > > [hidden email]>
>> > > > > >> > wrote:
>> > > > > >> >
>> > > > > >> > > Hi Igniters,
>> > > > > >> > >
>> > > > > >> > > I like automatic redeploy which can be disabled by config
>> if
>> > > user
>> > > > > >> wants
>> > > > > >> > to
>> > > > > >> > > control this process. What do you think?
>> > > > > >> > >
>> > > > > >> >
>> > > > > >> > I do not think we should have anything automatic when it
>> comes
>> > to
>> > > > > >> > deployment, everything should be explicit. However, if we use
>> > the
>> > > > > >> > deployment SPI, then a user should be able to do "hot"
>> redeploy,
>> > > > > where a
>> > > > > >> > new service will be deployed if the user drops an updated
>> jar.
>> > > > > >> >
>> > > > > >> > We should not create anything new here. Ignite already has a
>> > > > > deployment
>> > > > > >> SPI
>> > > > > >> > and it already works in a certain way. Let's not change it.
>> > > > > >> >
>> > > > > >> > D.
>> > > > > >> >
>> > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

[dmagda]

Denis,

I think that the service deployment state needs be persisted cluster-wide.
I guess that our meta-store is capable of doing so. Alex G., Vladimir,
could you confirm?

As for the split-brain scenarios, I would put them aside for now because,
anyway, they have to be solved at lower levels (meta store, discovery,
etc.).

Also, I heard that presently we store a service configuration in the system
cache that doesn't give us a way to deploy a new version of a service
without undeployment of the previous one. Will this issue be addressed by
the new deployment approach?

--
Denis

On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <[hidden email]>
wrote:

> Denis,
>
> Sounds reasonable. It's not clear, though, what should happen, if a joining
> node has some services persisted, that are missing on other nodes.
> Should we deploy them?
> If we do so, it could lead to surprising behaviour. For example you could
> kill a node, undeploy a service, then bring back an old node, and it would
> make the service resurrect.
> We could store some deployment counter along with the service
> configurations on all nodes, that would show how many times the service
> state has changed, i.e. it has been undeployed/redeployed. It should be
> kept for undeployed services as well to avoid situations like I described.
>
> But it still leaves a possibility of incorrect behaviour, if there was a
> split-brain situation at some point. I don't think we should precess it
> somehow, though. If we choose to tackle it, it will overcomplicate things
> for a sake of a minor improvement.
>
> Denis
>
> вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> [hidden email]>:
>
> > I was responding to another Denis :) Agree with you on your point though.
> >
> > -Val
> >
> > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]> wrote:
> >
> > > Val,
> > >
> > > Guess we're talking about other situations. I'm bringing up the case
> > when a
> > > service was deployed dynamically and has to be brought up after a full
> > > cluster restart w/o user intervention. To achieve this we need to
> persist
> > > the service's configuration somewhere.
> > >
> > > --
> > > Denis
> > >
> > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > > [hidden email]> wrote:
> > >
> > > > Denis,
> > > >
> > > > EVT_CLASS_DEPLOYED should be fired every time a class is deployed or
> > > > redeployed. If this doesn't happen in some cases, I believe this
> would
> > > be a
> > > > bug. I don't think we need to add any new events.
> > > >
> > > > -Val
> > > >
> > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]>
> > wrote:
> > > >
> > > > > Denis,
> > > > >
> > > > > I would encourage us to persist a service configuration in the meta
> > > store
> > > > > and have this capability enabled by default. That's essential for
> > > > services
> > > > > started dynamically. Moreover, we support similar behavior for
> > caches,
> > > > > indexes, and other DDL changes happened at runtime.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Another question, that I would like to discuss is whether
> services
> > > > should
> > > > > > be preserved on cluster restarts.
> > > > > >
> > > > > > Currently it depends on persistence configuration. If persistence
> > for
> > > > any
> > > > > > data region is enabled, then services will be persisted as well.
> > This
> > > > is
> > > > > a
> > > > > > pretty strange way of configuring this behaviour.
> > > > > > I'm not sure, if anybody relies on this functionality right now.
> > > Should
> > > > > we
> > > > > > support it at all? If yes, should we make it configurable?
> > > > > >
> > > > > > Denis
> > > > > >
> > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > [hidden email]
> > > >:
> > > > > >
> > > > > > > Val,
> > > > > > >
> > > > > > > Sounds reasonable. I just think, that user should have some way
> > to
> > > > > know,
> > > > > > > that new version of a service class was deployed.
> > > > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm not
> > > sure,
> > > > > > > whether it is triggered on class redeployment, though. If not,
> > then
> > > > > > another
> > > > > > > event type should be added.
> > > > > > >
> > > > > > > I don't think, that a lot of people will implement their own
> > > > > > > *DeploymentSpi*-s, so we should make work with
> *UriDeploymentSpi*
> > > as
> > > > > > > comfortable as possible.
> > > > > > >
> > > > > > > Denis
> > > > > > >
> > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > > > > > [hidden email]>:
> > > > > > >
> > > > > > >> Yes, the class deployment itself has to be explicit. I.e.,
> there
> > > has
> > > > > to
> > > > > > be
> > > > > > >> a manual step where user updates the class, and the exact step
> > > > > required
> > > > > > >> would depend on DeploymentSpi implementation. But then Ignite
> > > takes
> > > > > care
> > > > > > >> of
> > > > > > >> everything else - service redeployment and restart is
> automatic.
> > > > > > >>
> > > > > > >> Dmitriy Pavlov, all this is going to be disabled if
> > DeploymentSpi
> > > is
> > > > > not
> > > > > > >> configured. In this case service class definitions have to be
> > > > deployed
> > > > > > on
> > > > > > >> local classpath and can't be updated in runtime. Just like it
> > > works
> > > > > > right
> > > > > > >> now.
> > > > > > >>
> > > > > > >> -Val
> > > > > > >>
> > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> > > > > > [hidden email]
> > > > > > >> >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > > > > [hidden email]>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Igniters,
> > > > > > >> > >
> > > > > > >> > > I like automatic redeploy which can be disabled by config
> if
> > > > user
> > > > > > >> wants
> > > > > > >> > to
> > > > > > >> > > control this process. What do you think?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > I do not think we should have anything automatic when it
> comes
> > > to
> > > > > > >> > deployment, everything should be explicit. However, if we
> use
> > > the
> > > > > > >> > deployment SPI, then a user should be able to do "hot"
> > redeploy,
> > > > > > where a
> > > > > > >> > new service will be deployed if the user drops an updated
> jar.
> > > > > > >> >
> > > > > > >> > We should not create anything new here. Ignite already has a
> > > > > > deployment
> > > > > > >> SPI
> > > > > > >> > and it already works in a certain way. Let's not change it.
> > > > > > >> >
> > > > > > >> > D.
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Alexey Goncharuk]

Denis,

This is a subtle question. It looks like we have now a number of use-cases
when persistent storage is required even for a pure in-memory mode. One of
the use-cases is thin client authentication, the other is service grid
configuration persistence.

Generally, I would agree that this is an expected behavior. However, this
means that a user cannot simply start and stop nodes randomly anymore.
Ignite start will require some sort of installation or work folder
initialization (sort of initdb in postgres) which is ok for
persistence-enabled modes, but I am not sure if this is expected for
in-memory. Of course, we can run this initialization automatically, but it
is not always a good idea.

If we are ok to have this restrictions for in-memory mode, then service
persistence makes sense.

--AG

2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:

> Denis,
>
> I think that the service deployment state needs be persisted cluster-wide.
> I guess that our meta-store is capable of doing so. Alex G., Vladimir,
> could you confirm?
>
> As for the split-brain scenarios, I would put them aside for now because,
> anyway, they have to be solved at lower levels (meta store, discovery,
> etc.).
>
> Also, I heard that presently we store a service configuration in the system
> cache that doesn't give us a way to deploy a new version of a service
> without undeployment of the previous one. Will this issue be addressed by
> the new deployment approach?
>
> --
> Denis
>
> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <[hidden email]>
> wrote:
>
> > Denis,
> >
> > Sounds reasonable. It's not clear, though, what should happen, if a
> joining
> > node has some services persisted, that are missing on other nodes.
> > Should we deploy them?
> > If we do so, it could lead to surprising behaviour. For example you could
> > kill a node, undeploy a service, then bring back an old node, and it
> would
> > make the service resurrect.
> > We could store some deployment counter along with the service
> > configurations on all nodes, that would show how many times the service
> > state has changed, i.e. it has been undeployed/redeployed. It should be
> > kept for undeployed services as well to avoid situations like I
> described.
> >
> > But it still leaves a possibility of incorrect behaviour, if there was a
> > split-brain situation at some point. I don't think we should precess it
> > somehow, though. If we choose to tackle it, it will overcomplicate things
> > for a sake of a minor improvement.
> >
> > Denis
> >
> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > [hidden email]>:
> >
> > > I was responding to another Denis :) Agree with you on your point
> though.
> > >
> > > -Val
> > >
> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]> wrote:
> > >
> > > > Val,
> > > >
> > > > Guess we're talking about other situations. I'm bringing up the case
> > > when a
> > > > service was deployed dynamically and has to be brought up after a
> full
> > > > cluster restart w/o user intervention. To achieve this we need to
> > persist
> > > > the service's configuration somewhere.
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > > > [hidden email]> wrote:
> > > >
> > > > > Denis,
> > > > >
> > > > > EVT_CLASS_DEPLOYED should be fired every time a class is deployed
> or
> > > > > redeployed. If this doesn't happen in some cases, I believe this
> > would
> > > > be a
> > > > > bug. I don't think we need to add any new events.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Denis,
> > > > > >
> > > > > > I would encourage us to persist a service configuration in the
> meta
> > > > store
> > > > > > and have this capability enabled by default. That's essential for
> > > > > services
> > > > > > started dynamically. Moreover, we support similar behavior for
> > > caches,
> > > > > > indexes, and other DDL changes happened at runtime.
> > > > > >
> > > > > > --
> > > > > > Denis
> > > > > >
> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Another question, that I would like to discuss is whether
> > services
> > > > > should
> > > > > > > be preserved on cluster restarts.
> > > > > > >
> > > > > > > Currently it depends on persistence configuration. If
> persistence
> > > for
> > > > > any
> > > > > > > data region is enabled, then services will be persisted as
> well.
> > > This
> > > > > is
> > > > > > a
> > > > > > > pretty strange way of configuring this behaviour.
> > > > > > > I'm not sure, if anybody relies on this functionality right
> now.
> > > > Should
> > > > > > we
> > > > > > > support it at all? If yes, should we make it configurable?
> > > > > > >
> > > > > > > Denis
> > > > > > >
> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > > [hidden email]
> > > > >:
> > > > > > >
> > > > > > > > Val,
> > > > > > > >
> > > > > > > > Sounds reasonable. I just think, that user should have some
> way
> > > to
> > > > > > know,
> > > > > > > > that new version of a service class was deployed.
> > > > > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm
> not
> > > > sure,
> > > > > > > > whether it is triggered on class redeployment, though. If
> not,
> > > then
> > > > > > > another
> > > > > > > > event type should be added.
> > > > > > > >
> > > > > > > > I don't think, that a lot of people will implement their own
> > > > > > > > *DeploymentSpi*-s, so we should make work with
> > *UriDeploymentSpi*
> > > > as
> > > > > > > > comfortable as possible.
> > > > > > > >
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > > > > > > [hidden email]>:
> > > > > > > >
> > > > > > > >> Yes, the class deployment itself has to be explicit. I.e.,
> > there
> > > > has
> > > > > > to
> > > > > > > be
> > > > > > > >> a manual step where user updates the class, and the exact
> step
> > > > > > required
> > > > > > > >> would depend on DeploymentSpi implementation. But then
> Ignite
> > > > takes
> > > > > > care
> > > > > > > >> of
> > > > > > > >> everything else - service redeployment and restart is
> > automatic.
> > > > > > > >>
> > > > > > > >> Dmitriy Pavlov, all this is going to be disabled if
> > > DeploymentSpi
> > > > is
> > > > > > not
> > > > > > > >> configured. In this case service class definitions have to
> be
> > > > > deployed
> > > > > > > on
> > > > > > > >> local classpath and can't be updated in runtime. Just like
> it
> > > > works
> > > > > > > right
> > > > > > > >> now.
> > > > > > > >>
> > > > > > > >> -Val
> > > > > > > >>
> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> > > > > > > [hidden email]
> > > > > > > >> >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > > > > > [hidden email]>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Igniters,
> > > > > > > >> > >
> > > > > > > >> > > I like automatic redeploy which can be disabled by
> config
> > if
> > > > > user
> > > > > > > >> wants
> > > > > > > >> > to
> > > > > > > >> > > control this process. What do you think?
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> > I do not think we should have anything automatic when it
> > comes
> > > > to
> > > > > > > >> > deployment, everything should be explicit. However, if we
> > use
> > > > the
> > > > > > > >> > deployment SPI, then a user should be able to do "hot"
> > > redeploy,
> > > > > > > where a
> > > > > > > >> > new service will be deployed if the user drops an updated
> > jar.
> > > > > > > >> >
> > > > > > > >> > We should not create anything new here. Ignite already
> has a
> > > > > > > deployment
> > > > > > > >> SPI
> > > > > > > >> > and it already works in a certain way. Let's not change
> it.
> > > > > > > >> >
> > > > > > > >> > D.
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Vladimir Ozerov]

Alex,

I would say that we've already had this behavior for years - marshaller
cache. I think it is time to agree that "in-memory" != stateless. Instead
"in-memory" means "data is not stored on disk".
May be we can have a flag which will wipe out all metadata on node restart
(e.g. it could make sense for embedded clients)?

On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
[hidden email]> wrote:

> Denis,
>
> This is a subtle question. It looks like we have now a number of use-cases
> when persistent storage is required even for a pure in-memory mode. One of
> the use-cases is thin client authentication, the other is service grid
> configuration persistence.
>
> Generally, I would agree that this is an expected behavior. However, this
> means that a user cannot simply start and stop nodes randomly anymore.
> Ignite start will require some sort of installation or work folder
> initialization (sort of initdb in postgres) which is ok for
> persistence-enabled modes, but I am not sure if this is expected for
> in-memory. Of course, we can run this initialization automatically, but it
> is not always a good idea.
>
> If we are ok to have this restrictions for in-memory mode, then service
> persistence makes sense.
>
> --AG
>
> 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:
>
>> Denis,
>>
>> I think that the service deployment state needs be persisted cluster-wide.
>> I guess that our meta-store is capable of doing so. Alex G., Vladimir,
>> could you confirm?
>>
>> As for the split-brain scenarios, I would put them aside for now because,
>> anyway, they have to be solved at lower levels (meta store, discovery,
>> etc.).
>>
>> Also, I heard that presently we store a service configuration in the
>> system
>> cache that doesn't give us a way to deploy a new version of a service
>> without undeployment of the previous one. Will this issue be addressed by
>> the new deployment approach?
>>
>> --
>> Denis
>>
>> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <[hidden email]>
>> wrote:
>>
>> > Denis,
>> >
>> > Sounds reasonable. It's not clear, though, what should happen, if a
>> joining
>> > node has some services persisted, that are missing on other nodes.
>> > Should we deploy them?
>> > If we do so, it could lead to surprising behaviour. For example you
>> could
>> > kill a node, undeploy a service, then bring back an old node, and it
>> would
>> > make the service resurrect.
>> > We could store some deployment counter along with the service
>> > configurations on all nodes, that would show how many times the service
>> > state has changed, i.e. it has been undeployed/redeployed. It should be
>> > kept for undeployed services as well to avoid situations like I
>> described.
>> >
>> > But it still leaves a possibility of incorrect behaviour, if there was a
>> > split-brain situation at some point. I don't think we should precess it
>> > somehow, though. If we choose to tackle it, it will overcomplicate
>> things
>> > for a sake of a minor improvement.
>> >
>> > Denis
>> >
>> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
>> > [hidden email]>:
>> >
>> > > I was responding to another Denis :) Agree with you on your point
>> though.
>> > >
>> > > -Val
>> > >
>> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]>
>> wrote:
>> > >
>> > > > Val,
>> > > >
>> > > > Guess we're talking about other situations. I'm bringing up the case
>> > > when a
>> > > > service was deployed dynamically and has to be brought up after a
>> full
>> > > > cluster restart w/o user intervention. To achieve this we need to
>> > persist
>> > > > the service's configuration somewhere.
>> > > >
>> > > > --
>> > > > Denis
>> > > >
>> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
>> > > > [hidden email]> wrote:
>> > > >
>> > > > > Denis,
>> > > > >
>> > > > > EVT_CLASS_DEPLOYED should be fired every time a class is deployed
>> or
>> > > > > redeployed. If this doesn't happen in some cases, I believe this
>> > would
>> > > > be a
>> > > > > bug. I don't think we need to add any new events.
>> > > > >
>> > > > > -Val
>> > > > >
>> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]>
>> > > wrote:
>> > > > >
>> > > > > > Denis,
>> > > > > >
>> > > > > > I would encourage us to persist a service configuration in the
>> meta
>> > > > store
>> > > > > > and have this capability enabled by default. That's essential
>> for
>> > > > > services
>> > > > > > started dynamically. Moreover, we support similar behavior for
>> > > caches,
>> > > > > > indexes, and other DDL changes happened at runtime.
>> > > > > >
>> > > > > > --
>> > > > > > Denis
>> > > > > >
>> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
>> > > > [hidden email]>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Another question, that I would like to discuss is whether
>> > services
>> > > > > should
>> > > > > > > be preserved on cluster restarts.
>> > > > > > >
>> > > > > > > Currently it depends on persistence configuration. If
>> persistence
>> > > for
>> > > > > any
>> > > > > > > data region is enabled, then services will be persisted as
>> well.
>> > > This
>> > > > > is
>> > > > > > a
>> > > > > > > pretty strange way of configuring this behaviour.
>> > > > > > > I'm not sure, if anybody relies on this functionality right
>> now.
>> > > > Should
>> > > > > > we
>> > > > > > > support it at all? If yes, should we make it configurable?
>> > > > > > >
>> > > > > > > Denis
>> > > > > > >
>> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
>> > > [hidden email]
>> > > > >:
>> > > > > > >
>> > > > > > > > Val,
>> > > > > > > >
>> > > > > > > > Sounds reasonable. I just think, that user should have some
>> way
>> > > to
>> > > > > > know,
>> > > > > > > > that new version of a service class was deployed.
>> > > > > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm
>> not
>> > > > sure,
>> > > > > > > > whether it is triggered on class redeployment, though. If
>> not,
>> > > then
>> > > > > > > another
>> > > > > > > > event type should be added.
>> > > > > > > >
>> > > > > > > > I don't think, that a lot of people will implement their own
>> > > > > > > > *DeploymentSpi*-s, so we should make work with
>> > *UriDeploymentSpi*
>> > > > as
>> > > > > > > > comfortable as possible.
>> > > > > > > >
>> > > > > > > > Denis
>> > > > > > > >
>> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
>> > > > > > > > [hidden email]>:
>> > > > > > > >
>> > > > > > > >> Yes, the class deployment itself has to be explicit. I.e.,
>> > there
>> > > > has
>> > > > > > to
>> > > > > > > be
>> > > > > > > >> a manual step where user updates the class, and the exact
>> step
>> > > > > > required
>> > > > > > > >> would depend on DeploymentSpi implementation. But then
>> Ignite
>> > > > takes
>> > > > > > care
>> > > > > > > >> of
>> > > > > > > >> everything else - service redeployment and restart is
>> > automatic.
>> > > > > > > >>
>> > > > > > > >> Dmitriy Pavlov, all this is going to be disabled if
>> > > DeploymentSpi
>> > > > is
>> > > > > > not
>> > > > > > > >> configured. In this case service class definitions have to
>> be
>> > > > > deployed
>> > > > > > > on
>> > > > > > > >> local classpath and can't be updated in runtime. Just like
>> it
>> > > > works
>> > > > > > > right
>> > > > > > > >> now.
>> > > > > > > >>
>> > > > > > > >> -Val
>> > > > > > > >>
>> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
>> > > > > > > [hidden email]
>> > > > > > > >> >
>> > > > > > > >> wrote:
>> > > > > > > >>
>> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
>> > > > > > [hidden email]>
>> > > > > > > >> > wrote:
>> > > > > > > >> >
>> > > > > > > >> > > Hi Igniters,
>> > > > > > > >> > >
>> > > > > > > >> > > I like automatic redeploy which can be disabled by
>> config
>> > if
>> > > > > user
>> > > > > > > >> wants
>> > > > > > > >> > to
>> > > > > > > >> > > control this process. What do you think?
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >> > I do not think we should have anything automatic when it
>> > comes
>> > > > to
>> > > > > > > >> > deployment, everything should be explicit. However, if we
>> > use
>> > > > the
>> > > > > > > >> > deployment SPI, then a user should be able to do "hot"
>> > > redeploy,
>> > > > > > > where a
>> > > > > > > >> > new service will be deployed if the user drops an updated
>> > jar.
>> > > > > > > >> >
>> > > > > > > >> > We should not create anything new here. Ignite already
>> has a
>> > > > > > > deployment
>> > > > > > > >> SPI
>> > > > > > > >> > and it already works in a certain way. Let's not change
>> it.
>> > > > > > > >> >
>> > > > > > > >> > D.
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

[Denis Mekhanikov]

Vladimir,

Currently we don't save binary metadata to disk, when persistence is
disabled.
But we still persist marshaller mappings for some reason, and I personally
believe, that we shouldn't.

But I agree, that we should separate data and service persistence
configuration.
Right now persistence of services is configured in a pretty non-obvious
manner.
It should be a clear way to tell Ignite, whether you want services to be
persisted or not.

I'm not sure, that we should make "statefullness" in general configurable.
Users don't care much, whether metadata is preserved on restarts, or not.

Denis

пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <[hidden email]>:

> Alex,
>
> I would say that we've already had this behavior for years - marshaller
> cache. I think it is time to agree that "in-memory" != stateless. Instead
> "in-memory" means "data is not stored on disk".
> May be we can have a flag which will wipe out all metadata on node restart
> (e.g. it could make sense for embedded clients)?
>
> On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > Denis,
> >
> > This is a subtle question. It looks like we have now a number of
> use-cases
> > when persistent storage is required even for a pure in-memory mode. One
> of
> > the use-cases is thin client authentication, the other is service grid
> > configuration persistence.
> >
> > Generally, I would agree that this is an expected behavior. However, this
> > means that a user cannot simply start and stop nodes randomly anymore.
> > Ignite start will require some sort of installation or work folder
> > initialization (sort of initdb in postgres) which is ok for
> > persistence-enabled modes, but I am not sure if this is expected for
> > in-memory. Of course, we can run this initialization automatically, but
> it
> > is not always a good idea.
> >
> > If we are ok to have this restrictions for in-memory mode, then service
> > persistence makes sense.
> >
> > --AG
> >
> > 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:
> >
> >> Denis,
> >>
> >> I think that the service deployment state needs be persisted
> cluster-wide.
> >> I guess that our meta-store is capable of doing so. Alex G., Vladimir,
> >> could you confirm?
> >>
> >> As for the split-brain scenarios, I would put them aside for now
> because,
> >> anyway, they have to be solved at lower levels (meta store, discovery,
> >> etc.).
> >>
> >> Also, I heard that presently we store a service configuration in the
> >> system
> >> cache that doesn't give us a way to deploy a new version of a service
> >> without undeployment of the previous one. Will this issue be addressed
> by
> >> the new deployment approach?
> >>
> >> --
> >> Denis
> >>
> >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> [hidden email]>
> >> wrote:
> >>
> >> > Denis,
> >> >
> >> > Sounds reasonable. It's not clear, though, what should happen, if a
> >> joining
> >> > node has some services persisted, that are missing on other nodes.
> >> > Should we deploy them?
> >> > If we do so, it could lead to surprising behaviour. For example you
> >> could
> >> > kill a node, undeploy a service, then bring back an old node, and it
> >> would
> >> > make the service resurrect.
> >> > We could store some deployment counter along with the service
> >> > configurations on all nodes, that would show how many times the
> service
> >> > state has changed, i.e. it has been undeployed/redeployed. It should
> be
> >> > kept for undeployed services as well to avoid situations like I
> >> described.
> >> >
> >> > But it still leaves a possibility of incorrect behaviour, if there
> was a
> >> > split-brain situation at some point. I don't think we should precess
> it
> >> > somehow, though. If we choose to tackle it, it will overcomplicate
> >> things
> >> > for a sake of a minor improvement.
> >> >
> >> > Denis
> >> >
> >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> >> > [hidden email]>:
> >> >
> >> > > I was responding to another Denis :) Agree with you on your point
> >> though.
> >> > >
> >> > > -Val
> >> > >
> >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]>
> >> wrote:
> >> > >
> >> > > > Val,
> >> > > >
> >> > > > Guess we're talking about other situations. I'm bringing up the
> case
> >> > > when a
> >> > > > service was deployed dynamically and has to be brought up after a
> >> full
> >> > > > cluster restart w/o user intervention. To achieve this we need to
> >> > persist
> >> > > > the service's configuration somewhere.
> >> > > >
> >> > > > --
> >> > > > Denis
> >> > > >
> >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> >> > > > [hidden email]> wrote:
> >> > > >
> >> > > > > Denis,
> >> > > > >
> >> > > > > EVT_CLASS_DEPLOYED should be fired every time a class is
> deployed
> >> or
> >> > > > > redeployed. If this doesn't happen in some cases, I believe this
> >> > would
> >> > > > be a
> >> > > > > bug. I don't think we need to add any new events.
> >> > > > >
> >> > > > > -Val
> >> > > > >
> >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <[hidden email]
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > Denis,
> >> > > > > >
> >> > > > > > I would encourage us to persist a service configuration in the
> >> meta
> >> > > > store
> >> > > > > > and have this capability enabled by default. That's essential
> >> for
> >> > > > > services
> >> > > > > > started dynamically. Moreover, we support similar behavior for
> >> > > caches,
> >> > > > > > indexes, and other DDL changes happened at runtime.
> >> > > > > >
> >> > > > > > --
> >> > > > > > Denis
> >> > > > > >
> >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> >> > > > [hidden email]>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Another question, that I would like to discuss is whether
> >> > services
> >> > > > > should
> >> > > > > > > be preserved on cluster restarts.
> >> > > > > > >
> >> > > > > > > Currently it depends on persistence configuration. If
> >> persistence
> >> > > for
> >> > > > > any
> >> > > > > > > data region is enabled, then services will be persisted as
> >> well.
> >> > > This
> >> > > > > is
> >> > > > > > a
> >> > > > > > > pretty strange way of configuring this behaviour.
> >> > > > > > > I'm not sure, if anybody relies on this functionality right
> >> now.
> >> > > > Should
> >> > > > > > we
> >> > > > > > > support it at all? If yes, should we make it configurable?
> >> > > > > > >
> >> > > > > > > Denis
> >> > > > > > >
> >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> >> > > [hidden email]
> >> > > > >:
> >> > > > > > >
> >> > > > > > > > Val,
> >> > > > > > > >
> >> > > > > > > > Sounds reasonable. I just think, that user should have
> some
> >> way
> >> > > to
> >> > > > > > know,
> >> > > > > > > > that new version of a service class was deployed.
> >> > > > > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED. *I'm
> >> not
> >> > > > sure,
> >> > > > > > > > whether it is triggered on class redeployment, though. If
> >> not,
> >> > > then
> >> > > > > > > another
> >> > > > > > > > event type should be added.
> >> > > > > > > >
> >> > > > > > > > I don't think, that a lot of people will implement their
> own
> >> > > > > > > > *DeploymentSpi*-s, so we should make work with
> >> > *UriDeploymentSpi*
> >> > > > as
> >> > > > > > > > comfortable as possible.
> >> > > > > > > >
> >> > > > > > > > Denis
> >> > > > > > > >
> >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> >> > > > > > > > [hidden email]>:
> >> > > > > > > >
> >> > > > > > > >> Yes, the class deployment itself has to be explicit.
> I.e.,
> >> > there
> >> > > > has
> >> > > > > > to
> >> > > > > > > be
> >> > > > > > > >> a manual step where user updates the class, and the exact
> >> step
> >> > > > > > required
> >> > > > > > > >> would depend on DeploymentSpi implementation. But then
> >> Ignite
> >> > > > takes
> >> > > > > > care
> >> > > > > > > >> of
> >> > > > > > > >> everything else - service redeployment and restart is
> >> > automatic.
> >> > > > > > > >>
> >> > > > > > > >> Dmitriy Pavlov, all this is going to be disabled if
> >> > > DeploymentSpi
> >> > > > is
> >> > > > > > not
> >> > > > > > > >> configured. In this case service class definitions have
> to
> >> be
> >> > > > > deployed
> >> > > > > > > on
> >> > > > > > > >> local classpath and can't be updated in runtime. Just
> like
> >> it
> >> > > > works
> >> > > > > > > right
> >> > > > > > > >> now.
> >> > > > > > > >>
> >> > > > > > > >> -Val
> >> > > > > > > >>
> >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> >> > > > > > > [hidden email]
> >> > > > > > > >> >
> >> > > > > > > >> wrote:
> >> > > > > > > >>
> >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> >> > > > > > [hidden email]>
> >> > > > > > > >> > wrote:
> >> > > > > > > >> >
> >> > > > > > > >> > > Hi Igniters,
> >> > > > > > > >> > >
> >> > > > > > > >> > > I like automatic redeploy which can be disabled by
> >> config
> >> > if
> >> > > > > user
> >> > > > > > > >> wants
> >> > > > > > > >> > to
> >> > > > > > > >> > > control this process. What do you think?
> >> > > > > > > >> > >
> >> > > > > > > >> >
> >> > > > > > > >> > I do not think we should have anything automatic when
> it
> >> > comes
> >> > > > to
> >> > > > > > > >> > deployment, everything should be explicit. However, if
> we
> >> > use
> >> > > > the
> >> > > > > > > >> > deployment SPI, then a user should be able to do "hot"
> >> > > redeploy,
> >> > > > > > > where a
> >> > > > > > > >> > new service will be deployed if the user drops an
> updated
> >> > jar.
> >> > > > > > > >> >
> >> > > > > > > >> > We should not create anything new here. Ignite already
> >> has a
> >> > > > > > > deployment
> >> > > > > > > >> SPI
> >> > > > > > > >> > and it already works in a certain way. Let's not change
> >> it.
> >> > > > > > > >> >
> >> > > > > > > >> > D.
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

[dmagda]

It sounds like it's not a trivial thing to support the automatic services
redeployment after a restart. Let's postpone it for now, guys concentrating
on more urgent things related to the services.

Alex, Vladimir,

Could you have a look at Denis question about the discovery-based
deployment? Guess it's the only one thing that prevents us from the IEP
finalization.

--
Denis

On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <[hidden email]>
wrote:

> Vladimir,
>
> Currently we don't save binary metadata to disk, when persistence is
> disabled.
> But we still persist marshaller mappings for some reason, and I personally
> believe, that we shouldn't.
>
> But I agree, that we should separate data and service persistence
> configuration.
> Right now persistence of services is configured in a pretty non-obvious
> manner.
> It should be a clear way to tell Ignite, whether you want services to be
> persisted or not.
>
> I'm not sure, that we should make "statefullness" in general configurable.
> Users don't care much, whether metadata is preserved on restarts, or not.
>
> Denis
>
> пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <[hidden email]>:
>
> > Alex,
> >
> > I would say that we've already had this behavior for years - marshaller
> > cache. I think it is time to agree that "in-memory" != stateless. Instead
> > "in-memory" means "data is not stored on disk".
> > May be we can have a flag which will wipe out all metadata on node
> restart
> > (e.g. it could make sense for embedded clients)?
> >
> > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > [hidden email]> wrote:
> >
> > > Denis,
> > >
> > > This is a subtle question. It looks like we have now a number of
> > use-cases
> > > when persistent storage is required even for a pure in-memory mode. One
> > of
> > > the use-cases is thin client authentication, the other is service grid
> > > configuration persistence.
> > >
> > > Generally, I would agree that this is an expected behavior. However,
> this
> > > means that a user cannot simply start and stop nodes randomly anymore.
> > > Ignite start will require some sort of installation or work folder
> > > initialization (sort of initdb in postgres) which is ok for
> > > persistence-enabled modes, but I am not sure if this is expected for
> > > in-memory. Of course, we can run this initialization automatically, but
> > it
> > > is not always a good idea.
> > >
> > > If we are ok to have this restrictions for in-memory mode, then service
> > > persistence makes sense.
> > >
> > > --AG
> > >
> > > 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:
> > >
> > >> Denis,
> > >>
> > >> I think that the service deployment state needs be persisted
> > cluster-wide.
> > >> I guess that our meta-store is capable of doing so. Alex G., Vladimir,
> > >> could you confirm?
> > >>
> > >> As for the split-brain scenarios, I would put them aside for now
> > because,
> > >> anyway, they have to be solved at lower levels (meta store, discovery,
> > >> etc.).
> > >>
> > >> Also, I heard that presently we store a service configuration in the
> > >> system
> > >> cache that doesn't give us a way to deploy a new version of a service
> > >> without undeployment of the previous one. Will this issue be addressed
> > by
> > >> the new deployment approach?
> > >>
> > >> --
> > >> Denis
> > >>
> > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > [hidden email]>
> > >> wrote:
> > >>
> > >> > Denis,
> > >> >
> > >> > Sounds reasonable. It's not clear, though, what should happen, if a
> > >> joining
> > >> > node has some services persisted, that are missing on other nodes.
> > >> > Should we deploy them?
> > >> > If we do so, it could lead to surprising behaviour. For example you
> > >> could
> > >> > kill a node, undeploy a service, then bring back an old node, and it
> > >> would
> > >> > make the service resurrect.
> > >> > We could store some deployment counter along with the service
> > >> > configurations on all nodes, that would show how many times the
> > service
> > >> > state has changed, i.e. it has been undeployed/redeployed. It should
> > be
> > >> > kept for undeployed services as well to avoid situations like I
> > >> described.
> > >> >
> > >> > But it still leaves a possibility of incorrect behaviour, if there
> > was a
> > >> > split-brain situation at some point. I don't think we should precess
> > it
> > >> > somehow, though. If we choose to tackle it, it will overcomplicate
> > >> things
> > >> > for a sake of a minor improvement.
> > >> >
> > >> > Denis
> > >> >
> > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > >> > [hidden email]>:
> > >> >
> > >> > > I was responding to another Denis :) Agree with you on your point
> > >> though.
> > >> > >
> > >> > > -Val
> > >> > >
> > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]>
> > >> wrote:
> > >> > >
> > >> > > > Val,
> > >> > > >
> > >> > > > Guess we're talking about other situations. I'm bringing up the
> > case
> > >> > > when a
> > >> > > > service was deployed dynamically and has to be brought up after
> a
> > >> full
> > >> > > > cluster restart w/o user intervention. To achieve this we need
> to
> > >> > persist
> > >> > > > the service's configuration somewhere.
> > >> > > >
> > >> > > > --
> > >> > > > Denis
> > >> > > >
> > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > >> > > > [hidden email]> wrote:
> > >> > > >
> > >> > > > > Denis,
> > >> > > > >
> > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a class is
> > deployed
> > >> or
> > >> > > > > redeployed. If this doesn't happen in some cases, I believe
> this
> > >> > would
> > >> > > > be a
> > >> > > > > bug. I don't think we need to add any new events.
> > >> > > > >
> > >> > > > > -Val
> > >> > > > >
> > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> [hidden email]
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Denis,
> > >> > > > > >
> > >> > > > > > I would encourage us to persist a service configuration in
> the
> > >> meta
> > >> > > > store
> > >> > > > > > and have this capability enabled by default. That's
> essential
> > >> for
> > >> > > > > services
> > >> > > > > > started dynamically. Moreover, we support similar behavior
> for
> > >> > > caches,
> > >> > > > > > indexes, and other DDL changes happened at runtime.
> > >> > > > > >
> > >> > > > > > --
> > >> > > > > > Denis
> > >> > > > > >
> > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > >> > > > [hidden email]>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Another question, that I would like to discuss is whether
> > >> > services
> > >> > > > > should
> > >> > > > > > > be preserved on cluster restarts.
> > >> > > > > > >
> > >> > > > > > > Currently it depends on persistence configuration. If
> > >> persistence
> > >> > > for
> > >> > > > > any
> > >> > > > > > > data region is enabled, then services will be persisted as
> > >> well.
> > >> > > This
> > >> > > > > is
> > >> > > > > > a
> > >> > > > > > > pretty strange way of configuring this behaviour.
> > >> > > > > > > I'm not sure, if anybody relies on this functionality
> right
> > >> now.
> > >> > > > Should
> > >> > > > > > we
> > >> > > > > > > support it at all? If yes, should we make it configurable?
> > >> > > > > > >
> > >> > > > > > > Denis
> > >> > > > > > >
> > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > >> > > [hidden email]
> > >> > > > >:
> > >> > > > > > >
> > >> > > > > > > > Val,
> > >> > > > > > > >
> > >> > > > > > > > Sounds reasonable. I just think, that user should have
> > some
> > >> way
> > >> > > to
> > >> > > > > > know,
> > >> > > > > > > > that new version of a service class was deployed.
> > >> > > > > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED.
> *I'm
> > >> not
> > >> > > > sure,
> > >> > > > > > > > whether it is triggered on class redeployment, though.
> If
> > >> not,
> > >> > > then
> > >> > > > > > > another
> > >> > > > > > > > event type should be added.
> > >> > > > > > > >
> > >> > > > > > > > I don't think, that a lot of people will implement their
> > own
> > >> > > > > > > > *DeploymentSpi*-s, so we should make work with
> > >> > *UriDeploymentSpi*
> > >> > > > as
> > >> > > > > > > > comfortable as possible.
> > >> > > > > > > >
> > >> > > > > > > > Denis
> > >> > > > > > > >
> > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > >> > > > > > > > [hidden email]>:
> > >> > > > > > > >
> > >> > > > > > > >> Yes, the class deployment itself has to be explicit.
> > I.e.,
> > >> > there
> > >> > > > has
> > >> > > > > > to
> > >> > > > > > > be
> > >> > > > > > > >> a manual step where user updates the class, and the
> exact
> > >> step
> > >> > > > > > required
> > >> > > > > > > >> would depend on DeploymentSpi implementation. But then
> > >> Ignite
> > >> > > > takes
> > >> > > > > > care
> > >> > > > > > > >> of
> > >> > > > > > > >> everything else - service redeployment and restart is
> > >> > automatic.
> > >> > > > > > > >>
> > >> > > > > > > >> Dmitriy Pavlov, all this is going to be disabled if
> > >> > > DeploymentSpi
> > >> > > > is
> > >> > > > > > not
> > >> > > > > > > >> configured. In this case service class definitions have
> > to
> > >> be
> > >> > > > > deployed
> > >> > > > > > > on
> > >> > > > > > > >> local classpath and can't be updated in runtime. Just
> > like
> > >> it
> > >> > > > works
> > >> > > > > > > right
> > >> > > > > > > >> now.
> > >> > > > > > > >>
> > >> > > > > > > >> -Val
> > >> > > > > > > >>
> > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> > >> > > > > > > [hidden email]
> > >> > > > > > > >> >
> > >> > > > > > > >> wrote:
> > >> > > > > > > >>
> > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > >> > > > > > [hidden email]>
> > >> > > > > > > >> > wrote:
> > >> > > > > > > >> >
> > >> > > > > > > >> > > Hi Igniters,
> > >> > > > > > > >> > >
> > >> > > > > > > >> > > I like automatic redeploy which can be disabled by
> > >> config
> > >> > if
> > >> > > > > user
> > >> > > > > > > >> wants
> > >> > > > > > > >> > to
> > >> > > > > > > >> > > control this process. What do you think?
> > >> > > > > > > >> > >
> > >> > > > > > > >> >
> > >> > > > > > > >> > I do not think we should have anything automatic when
> > it
> > >> > comes
> > >> > > > to
> > >> > > > > > > >> > deployment, everything should be explicit. However,
> if
> > we
> > >> > use
> > >> > > > the
> > >> > > > > > > >> > deployment SPI, then a user should be able to do
> "hot"
> > >> > > redeploy,
> > >> > > > > > > where a
> > >> > > > > > > >> > new service will be deployed if the user drops an
> > updated
> > >> > jar.
> > >> > > > > > > >> >
> > >> > > > > > > >> > We should not create anything new here. Ignite
> already
> > >> has a
> > >> > > > > > > deployment
> > >> > > > > > > >> SPI
> > >> > > > > > > >> > and it already works in a certain way. Let's not
> change
> > >> it.
> > >> > > > > > > >> >
> > >> > > > > > > >> > D.
> > >> > > > > > > >> >
> > >> > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

[Denis Mekhanikov]

Basically, my question is: at which moment services should be deployed on
connecting nodes?
Should we reject a node from being included into a topology, if services,
that are assigned to it, fail to deploy?

It would be good to be sure, that all assigned services are initialised and
working, when node start is finished.
Otherwise it's unclear, how to notify a user about failures in service
initialisation on new nodes.

If we decide to provide such guarantee, then how are we going to do that?
Is procedure, that I described, viable?
It requires hacking through the discovery protocol, which is a thing, that
should be avoided.
So, maybe there is another way to achieve the same thing?

Denis

сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:

> It sounds like it's not a trivial thing to support the automatic services
> redeployment after a restart. Let's postpone it for now, guys concentrating
> on more urgent things related to the services.
>
> Alex, Vladimir,
>
> Could you have a look at Denis question about the discovery-based
> deployment? Guess it's the only one thing that prevents us from the IEP
> finalization.
>
> --
> Denis
>
> On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <[hidden email]>
> wrote:
>
> > Vladimir,
> >
> > Currently we don't save binary metadata to disk, when persistence is
> > disabled.
> > But we still persist marshaller mappings for some reason, and I
> personally
> > believe, that we shouldn't.
> >
> > But I agree, that we should separate data and service persistence
> > configuration.
> > Right now persistence of services is configured in a pretty non-obvious
> > manner.
> > It should be a clear way to tell Ignite, whether you want services to be
> > persisted or not.
> >
> > I'm not sure, that we should make "statefullness" in general
> configurable.
> > Users don't care much, whether metadata is preserved on restarts, or not.
> >
> > Denis
> >
> > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <[hidden email]>:
> >
> > > Alex,
> > >
> > > I would say that we've already had this behavior for years - marshaller
> > > cache. I think it is time to agree that "in-memory" != stateless.
> Instead
> > > "in-memory" means "data is not stored on disk".
> > > May be we can have a flag which will wipe out all metadata on node
> > restart
> > > (e.g. it could make sense for embedded clients)?
> > >
> > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > [hidden email]> wrote:
> > >
> > > > Denis,
> > > >
> > > > This is a subtle question. It looks like we have now a number of
> > > use-cases
> > > > when persistent storage is required even for a pure in-memory mode.
> One
> > > of
> > > > the use-cases is thin client authentication, the other is service
> grid
> > > > configuration persistence.
> > > >
> > > > Generally, I would agree that this is an expected behavior. However,
> > this
> > > > means that a user cannot simply start and stop nodes randomly
> anymore.
> > > > Ignite start will require some sort of installation or work folder
> > > > initialization (sort of initdb in postgres) which is ok for
> > > > persistence-enabled modes, but I am not sure if this is expected for
> > > > in-memory. Of course, we can run this initialization automatically,
> but
> > > it
> > > > is not always a good idea.
> > > >
> > > > If we are ok to have this restrictions for in-memory mode, then
> service
> > > > persistence makes sense.
> > > >
> > > > --AG
> > > >
> > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:
> > > >
> > > >> Denis,
> > > >>
> > > >> I think that the service deployment state needs be persisted
> > > cluster-wide.
> > > >> I guess that our meta-store is capable of doing so. Alex G.,
> Vladimir,
> > > >> could you confirm?
> > > >>
> > > >> As for the split-brain scenarios, I would put them aside for now
> > > because,
> > > >> anyway, they have to be solved at lower levels (meta store,
> discovery,
> > > >> etc.).
> > > >>
> > > >> Also, I heard that presently we store a service configuration in the
> > > >> system
> > > >> cache that doesn't give us a way to deploy a new version of a
> service
> > > >> without undeployment of the previous one. Will this issue be
> addressed
> > > by
> > > >> the new deployment approach?
> > > >>
> > > >> --
> > > >> Denis
> > > >>
> > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > [hidden email]>
> > > >> wrote:
> > > >>
> > > >> > Denis,
> > > >> >
> > > >> > Sounds reasonable. It's not clear, though, what should happen, if
> a
> > > >> joining
> > > >> > node has some services persisted, that are missing on other nodes.
> > > >> > Should we deploy them?
> > > >> > If we do so, it could lead to surprising behaviour. For example
> you
> > > >> could
> > > >> > kill a node, undeploy a service, then bring back an old node, and
> it
> > > >> would
> > > >> > make the service resurrect.
> > > >> > We could store some deployment counter along with the service
> > > >> > configurations on all nodes, that would show how many times the
> > > service
> > > >> > state has changed, i.e. it has been undeployed/redeployed. It
> should
> > > be
> > > >> > kept for undeployed services as well to avoid situations like I
> > > >> described.
> > > >> >
> > > >> > But it still leaves a possibility of incorrect behaviour, if there
> > > was a
> > > >> > split-brain situation at some point. I don't think we should
> precess
> > > it
> > > >> > somehow, though. If we choose to tackle it, it will overcomplicate
> > > >> things
> > > >> > for a sake of a minor improvement.
> > > >> >
> > > >> > Denis
> > > >> >
> > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > >> > [hidden email]>:
> > > >> >
> > > >> > > I was responding to another Denis :) Agree with you on your
> point
> > > >> though.
> > > >> > >
> > > >> > > -Val
> > > >> > >
> > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <[hidden email]>
> > > >> wrote:
> > > >> > >
> > > >> > > > Val,
> > > >> > > >
> > > >> > > > Guess we're talking about other situations. I'm bringing up
> the
> > > case
> > > >> > > when a
> > > >> > > > service was deployed dynamically and has to be brought up
> after
> > a
> > > >> full
> > > >> > > > cluster restart w/o user intervention. To achieve this we need
> > to
> > > >> > persist
> > > >> > > > the service's configuration somewhere.
> > > >> > > >
> > > >> > > > --
> > > >> > > > Denis
> > > >> > > >
> > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > > >> > > > [hidden email]> wrote:
> > > >> > > >
> > > >> > > > > Denis,
> > > >> > > > >
> > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a class is
> > > deployed
> > > >> or
> > > >> > > > > redeployed. If this doesn't happen in some cases, I believe
> > this
> > > >> > would
> > > >> > > > be a
> > > >> > > > > bug. I don't think we need to add any new events.
> > > >> > > > >
> > > >> > > > > -Val
> > > >> > > > >
> > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > [hidden email]
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Denis,
> > > >> > > > > >
> > > >> > > > > > I would encourage us to persist a service configuration in
> > the
> > > >> meta
> > > >> > > > store
> > > >> > > > > > and have this capability enabled by default. That's
> > essential
> > > >> for
> > > >> > > > > services
> > > >> > > > > > started dynamically. Moreover, we support similar behavior
> > for
> > > >> > > caches,
> > > >> > > > > > indexes, and other DDL changes happened at runtime.
> > > >> > > > > >
> > > >> > > > > > --
> > > >> > > > > > Denis
> > > >> > > > > >
> > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > > >> > > > [hidden email]>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Another question, that I would like to discuss is
> whether
> > > >> > services
> > > >> > > > > should
> > > >> > > > > > > be preserved on cluster restarts.
> > > >> > > > > > >
> > > >> > > > > > > Currently it depends on persistence configuration. If
> > > >> persistence
> > > >> > > for
> > > >> > > > > any
> > > >> > > > > > > data region is enabled, then services will be persisted
> as
> > > >> well.
> > > >> > > This
> > > >> > > > > is
> > > >> > > > > > a
> > > >> > > > > > > pretty strange way of configuring this behaviour.
> > > >> > > > > > > I'm not sure, if anybody relies on this functionality
> > right
> > > >> now.
> > > >> > > > Should
> > > >> > > > > > we
> > > >> > > > > > > support it at all? If yes, should we make it
> configurable?
> > > >> > > > > > >
> > > >> > > > > > > Denis
> > > >> > > > > > >
> > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > > >> > > [hidden email]
> > > >> > > > >:
> > > >> > > > > > >
> > > >> > > > > > > > Val,
> > > >> > > > > > > >
> > > >> > > > > > > > Sounds reasonable. I just think, that user should have
> > > some
> > > >> way
> > > >> > > to
> > > >> > > > > > know,
> > > >> > > > > > > > that new version of a service class was deployed.
> > > >> > > > > > > > One way to do it is to listen to *EVT_CLASS_DEPLOYED.
> > *I'm
> > > >> not
> > > >> > > > sure,
> > > >> > > > > > > > whether it is triggered on class redeployment, though.
> > If
> > > >> not,
> > > >> > > then
> > > >> > > > > > > another
> > > >> > > > > > > > event type should be added.
> > > >> > > > > > > >
> > > >> > > > > > > > I don't think, that a lot of people will implement
> their
> > > own
> > > >> > > > > > > > *DeploymentSpi*-s, so we should make work with
> > > >> > *UriDeploymentSpi*
> > > >> > > > as
> > > >> > > > > > > > comfortable as possible.
> > > >> > > > > > > >
> > > >> > > > > > > > Denis
> > > >> > > > > > > >
> > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > >> > > > > > > > [hidden email]>:
> > > >> > > > > > > >
> > > >> > > > > > > >> Yes, the class deployment itself has to be explicit.
> > > I.e.,
> > > >> > there
> > > >> > > > has
> > > >> > > > > > to
> > > >> > > > > > > be
> > > >> > > > > > > >> a manual step where user updates the class, and the
> > exact
> > > >> step
> > > >> > > > > > required
> > > >> > > > > > > >> would depend on DeploymentSpi implementation. But
> then
> > > >> Ignite
> > > >> > > > takes
> > > >> > > > > > care
> > > >> > > > > > > >> of
> > > >> > > > > > > >> everything else - service redeployment and restart is
> > > >> > automatic.
> > > >> > > > > > > >>
> > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be disabled if
> > > >> > > DeploymentSpi
> > > >> > > > is
> > > >> > > > > > not
> > > >> > > > > > > >> configured. In this case service class definitions
> have
> > > to
> > > >> be
> > > >> > > > > deployed
> > > >> > > > > > > on
> > > >> > > > > > > >> local classpath and can't be updated in runtime. Just
> > > like
> > > >> it
> > > >> > > > works
> > > >> > > > > > > right
> > > >> > > > > > > >> now.
> > > >> > > > > > > >>
> > > >> > > > > > > >> -Val
> > > >> > > > > > > >>
> > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan <
> > > >> > > > > > > [hidden email]
> > > >> > > > > > > >> >
> > > >> > > > > > > >> wrote:
> > > >> > > > > > > >>
> > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > > >> > > > > > [hidden email]>
> > > >> > > > > > > >> > wrote:
> > > >> > > > > > > >> >
> > > >> > > > > > > >> > > Hi Igniters,
> > > >> > > > > > > >> > >
> > > >> > > > > > > >> > > I like automatic redeploy which can be disabled
> by
> > > >> config
> > > >> > if
> > > >> > > > > user
> > > >> > > > > > > >> wants
> > > >> > > > > > > >> > to
> > > >> > > > > > > >> > > control this process. What do you think?
> > > >> > > > > > > >> > >
> > > >> > > > > > > >> >
> > > >> > > > > > > >> > I do not think we should have anything automatic
> when
> > > it
> > > >> > comes
> > > >> > > > to
> > > >> > > > > > > >> > deployment, everything should be explicit. However,
> > if
> > > we
> > > >> > use
> > > >> > > > the
> > > >> > > > > > > >> > deployment SPI, then a user should be able to do
> > "hot"
> > > >> > > redeploy,
> > > >> > > > > > > where a
> > > >> > > > > > > >> > new service will be deployed if the user drops an
> > > updated
> > > >> > jar.
> > > >> > > > > > > >> >
> > > >> > > > > > > >> > We should not create anything new here. Ignite
> > already
> > > >> has a
> > > >> > > > > > > deployment
> > > >> > > > > > > >> SPI
> > > >> > > > > > > >> > and it already works in a certain way. Let's not
> > change
> > > >> it.
> > > >> > > > > > > >> >
> > > >> > > > > > > >> > D.
> > > >> > > > > > > >> >
> > > >> > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

[dmagda]

Denis,

In general, service initialization shouldn't prevent a node from joining
the cluster or slowing down that process. Thus, I would start the
initialization routines only after the node is accepted by the cluster. If
the initialization fails then we need to report a respective message to the
logs and, probably, generate a system event the user can be subscribed to.

Regardless, of the service initialization time, I think we still need to
utilize discovery SPI to avoid problems discussed later.

Val, others, what do you think about that?


--
Denis

On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <[hidden email]>
wrote:

> Basically, my question is: at which moment services should be deployed on
> connecting nodes?
> Should we reject a node from being included into a topology, if services,
> that are assigned to it, fail to deploy?
>
> It would be good to be sure, that all assigned services are initialised and
> working, when node start is finished.
> Otherwise it's unclear, how to notify a user about failures in service
> initialisation on new nodes.
>
> If we decide to provide such guarantee, then how are we going to do that?
> Is procedure, that I described, viable?
> It requires hacking through the discovery protocol, which is a thing, that
> should be avoided.
> So, maybe there is another way to achieve the same thing?
>
> Denis
>
> сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:
>
> > It sounds like it's not a trivial thing to support the automatic services
> > redeployment after a restart. Let's postpone it for now, guys
> concentrating
> > on more urgent things related to the services.
> >
> > Alex, Vladimir,
> >
> > Could you have a look at Denis question about the discovery-based
> > deployment? Guess it's the only one thing that prevents us from the IEP
> > finalization.
> >
> > --
> > Denis
> >
> > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <[hidden email]
> >
> > wrote:
> >
> > > Vladimir,
> > >
> > > Currently we don't save binary metadata to disk, when persistence is
> > > disabled.
> > > But we still persist marshaller mappings for some reason, and I
> > personally
> > > believe, that we shouldn't.
> > >
> > > But I agree, that we should separate data and service persistence
> > > configuration.
> > > Right now persistence of services is configured in a pretty non-obvious
> > > manner.
> > > It should be a clear way to tell Ignite, whether you want services to
> be
> > > persisted or not.
> > >
> > > I'm not sure, that we should make "statefullness" in general
> > configurable.
> > > Users don't care much, whether metadata is preserved on restarts, or
> not.
> > >
> > > Denis
> > >
> > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <[hidden email]>:
> > >
> > > > Alex,
> > > >
> > > > I would say that we've already had this behavior for years -
> marshaller
> > > > cache. I think it is time to agree that "in-memory" != stateless.
> > Instead
> > > > "in-memory" means "data is not stored on disk".
> > > > May be we can have a flag which will wipe out all metadata on node
> > > restart
> > > > (e.g. it could make sense for embedded clients)?
> > > >
> > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > > [hidden email]> wrote:
> > > >
> > > > > Denis,
> > > > >
> > > > > This is a subtle question. It looks like we have now a number of
> > > > use-cases
> > > > > when persistent storage is required even for a pure in-memory mode.
> > One
> > > > of
> > > > > the use-cases is thin client authentication, the other is service
> > grid
> > > > > configuration persistence.
> > > > >
> > > > > Generally, I would agree that this is an expected behavior.
> However,
> > > this
> > > > > means that a user cannot simply start and stop nodes randomly
> > anymore.
> > > > > Ignite start will require some sort of installation or work folder
> > > > > initialization (sort of initdb in postgres) which is ok for
> > > > > persistence-enabled modes, but I am not sure if this is expected
> for
> > > > > in-memory. Of course, we can run this initialization automatically,
> > but
> > > > it
> > > > > is not always a good idea.
> > > > >
> > > > > If we are ok to have this restrictions for in-memory mode, then
> > service
> > > > > persistence makes sense.
> > > > >
> > > > > --AG
> > > > >
> > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:
> > > > >
> > > > >> Denis,
> > > > >>
> > > > >> I think that the service deployment state needs be persisted
> > > > cluster-wide.
> > > > >> I guess that our meta-store is capable of doing so. Alex G.,
> > Vladimir,
> > > > >> could you confirm?
> > > > >>
> > > > >> As for the split-brain scenarios, I would put them aside for now
> > > > because,
> > > > >> anyway, they have to be solved at lower levels (meta store,
> > discovery,
> > > > >> etc.).
> > > > >>
> > > > >> Also, I heard that presently we store a service configuration in
> the
> > > > >> system
> > > > >> cache that doesn't give us a way to deploy a new version of a
> > service
> > > > >> without undeployment of the previous one. Will this issue be
> > addressed
> > > > by
> > > > >> the new deployment approach?
> > > > >>
> > > > >> --
> > > > >> Denis
> > > > >>
> > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > > [hidden email]>
> > > > >> wrote:
> > > > >>
> > > > >> > Denis,
> > > > >> >
> > > > >> > Sounds reasonable. It's not clear, though, what should happen,
> if
> > a
> > > > >> joining
> > > > >> > node has some services persisted, that are missing on other
> nodes.
> > > > >> > Should we deploy them?
> > > > >> > If we do so, it could lead to surprising behaviour. For example
> > you
> > > > >> could
> > > > >> > kill a node, undeploy a service, then bring back an old node,
> and
> > it
> > > > >> would
> > > > >> > make the service resurrect.
> > > > >> > We could store some deployment counter along with the service
> > > > >> > configurations on all nodes, that would show how many times the
> > > > service
> > > > >> > state has changed, i.e. it has been undeployed/redeployed. It
> > should
> > > > be
> > > > >> > kept for undeployed services as well to avoid situations like I
> > > > >> described.
> > > > >> >
> > > > >> > But it still leaves a possibility of incorrect behaviour, if
> there
> > > > was a
> > > > >> > split-brain situation at some point. I don't think we should
> > precess
> > > > it
> > > > >> > somehow, though. If we choose to tackle it, it will
> overcomplicate
> > > > >> things
> > > > >> > for a sake of a minor improvement.
> > > > >> >
> > > > >> > Denis
> > > > >> >
> > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > >> > [hidden email]>:
> > > > >> >
> > > > >> > > I was responding to another Denis :) Agree with you on your
> > point
> > > > >> though.
> > > > >> > >
> > > > >> > > -Val
> > > > >> > >
> > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> [hidden email]>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Val,
> > > > >> > > >
> > > > >> > > > Guess we're talking about other situations. I'm bringing up
> > the
> > > > case
> > > > >> > > when a
> > > > >> > > > service was deployed dynamically and has to be brought up
> > after
> > > a
> > > > >> full
> > > > >> > > > cluster restart w/o user intervention. To achieve this we
> need
> > > to
> > > > >> > persist
> > > > >> > > > the service's configuration somewhere.
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > > Denis
> > > > >> > > >
> > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > > > >> > > > [hidden email]> wrote:
> > > > >> > > >
> > > > >> > > > > Denis,
> > > > >> > > > >
> > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a class is
> > > > deployed
> > > > >> or
> > > > >> > > > > redeployed. If this doesn't happen in some cases, I
> believe
> > > this
> > > > >> > would
> > > > >> > > > be a
> > > > >> > > > > bug. I don't think we need to add any new events.
> > > > >> > > > >
> > > > >> > > > > -Val
> > > > >> > > > >
> > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > > [hidden email]
> > > > >
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Denis,
> > > > >> > > > > >
> > > > >> > > > > > I would encourage us to persist a service configuration
> in
> > > the
> > > > >> meta
> > > > >> > > > store
> > > > >> > > > > > and have this capability enabled by default. That's
> > > essential
> > > > >> for
> > > > >> > > > > services
> > > > >> > > > > > started dynamically. Moreover, we support similar
> behavior
> > > for
> > > > >> > > caches,
> > > > >> > > > > > indexes, and other DDL changes happened at runtime.
> > > > >> > > > > >
> > > > >> > > > > > --
> > > > >> > > > > > Denis
> > > > >> > > > > >
> > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > > > >> > > > [hidden email]>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Another question, that I would like to discuss is
> > whether
> > > > >> > services
> > > > >> > > > > should
> > > > >> > > > > > > be preserved on cluster restarts.
> > > > >> > > > > > >
> > > > >> > > > > > > Currently it depends on persistence configuration. If
> > > > >> persistence
> > > > >> > > for
> > > > >> > > > > any
> > > > >> > > > > > > data region is enabled, then services will be
> persisted
> > as
> > > > >> well.
> > > > >> > > This
> > > > >> > > > > is
> > > > >> > > > > > a
> > > > >> > > > > > > pretty strange way of configuring this behaviour.
> > > > >> > > > > > > I'm not sure, if anybody relies on this functionality
> > > right
> > > > >> now.
> > > > >> > > > Should
> > > > >> > > > > > we
> > > > >> > > > > > > support it at all? If yes, should we make it
> > configurable?
> > > > >> > > > > > >
> > > > >> > > > > > > Denis
> > > > >> > > > > > >
> > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > > > >> > > [hidden email]
> > > > >> > > > >:
> > > > >> > > > > > >
> > > > >> > > > > > > > Val,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Sounds reasonable. I just think, that user should
> have
> > > > some
> > > > >> way
> > > > >> > > to
> > > > >> > > > > > know,
> > > > >> > > > > > > > that new version of a service class was deployed.
> > > > >> > > > > > > > One way to do it is to listen to
> *EVT_CLASS_DEPLOYED.
> > > *I'm
> > > > >> not
> > > > >> > > > sure,
> > > > >> > > > > > > > whether it is triggered on class redeployment,
> though.
> > > If
> > > > >> not,
> > > > >> > > then
> > > > >> > > > > > > another
> > > > >> > > > > > > > event type should be added.
> > > > >> > > > > > > >
> > > > >> > > > > > > > I don't think, that a lot of people will implement
> > their
> > > > own
> > > > >> > > > > > > > *DeploymentSpi*-s, so we should make work with
> > > > >> > *UriDeploymentSpi*
> > > > >> > > > as
> > > > >> > > > > > > > comfortable as possible.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Denis
> > > > >> > > > > > > >
> > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > > >> > > > > > > > [hidden email]>:
> > > > >> > > > > > > >
> > > > >> > > > > > > >> Yes, the class deployment itself has to be
> explicit.
> > > > I.e.,
> > > > >> > there
> > > > >> > > > has
> > > > >> > > > > > to
> > > > >> > > > > > > be
> > > > >> > > > > > > >> a manual step where user updates the class, and the
> > > exact
> > > > >> step
> > > > >> > > > > > required
> > > > >> > > > > > > >> would depend on DeploymentSpi implementation. But
> > then
> > > > >> Ignite
> > > > >> > > > takes
> > > > >> > > > > > care
> > > > >> > > > > > > >> of
> > > > >> > > > > > > >> everything else - service redeployment and restart
> is
> > > > >> > automatic.
> > > > >> > > > > > > >>
> > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be disabled if
> > > > >> > > DeploymentSpi
> > > > >> > > > is
> > > > >> > > > > > not
> > > > >> > > > > > > >> configured. In this case service class definitions
> > have
> > > > to
> > > > >> be
> > > > >> > > > > deployed
> > > > >> > > > > > > on
> > > > >> > > > > > > >> local classpath and can't be updated in runtime.
> Just
> > > > like
> > > > >> it
> > > > >> > > > works
> > > > >> > > > > > > right
> > > > >> > > > > > > >> now.
> > > > >> > > > > > > >>
> > > > >> > > > > > > >> -Val
> > > > >> > > > > > > >>
> > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan
> <
> > > > >> > > > > > > [hidden email]
> > > > >> > > > > > > >> >
> > > > >> > > > > > > >> wrote:
> > > > >> > > > > > > >>
> > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > > > >> > > > > > [hidden email]>
> > > > >> > > > > > > >> > wrote:
> > > > >> > > > > > > >> >
> > > > >> > > > > > > >> > > Hi Igniters,
> > > > >> > > > > > > >> > >
> > > > >> > > > > > > >> > > I like automatic redeploy which can be disabled
> > by
> > > > >> config
> > > > >> > if
> > > > >> > > > > user
> > > > >> > > > > > > >> wants
> > > > >> > > > > > > >> > to
> > > > >> > > > > > > >> > > control this process. What do you think?
> > > > >> > > > > > > >> > >
> > > > >> > > > > > > >> >
> > > > >> > > > > > > >> > I do not think we should have anything automatic
> > when
> > > > it
> > > > >> > comes
> > > > >> > > > to
> > > > >> > > > > > > >> > deployment, everything should be explicit.
> However,
> > > if
> > > > we
> > > > >> > use
> > > > >> > > > the
> > > > >> > > > > > > >> > deployment SPI, then a user should be able to do
> > > "hot"
> > > > >> > > redeploy,
> > > > >> > > > > > > where a
> > > > >> > > > > > > >> > new service will be deployed if the user drops an
> > > > updated
> > > > >> > jar.
> > > > >> > > > > > > >> >
> > > > >> > > > > > > >> > We should not create anything new here. Ignite
> > > already
> > > > >> has a
> > > > >> > > > > > > deployment
> > > > >> > > > > > > >> SPI
> > > > >> > > > > > > >> > and it already works in a certain way. Let's not
> > > change
> > > > >> it.
> > > > >> > > > > > > >> >
> > > > >> > > > > > > >> > D.
> > > > >> > > > > > > >> >
> > > > >> > > > > > > >>
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

[Valentin Kulichenko]

I agree we shouldn't do anything synchronously within discovery threads. If
something goes wrong, we just need to properly notify the user, logging and
events seem to be right options to achieve that.

BTW, with this design I'm not sure init() method makes sense, probably we
should deprecate it.

-Val

On Mon, Apr 16, 2018 at 12:03 PM, Denis Magda <[hidden email]> wrote:

> Denis,
>
> In general, service initialization shouldn't prevent a node from joining
> the cluster or slowing down that process. Thus, I would start the
> initialization routines only after the node is accepted by the cluster. If
> the initialization fails then we need to report a respective message to the
> logs and, probably, generate a system event the user can be subscribed to.
>
> Regardless, of the service initialization time, I think we still need to
> utilize discovery SPI to avoid problems discussed later.
>
> Val, others, what do you think about that?
>
>
> --
> Denis
>
> On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <[hidden email]>
> wrote:
>
> > Basically, my question is: at which moment services should be deployed on
> > connecting nodes?
> > Should we reject a node from being included into a topology, if services,
> > that are assigned to it, fail to deploy?
> >
> > It would be good to be sure, that all assigned services are initialised
> and
> > working, when node start is finished.
> > Otherwise it's unclear, how to notify a user about failures in service
> > initialisation on new nodes.
> >
> > If we decide to provide such guarantee, then how are we going to do that?
> > Is procedure, that I described, viable?
> > It requires hacking through the discovery protocol, which is a thing,
> that
> > should be avoided.
> > So, maybe there is another way to achieve the same thing?
> >
> > Denis
> >
> > сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:
> >
> > > It sounds like it's not a trivial thing to support the automatic
> services
> > > redeployment after a restart. Let's postpone it for now, guys
> > concentrating
> > > on more urgent things related to the services.
> > >
> > > Alex, Vladimir,
> > >
> > > Could you have a look at Denis question about the discovery-based
> > > deployment? Guess it's the only one thing that prevents us from the IEP
> > > finalization.
> > >
> > > --
> > > Denis
> > >
> > > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Vladimir,
> > > >
> > > > Currently we don't save binary metadata to disk, when persistence is
> > > > disabled.
> > > > But we still persist marshaller mappings for some reason, and I
> > > personally
> > > > believe, that we shouldn't.
> > > >
> > > > But I agree, that we should separate data and service persistence
> > > > configuration.
> > > > Right now persistence of services is configured in a pretty
> non-obvious
> > > > manner.
> > > > It should be a clear way to tell Ignite, whether you want services to
> > be
> > > > persisted or not.
> > > >
> > > > I'm not sure, that we should make "statefullness" in general
> > > configurable.
> > > > Users don't care much, whether metadata is preserved on restarts, or
> > not.
> > > >
> > > > Denis
> > > >
> > > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <[hidden email]>:
> > > >
> > > > > Alex,
> > > > >
> > > > > I would say that we've already had this behavior for years -
> > marshaller
> > > > > cache. I think it is time to agree that "in-memory" != stateless.
> > > Instead
> > > > > "in-memory" means "data is not stored on disk".
> > > > > May be we can have a flag which will wipe out all metadata on node
> > > > restart
> > > > > (e.g. it could make sense for embedded clients)?
> > > > >
> > > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > > > [hidden email]> wrote:
> > > > >
> > > > > > Denis,
> > > > > >
> > > > > > This is a subtle question. It looks like we have now a number of
> > > > > use-cases
> > > > > > when persistent storage is required even for a pure in-memory
> mode.
> > > One
> > > > > of
> > > > > > the use-cases is thin client authentication, the other is service
> > > grid
> > > > > > configuration persistence.
> > > > > >
> > > > > > Generally, I would agree that this is an expected behavior.
> > However,
> > > > this
> > > > > > means that a user cannot simply start and stop nodes randomly
> > > anymore.
> > > > > > Ignite start will require some sort of installation or work
> folder
> > > > > > initialization (sort of initdb in postgres) which is ok for
> > > > > > persistence-enabled modes, but I am not sure if this is expected
> > for
> > > > > > in-memory. Of course, we can run this initialization
> automatically,
> > > but
> > > > > it
> > > > > > is not always a good idea.
> > > > > >
> > > > > > If we are ok to have this restrictions for in-memory mode, then
> > > service
> > > > > > persistence makes sense.
> > > > > >
> > > > > > --AG
> > > > > >
> > > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:
> > > > > >
> > > > > >> Denis,
> > > > > >>
> > > > > >> I think that the service deployment state needs be persisted
> > > > > cluster-wide.
> > > > > >> I guess that our meta-store is capable of doing so. Alex G.,
> > > Vladimir,
> > > > > >> could you confirm?
> > > > > >>
> > > > > >> As for the split-brain scenarios, I would put them aside for now
> > > > > because,
> > > > > >> anyway, they have to be solved at lower levels (meta store,
> > > discovery,
> > > > > >> etc.).
> > > > > >>
> > > > > >> Also, I heard that presently we store a service configuration in
> > the
> > > > > >> system
> > > > > >> cache that doesn't give us a way to deploy a new version of a
> > > service
> > > > > >> without undeployment of the previous one. Will this issue be
> > > addressed
> > > > > by
> > > > > >> the new deployment approach?
> > > > > >>
> > > > > >> --
> > > > > >> Denis
> > > > > >>
> > > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > > > [hidden email]>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Denis,
> > > > > >> >
> > > > > >> > Sounds reasonable. It's not clear, though, what should happen,
> > if
> > > a
> > > > > >> joining
> > > > > >> > node has some services persisted, that are missing on other
> > nodes.
> > > > > >> > Should we deploy them?
> > > > > >> > If we do so, it could lead to surprising behaviour. For
> example
> > > you
> > > > > >> could
> > > > > >> > kill a node, undeploy a service, then bring back an old node,
> > and
> > > it
> > > > > >> would
> > > > > >> > make the service resurrect.
> > > > > >> > We could store some deployment counter along with the service
> > > > > >> > configurations on all nodes, that would show how many times
> the
> > > > > service
> > > > > >> > state has changed, i.e. it has been undeployed/redeployed. It
> > > should
> > > > > be
> > > > > >> > kept for undeployed services as well to avoid situations like
> I
> > > > > >> described.
> > > > > >> >
> > > > > >> > But it still leaves a possibility of incorrect behaviour, if
> > there
> > > > > was a
> > > > > >> > split-brain situation at some point. I don't think we should
> > > precess
> > > > > it
> > > > > >> > somehow, though. If we choose to tackle it, it will
> > overcomplicate
> > > > > >> things
> > > > > >> > for a sake of a minor improvement.
> > > > > >> >
> > > > > >> > Denis
> > > > > >> >
> > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > > >> > [hidden email]>:
> > > > > >> >
> > > > > >> > > I was responding to another Denis :) Agree with you on your
> > > point
> > > > > >> though.
> > > > > >> > >
> > > > > >> > > -Val
> > > > > >> > >
> > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> > [hidden email]>
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > Val,
> > > > > >> > > >
> > > > > >> > > > Guess we're talking about other situations. I'm bringing
> up
> > > the
> > > > > case
> > > > > >> > > when a
> > > > > >> > > > service was deployed dynamically and has to be brought up
> > > after
> > > > a
> > > > > >> full
> > > > > >> > > > cluster restart w/o user intervention. To achieve this we
> > need
> > > > to
> > > > > >> > persist
> > > > > >> > > > the service's configuration somewhere.
> > > > > >> > > >
> > > > > >> > > > --
> > > > > >> > > > Denis
> > > > > >> > > >
> > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > > > > >> > > > [hidden email]> wrote:
> > > > > >> > > >
> > > > > >> > > > > Denis,
> > > > > >> > > > >
> > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a class is
> > > > > deployed
> > > > > >> or
> > > > > >> > > > > redeployed. If this doesn't happen in some cases, I
> > believe
> > > > this
> > > > > >> > would
> > > > > >> > > > be a
> > > > > >> > > > > bug. I don't think we need to add any new events.
> > > > > >> > > > >
> > > > > >> > > > > -Val
> > > > > >> > > > >
> > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > > > [hidden email]
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Denis,
> > > > > >> > > > > >
> > > > > >> > > > > > I would encourage us to persist a service
> configuration
> > in
> > > > the
> > > > > >> meta
> > > > > >> > > > store
> > > > > >> > > > > > and have this capability enabled by default. That's
> > > > essential
> > > > > >> for
> > > > > >> > > > > services
> > > > > >> > > > > > started dynamically. Moreover, we support similar
> > behavior
> > > > for
> > > > > >> > > caches,
> > > > > >> > > > > > indexes, and other DDL changes happened at runtime.
> > > > > >> > > > > >
> > > > > >> > > > > > --
> > > > > >> > > > > > Denis
> > > > > >> > > > > >
> > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > > > > >> > > > [hidden email]>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Another question, that I would like to discuss is
> > > whether
> > > > > >> > services
> > > > > >> > > > > should
> > > > > >> > > > > > > be preserved on cluster restarts.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Currently it depends on persistence configuration.
> If
> > > > > >> persistence
> > > > > >> > > for
> > > > > >> > > > > any
> > > > > >> > > > > > > data region is enabled, then services will be
> > persisted
> > > as
> > > > > >> well.
> > > > > >> > > This
> > > > > >> > > > > is
> > > > > >> > > > > > a
> > > > > >> > > > > > > pretty strange way of configuring this behaviour.
> > > > > >> > > > > > > I'm not sure, if anybody relies on this
> functionality
> > > > right
> > > > > >> now.
> > > > > >> > > > Should
> > > > > >> > > > > > we
> > > > > >> > > > > > > support it at all? If yes, should we make it
> > > configurable?
> > > > > >> > > > > > >
> > > > > >> > > > > > > Denis
> > > > > >> > > > > > >
> > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > > > > >> > > [hidden email]
> > > > > >> > > > >:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Val,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Sounds reasonable. I just think, that user should
> > have
> > > > > some
> > > > > >> way
> > > > > >> > > to
> > > > > >> > > > > > know,
> > > > > >> > > > > > > > that new version of a service class was deployed.
> > > > > >> > > > > > > > One way to do it is to listen to
> > *EVT_CLASS_DEPLOYED.
> > > > *I'm
> > > > > >> not
> > > > > >> > > > sure,
> > > > > >> > > > > > > > whether it is triggered on class redeployment,
> > though.
> > > > If
> > > > > >> not,
> > > > > >> > > then
> > > > > >> > > > > > > another
> > > > > >> > > > > > > > event type should be added.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I don't think, that a lot of people will implement
> > > their
> > > > > own
> > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make work with
> > > > > >> > *UriDeploymentSpi*
> > > > > >> > > > as
> > > > > >> > > > > > > > comfortable as possible.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Denis
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko <
> > > > > >> > > > > > > > [hidden email]>:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >> Yes, the class deployment itself has to be
> > explicit.
> > > > > I.e.,
> > > > > >> > there
> > > > > >> > > > has
> > > > > >> > > > > > to
> > > > > >> > > > > > > be
> > > > > >> > > > > > > >> a manual step where user updates the class, and
> the
> > > > exact
> > > > > >> step
> > > > > >> > > > > > required
> > > > > >> > > > > > > >> would depend on DeploymentSpi implementation. But
> > > then
> > > > > >> Ignite
> > > > > >> > > > takes
> > > > > >> > > > > > care
> > > > > >> > > > > > > >> of
> > > > > >> > > > > > > >> everything else - service redeployment and
> restart
> > is
> > > > > >> > automatic.
> > > > > >> > > > > > > >>
> > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be disabled
> if
> > > > > >> > > DeploymentSpi
> > > > > >> > > > is
> > > > > >> > > > > > not
> > > > > >> > > > > > > >> configured. In this case service class
> definitions
> > > have
> > > > > to
> > > > > >> be
> > > > > >> > > > > deployed
> > > > > >> > > > > > > on
> > > > > >> > > > > > > >> local classpath and can't be updated in runtime.
> > Just
> > > > > like
> > > > > >> it
> > > > > >> > > > works
> > > > > >> > > > > > > right
> > > > > >> > > > > > > >> now.
> > > > > >> > > > > > > >>
> > > > > >> > > > > > > >> -Val
> > > > > >> > > > > > > >>
> > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy
> Setrakyan
> > <
> > > > > >> > > > > > > [hidden email]
> > > > > >> > > > > > > >> >
> > > > > >> > > > > > > >> wrote:
> > > > > >> > > > > > > >>
> > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov <
> > > > > >> > > > > > [hidden email]>
> > > > > >> > > > > > > >> > wrote:
> > > > > >> > > > > > > >> >
> > > > > >> > > > > > > >> > > Hi Igniters,
> > > > > >> > > > > > > >> > >
> > > > > >> > > > > > > >> > > I like automatic redeploy which can be
> disabled
> > > by
> > > > > >> config
> > > > > >> > if
> > > > > >> > > > > user
> > > > > >> > > > > > > >> wants
> > > > > >> > > > > > > >> > to
> > > > > >> > > > > > > >> > > control this process. What do you think?
> > > > > >> > > > > > > >> > >
> > > > > >> > > > > > > >> >
> > > > > >> > > > > > > >> > I do not think we should have anything
> automatic
> > > when
> > > > > it
> > > > > >> > comes
> > > > > >> > > > to
> > > > > >> > > > > > > >> > deployment, everything should be explicit.
> > However,
> > > > if
> > > > > we
> > > > > >> > use
> > > > > >> > > > the
> > > > > >> > > > > > > >> > deployment SPI, then a user should be able to
> do
> > > > "hot"
> > > > > >> > > redeploy,
> > > > > >> > > > > > > where a
> > > > > >> > > > > > > >> > new service will be deployed if the user drops
> an
> > > > > updated
> > > > > >> > jar.
> > > > > >> > > > > > > >> >
> > > > > >> > > > > > > >> > We should not create anything new here. Ignite
> > > > already
> > > > > >> has a
> > > > > >> > > > > > > deployment
> > > > > >> > > > > > > >> SPI
> > > > > >> > > > > > > >> > and it already works in a certain way. Let's
> not
> > > > change
> > > > > >> it.
> > > > > >> > > > > > > >> >
> > > > > >> > > > > > > >> > D.
> > > > > >> > > > > > > >> >
> > > > > >> > > > > > > >>
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Denis Mekhanikov]

Val,

Service initialisation is not going to happen in the discovery thread.
It should be done asynchronously, and initialisation results should be sent
to the coordinator over communication.
This is described in the IEP:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-Successfulscenario

*init()* method is a validation step, making sure, that service is ready
for work.
And deployment shouldn't be considered successful until *init()* methods
finish their work on the assigned nodes.
Also *cancel() *and *init() *methods may be useful if we decide to
implement moving existing services to new nodes
<https://cwiki.apache.org/confluence/display/IGNITE/IEP-17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-Movingexistingservicestonewnodes>
 in
future.
These methods can be used to save and restore service's state from cache,
when it is rebalanced to another node.

As Denis said, if we are not going to prevent nodes from starting on
service failures, then we should at least generate corresponding events.
Otherwise there won't be any way to react to service initialization
failures during nodes startup.

Denis

вт, 17 апр. 2018 г. в 6:59, Valentin Kulichenko <
[hidden email]>:

> I agree we shouldn't do anything synchronously within discovery threads. If
> something goes wrong, we just need to properly notify the user, logging and
> events seem to be right options to achieve that.
>
> BTW, with this design I'm not sure init() method makes sense, probably we
> should deprecate it.
>
> -Val
>
> On Mon, Apr 16, 2018 at 12:03 PM, Denis Magda <[hidden email]> wrote:
>
> > Denis,
> >
> > In general, service initialization shouldn't prevent a node from joining
> > the cluster or slowing down that process. Thus, I would start the
> > initialization routines only after the node is accepted by the cluster.
> If
> > the initialization fails then we need to report a respective message to
> the
> > logs and, probably, generate a system event the user can be subscribed
> to.
> >
> > Regardless, of the service initialization time, I think we still need to
> > utilize discovery SPI to avoid problems discussed later.
> >
> > Val, others, what do you think about that?
> >
> >
> > --
> > Denis
> >
> > On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <
> [hidden email]>
> > wrote:
> >
> > > Basically, my question is: at which moment services should be deployed
> on
> > > connecting nodes?
> > > Should we reject a node from being included into a topology, if
> services,
> > > that are assigned to it, fail to deploy?
> > >
> > > It would be good to be sure, that all assigned services are initialised
> > and
> > > working, when node start is finished.
> > > Otherwise it's unclear, how to notify a user about failures in service
> > > initialisation on new nodes.
> > >
> > > If we decide to provide such guarantee, then how are we going to do
> that?
> > > Is procedure, that I described, viable?
> > > It requires hacking through the discovery protocol, which is a thing,
> > that
> > > should be avoided.
> > > So, maybe there is another way to achieve the same thing?
> > >
> > > Denis
> > >
> > > сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:
> > >
> > > > It sounds like it's not a trivial thing to support the automatic
> > services
> > > > redeployment after a restart. Let's postpone it for now, guys
> > > concentrating
> > > > on more urgent things related to the services.
> > > >
> > > > Alex, Vladimir,
> > > >
> > > > Could you have a look at Denis question about the discovery-based
> > > > deployment? Guess it's the only one thing that prevents us from the
> IEP
> > > > finalization.
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Vladimir,
> > > > >
> > > > > Currently we don't save binary metadata to disk, when persistence
> is
> > > > > disabled.
> > > > > But we still persist marshaller mappings for some reason, and I
> > > > personally
> > > > > believe, that we shouldn't.
> > > > >
> > > > > But I agree, that we should separate data and service persistence
> > > > > configuration.
> > > > > Right now persistence of services is configured in a pretty
> > non-obvious
> > > > > manner.
> > > > > It should be a clear way to tell Ignite, whether you want services
> to
> > > be
> > > > > persisted or not.
> > > > >
> > > > > I'm not sure, that we should make "statefullness" in general
> > > > configurable.
> > > > > Users don't care much, whether metadata is preserved on restarts,
> or
> > > not.
> > > > >
> > > > > Denis
> > > > >
> > > > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <[hidden email]
> >:
> > > > >
> > > > > > Alex,
> > > > > >
> > > > > > I would say that we've already had this behavior for years -
> > > marshaller
> > > > > > cache. I think it is time to agree that "in-memory" != stateless.
> > > > Instead
> > > > > > "in-memory" means "data is not stored on disk".
> > > > > > May be we can have a flag which will wipe out all metadata on
> node
> > > > > restart
> > > > > > (e.g. it could make sense for embedded clients)?
> > > > > >
> > > > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > > > > [hidden email]> wrote:
> > > > > >
> > > > > > > Denis,
> > > > > > >
> > > > > > > This is a subtle question. It looks like we have now a number
> of
> > > > > > use-cases
> > > > > > > when persistent storage is required even for a pure in-memory
> > mode.
> > > > One
> > > > > > of
> > > > > > > the use-cases is thin client authentication, the other is
> service
> > > > grid
> > > > > > > configuration persistence.
> > > > > > >
> > > > > > > Generally, I would agree that this is an expected behavior.
> > > However,
> > > > > this
> > > > > > > means that a user cannot simply start and stop nodes randomly
> > > > anymore.
> > > > > > > Ignite start will require some sort of installation or work
> > folder
> > > > > > > initialization (sort of initdb in postgres) which is ok for
> > > > > > > persistence-enabled modes, but I am not sure if this is
> expected
> > > for
> > > > > > > in-memory. Of course, we can run this initialization
> > automatically,
> > > > but
> > > > > > it
> > > > > > > is not always a good idea.
> > > > > > >
> > > > > > > If we are ok to have this restrictions for in-memory mode, then
> > > > service
> > > > > > > persistence makes sense.
> > > > > > >
> > > > > > > --AG
> > > > > > >
> > > > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:
> > > > > > >
> > > > > > >> Denis,
> > > > > > >>
> > > > > > >> I think that the service deployment state needs be persisted
> > > > > > cluster-wide.
> > > > > > >> I guess that our meta-store is capable of doing so. Alex G.,
> > > > Vladimir,
> > > > > > >> could you confirm?
> > > > > > >>
> > > > > > >> As for the split-brain scenarios, I would put them aside for
> now
> > > > > > because,
> > > > > > >> anyway, they have to be solved at lower levels (meta store,
> > > > discovery,
> > > > > > >> etc.).
> > > > > > >>
> > > > > > >> Also, I heard that presently we store a service configuration
> in
> > > the
> > > > > > >> system
> > > > > > >> cache that doesn't give us a way to deploy a new version of a
> > > > service
> > > > > > >> without undeployment of the previous one. Will this issue be
> > > > addressed
> > > > > > by
> > > > > > >> the new deployment approach?
> > > > > > >>
> > > > > > >> --
> > > > > > >> Denis
> > > > > > >>
> > > > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > > > > [hidden email]>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Denis,
> > > > > > >> >
> > > > > > >> > Sounds reasonable. It's not clear, though, what should
> happen,
> > > if
> > > > a
> > > > > > >> joining
> > > > > > >> > node has some services persisted, that are missing on other
> > > nodes.
> > > > > > >> > Should we deploy them?
> > > > > > >> > If we do so, it could lead to surprising behaviour. For
> > example
> > > > you
> > > > > > >> could
> > > > > > >> > kill a node, undeploy a service, then bring back an old
> node,
> > > and
> > > > it
> > > > > > >> would
> > > > > > >> > make the service resurrect.
> > > > > > >> > We could store some deployment counter along with the
> service
> > > > > > >> > configurations on all nodes, that would show how many times
> > the
> > > > > > service
> > > > > > >> > state has changed, i.e. it has been undeployed/redeployed.
> It
> > > > should
> > > > > > be
> > > > > > >> > kept for undeployed services as well to avoid situations
> like
> > I
> > > > > > >> described.
> > > > > > >> >
> > > > > > >> > But it still leaves a possibility of incorrect behaviour, if
> > > there
> > > > > > was a
> > > > > > >> > split-brain situation at some point. I don't think we should
> > > > precess
> > > > > > it
> > > > > > >> > somehow, though. If we choose to tackle it, it will
> > > overcomplicate
> > > > > > >> things
> > > > > > >> > for a sake of a minor improvement.
> > > > > > >> >
> > > > > > >> > Denis
> > > > > > >> >
> > > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > > > >> > [hidden email]>:
> > > > > > >> >
> > > > > > >> > > I was responding to another Denis :) Agree with you on
> your
> > > > point
> > > > > > >> though.
> > > > > > >> > >
> > > > > > >> > > -Val
> > > > > > >> > >
> > > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> > > [hidden email]>
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > > Val,
> > > > > > >> > > >
> > > > > > >> > > > Guess we're talking about other situations. I'm bringing
> > up
> > > > the
> > > > > > case
> > > > > > >> > > when a
> > > > > > >> > > > service was deployed dynamically and has to be brought
> up
> > > > after
> > > > > a
> > > > > > >> full
> > > > > > >> > > > cluster restart w/o user intervention. To achieve this
> we
> > > need
> > > > > to
> > > > > > >> > persist
> > > > > > >> > > > the service's configuration somewhere.
> > > > > > >> > > >
> > > > > > >> > > > --
> > > > > > >> > > > Denis
> > > > > > >> > > >
> > > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > > > > > >> > > > [hidden email]> wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Denis,
> > > > > > >> > > > >
> > > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a class
> is
> > > > > > deployed
> > > > > > >> or
> > > > > > >> > > > > redeployed. If this doesn't happen in some cases, I
> > > believe
> > > > > this
> > > > > > >> > would
> > > > > > >> > > > be a
> > > > > > >> > > > > bug. I don't think we need to add any new events.
> > > > > > >> > > > >
> > > > > > >> > > > > -Val
> > > > > > >> > > > >
> > > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > > > > [hidden email]
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Denis,
> > > > > > >> > > > > >
> > > > > > >> > > > > > I would encourage us to persist a service
> > configuration
> > > in
> > > > > the
> > > > > > >> meta
> > > > > > >> > > > store
> > > > > > >> > > > > > and have this capability enabled by default. That's
> > > > > essential
> > > > > > >> for
> > > > > > >> > > > > services
> > > > > > >> > > > > > started dynamically. Moreover, we support similar
> > > behavior
> > > > > for
> > > > > > >> > > caches,
> > > > > > >> > > > > > indexes, and other DDL changes happened at runtime.
> > > > > > >> > > > > >
> > > > > > >> > > > > > --
> > > > > > >> > > > > > Denis
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > > > > > >> > > > [hidden email]>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Another question, that I would like to discuss is
> > > > whether
> > > > > > >> > services
> > > > > > >> > > > > should
> > > > > > >> > > > > > > be preserved on cluster restarts.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Currently it depends on persistence configuration.
> > If
> > > > > > >> persistence
> > > > > > >> > > for
> > > > > > >> > > > > any
> > > > > > >> > > > > > > data region is enabled, then services will be
> > > persisted
> > > > as
> > > > > > >> well.
> > > > > > >> > > This
> > > > > > >> > > > > is
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > pretty strange way of configuring this behaviour.
> > > > > > >> > > > > > > I'm not sure, if anybody relies on this
> > functionality
> > > > > right
> > > > > > >> now.
> > > > > > >> > > > Should
> > > > > > >> > > > > > we
> > > > > > >> > > > > > > support it at all? If yes, should we make it
> > > > configurable?
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Denis
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > > > > > >> > > [hidden email]
> > > > > > >> > > > >:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Val,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Sounds reasonable. I just think, that user
> should
> > > have
> > > > > > some
> > > > > > >> way
> > > > > > >> > > to
> > > > > > >> > > > > > know,
> > > > > > >> > > > > > > > that new version of a service class was
> deployed.
> > > > > > >> > > > > > > > One way to do it is to listen to
> > > *EVT_CLASS_DEPLOYED.
> > > > > *I'm
> > > > > > >> not
> > > > > > >> > > > sure,
> > > > > > >> > > > > > > > whether it is triggered on class redeployment,
> > > though.
> > > > > If
> > > > > > >> not,
> > > > > > >> > > then
> > > > > > >> > > > > > > another
> > > > > > >> > > > > > > > event type should be added.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > I don't think, that a lot of people will
> implement
> > > > their
> > > > > > own
> > > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make work with
> > > > > > >> > *UriDeploymentSpi*
> > > > > > >> > > > as
> > > > > > >> > > > > > > > comfortable as possible.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Denis
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko
> <
> > > > > > >> > > > > > > > [hidden email]>:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >> Yes, the class deployment itself has to be
> > > explicit.
> > > > > > I.e.,
> > > > > > >> > there
> > > > > > >> > > > has
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > be
> > > > > > >> > > > > > > >> a manual step where user updates the class, and
> > the
> > > > > exact
> > > > > > >> step
> > > > > > >> > > > > > required
> > > > > > >> > > > > > > >> would depend on DeploymentSpi implementation.
> But
> > > > then
> > > > > > >> Ignite
> > > > > > >> > > > takes
> > > > > > >> > > > > > care
> > > > > > >> > > > > > > >> of
> > > > > > >> > > > > > > >> everything else - service redeployment and
> > restart
> > > is
> > > > > > >> > automatic.
> > > > > > >> > > > > > > >>
> > > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be
> disabled
> > if
> > > > > > >> > > DeploymentSpi
> > > > > > >> > > > is
> > > > > > >> > > > > > not
> > > > > > >> > > > > > > >> configured. In this case service class
> > definitions
> > > > have
> > > > > > to
> > > > > > >> be
> > > > > > >> > > > > deployed
> > > > > > >> > > > > > > on
> > > > > > >> > > > > > > >> local classpath and can't be updated in
> runtime.
> > > Just
> > > > > > like
> > > > > > >> it
> > > > > > >> > > > works
> > > > > > >> > > > > > > right
> > > > > > >> > > > > > > >> now.
> > > > > > >> > > > > > > >>
> > > > > > >> > > > > > > >> -Val
> > > > > > >> > > > > > > >>
> > > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy
> > Setrakyan
> > > <
> > > > > > >> > > > > > > [hidden email]
> > > > > > >> > > > > > > >> >
> > > > > > >> > > > > > > >> wrote:
> > > > > > >> > > > > > > >>
> > > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry
> Pavlov <
> > > > > > >> > > > > > [hidden email]>
> > > > > > >> > > > > > > >> > wrote:
> > > > > > >> > > > > > > >> >
> > > > > > >> > > > > > > >> > > Hi Igniters,
> > > > > > >> > > > > > > >> > >
> > > > > > >> > > > > > > >> > > I like automatic redeploy which can be
> > disabled
> > > > by
> > > > > > >> config
> > > > > > >> > if
> > > > > > >> > > > > user
> > > > > > >> > > > > > > >> wants
> > > > > > >> > > > > > > >> > to
> > > > > > >> > > > > > > >> > > control this process. What do you think?
> > > > > > >> > > > > > > >> > >
> > > > > > >> > > > > > > >> >
> > > > > > >> > > > > > > >> > I do not think we should have anything
> > automatic
> > > > when
> > > > > > it
> > > > > > >> > comes
> > > > > > >> > > > to
> > > > > > >> > > > > > > >> > deployment, everything should be explicit.
> > > However,
> > > > > if
> > > > > > we
> > > > > > >> > use
> > > > > > >> > > > the
> > > > > > >> > > > > > > >> > deployment SPI, then a user should be able to
> > do
> > > > > "hot"
> > > > > > >> > > redeploy,
> > > > > > >> > > > > > > where a
> > > > > > >> > > > > > > >> > new service will be deployed if the user
> drops
> > an
> > > > > > updated
> > > > > > >> > jar.
> > > > > > >> > > > > > > >> >
> > > > > > >> > > > > > > >> > We should not create anything new here.
> Ignite
> > > > > already
> > > > > > >> has a
> > > > > > >> > > > > > > deployment
> > > > > > >> > > > > > > >> SPI
> > > > > > >> > > > > > > >> > and it already works in a certain way. Let's
> > not
> > > > > change
> > > > > > >> it.
> > > > > > >> > > > > > > >> >
> > > > > > >> > > > > > > >> > D.
> > > > > > >> > > > > > > >> >
> > > > > > >> > > > > > > >>
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Valentin Kulichenko]

Denis,

I totally agree with you. I'm just not sure why do we need two methods
(init and execute) that have virtually same semantics. With the new design,
what would be the difference between these methods from user point of view,
and how one would determine what exactly should go in each of them? Unless
I'm missing something, it looks like unnecessary complication.

-Val

On Tue, Apr 17, 2018 at 1:00 AM, Denis Mekhanikov <[hidden email]>
wrote:

> Val,
>
> Service initialisation is not going to happen in the discovery thread.
> It should be done asynchronously, and initialisation results should be sent
> to the coordinator over communication.
> This is described in the IEP:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> Successfulscenario
>
> *init()* method is a validation step, making sure, that service is ready
> for work.
> And deployment shouldn't be considered successful until *init()* methods
> finish their work on the assigned nodes.
> Also *cancel() *and *init() *methods may be useful if we decide to
> implement moving existing services to new nodes
> <https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> Movingexistingservicestonewnodes>
>  in
> future.
> These methods can be used to save and restore service's state from cache,
> when it is rebalanced to another node.
>
> As Denis said, if we are not going to prevent nodes from starting on
> service failures, then we should at least generate corresponding events.
> Otherwise there won't be any way to react to service initialization
> failures during nodes startup.
>
> Denis
>
> вт, 17 апр. 2018 г. в 6:59, Valentin Kulichenko <
> [hidden email]>:
>
> > I agree we shouldn't do anything synchronously within discovery threads.
> If
> > something goes wrong, we just need to properly notify the user, logging
> and
> > events seem to be right options to achieve that.
> >
> > BTW, with this design I'm not sure init() method makes sense, probably we
> > should deprecate it.
> >
> > -Val
> >
> > On Mon, Apr 16, 2018 at 12:03 PM, Denis Magda <[hidden email]> wrote:
> >
> > > Denis,
> > >
> > > In general, service initialization shouldn't prevent a node from
> joining
> > > the cluster or slowing down that process. Thus, I would start the
> > > initialization routines only after the node is accepted by the cluster.
> > If
> > > the initialization fails then we need to report a respective message to
> > the
> > > logs and, probably, generate a system event the user can be subscribed
> > to.
> > >
> > > Regardless, of the service initialization time, I think we still need
> to
> > > utilize discovery SPI to avoid problems discussed later.
> > >
> > > Val, others, what do you think about that?
> > >
> > >
> > > --
> > > Denis
> > >
> > > On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <
> > [hidden email]>
> > > wrote:
> > >
> > > > Basically, my question is: at which moment services should be
> deployed
> > on
> > > > connecting nodes?
> > > > Should we reject a node from being included into a topology, if
> > services,
> > > > that are assigned to it, fail to deploy?
> > > >
> > > > It would be good to be sure, that all assigned services are
> initialised
> > > and
> > > > working, when node start is finished.
> > > > Otherwise it's unclear, how to notify a user about failures in
> service
> > > > initialisation on new nodes.
> > > >
> > > > If we decide to provide such guarantee, then how are we going to do
> > that?
> > > > Is procedure, that I described, viable?
> > > > It requires hacking through the discovery protocol, which is a thing,
> > > that
> > > > should be avoided.
> > > > So, maybe there is another way to achieve the same thing?
> > > >
> > > > Denis
> > > >
> > > > сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:
> > > >
> > > > > It sounds like it's not a trivial thing to support the automatic
> > > services
> > > > > redeployment after a restart. Let's postpone it for now, guys
> > > > concentrating
> > > > > on more urgent things related to the services.
> > > > >
> > > > > Alex, Vladimir,
> > > > >
> > > > > Could you have a look at Denis question about the discovery-based
> > > > > deployment? Guess it's the only one thing that prevents us from the
> > IEP
> > > > > finalization.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Vladimir,
> > > > > >
> > > > > > Currently we don't save binary metadata to disk, when persistence
> > is
> > > > > > disabled.
> > > > > > But we still persist marshaller mappings for some reason, and I
> > > > > personally
> > > > > > believe, that we shouldn't.
> > > > > >
> > > > > > But I agree, that we should separate data and service persistence
> > > > > > configuration.
> > > > > > Right now persistence of services is configured in a pretty
> > > non-obvious
> > > > > > manner.
> > > > > > It should be a clear way to tell Ignite, whether you want
> services
> > to
> > > > be
> > > > > > persisted or not.
> > > > > >
> > > > > > I'm not sure, that we should make "statefullness" in general
> > > > > configurable.
> > > > > > Users don't care much, whether metadata is preserved on restarts,
> > or
> > > > not.
> > > > > >
> > > > > > Denis
> > > > > >
> > > > > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <
> [hidden email]
> > >:
> > > > > >
> > > > > > > Alex,
> > > > > > >
> > > > > > > I would say that we've already had this behavior for years -
> > > > marshaller
> > > > > > > cache. I think it is time to agree that "in-memory" !=
> stateless.
> > > > > Instead
> > > > > > > "in-memory" means "data is not stored on disk".
> > > > > > > May be we can have a flag which will wipe out all metadata on
> > node
> > > > > > restart
> > > > > > > (e.g. it could make sense for embedded clients)?
> > > > > > >
> > > > > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > > > > > [hidden email]> wrote:
> > > > > > >
> > > > > > > > Denis,
> > > > > > > >
> > > > > > > > This is a subtle question. It looks like we have now a number
> > of
> > > > > > > use-cases
> > > > > > > > when persistent storage is required even for a pure in-memory
> > > mode.
> > > > > One
> > > > > > > of
> > > > > > > > the use-cases is thin client authentication, the other is
> > service
> > > > > grid
> > > > > > > > configuration persistence.
> > > > > > > >
> > > > > > > > Generally, I would agree that this is an expected behavior.
> > > > However,
> > > > > > this
> > > > > > > > means that a user cannot simply start and stop nodes randomly
> > > > > anymore.
> > > > > > > > Ignite start will require some sort of installation or work
> > > folder
> > > > > > > > initialization (sort of initdb in postgres) which is ok for
> > > > > > > > persistence-enabled modes, but I am not sure if this is
> > expected
> > > > for
> > > > > > > > in-memory. Of course, we can run this initialization
> > > automatically,
> > > > > but
> > > > > > > it
> > > > > > > > is not always a good idea.
> > > > > > > >
> > > > > > > > If we are ok to have this restrictions for in-memory mode,
> then
> > > > > service
> > > > > > > > persistence makes sense.
> > > > > > > >
> > > > > > > > --AG
> > > > > > > >
> > > > > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]>:
> > > > > > > >
> > > > > > > >> Denis,
> > > > > > > >>
> > > > > > > >> I think that the service deployment state needs be persisted
> > > > > > > cluster-wide.
> > > > > > > >> I guess that our meta-store is capable of doing so. Alex G.,
> > > > > Vladimir,
> > > > > > > >> could you confirm?
> > > > > > > >>
> > > > > > > >> As for the split-brain scenarios, I would put them aside for
> > now
> > > > > > > because,
> > > > > > > >> anyway, they have to be solved at lower levels (meta store,
> > > > > discovery,
> > > > > > > >> etc.).
> > > > > > > >>
> > > > > > > >> Also, I heard that presently we store a service
> configuration
> > in
> > > > the
> > > > > > > >> system
> > > > > > > >> cache that doesn't give us a way to deploy a new version of
> a
> > > > > service
> > > > > > > >> without undeployment of the previous one. Will this issue be
> > > > > addressed
> > > > > > > by
> > > > > > > >> the new deployment approach?
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Denis
> > > > > > > >>
> > > > > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > > > > > [hidden email]>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Denis,
> > > > > > > >> >
> > > > > > > >> > Sounds reasonable. It's not clear, though, what should
> > happen,
> > > > if
> > > > > a
> > > > > > > >> joining
> > > > > > > >> > node has some services persisted, that are missing on
> other
> > > > nodes.
> > > > > > > >> > Should we deploy them?
> > > > > > > >> > If we do so, it could lead to surprising behaviour. For
> > > example
> > > > > you
> > > > > > > >> could
> > > > > > > >> > kill a node, undeploy a service, then bring back an old
> > node,
> > > > and
> > > > > it
> > > > > > > >> would
> > > > > > > >> > make the service resurrect.
> > > > > > > >> > We could store some deployment counter along with the
> > service
> > > > > > > >> > configurations on all nodes, that would show how many
> times
> > > the
> > > > > > > service
> > > > > > > >> > state has changed, i.e. it has been undeployed/redeployed.
> > It
> > > > > should
> > > > > > > be
> > > > > > > >> > kept for undeployed services as well to avoid situations
> > like
> > > I
> > > > > > > >> described.
> > > > > > > >> >
> > > > > > > >> > But it still leaves a possibility of incorrect behaviour,
> if
> > > > there
> > > > > > > was a
> > > > > > > >> > split-brain situation at some point. I don't think we
> should
> > > > > precess
> > > > > > > it
> > > > > > > >> > somehow, though. If we choose to tackle it, it will
> > > > overcomplicate
> > > > > > > >> things
> > > > > > > >> > for a sake of a minor improvement.
> > > > > > > >> >
> > > > > > > >> > Denis
> > > > > > > >> >
> > > > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > > > > >> > [hidden email]>:
> > > > > > > >> >
> > > > > > > >> > > I was responding to another Denis :) Agree with you on
> > your
> > > > > point
> > > > > > > >> though.
> > > > > > > >> > >
> > > > > > > >> > > -Val
> > > > > > > >> > >
> > > > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> > > > [hidden email]>
> > > > > > > >> wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Val,
> > > > > > > >> > > >
> > > > > > > >> > > > Guess we're talking about other situations. I'm
> bringing
> > > up
> > > > > the
> > > > > > > case
> > > > > > > >> > > when a
> > > > > > > >> > > > service was deployed dynamically and has to be brought
> > up
> > > > > after
> > > > > > a
> > > > > > > >> full
> > > > > > > >> > > > cluster restart w/o user intervention. To achieve this
> > we
> > > > need
> > > > > > to
> > > > > > > >> > persist
> > > > > > > >> > > > the service's configuration somewhere.
> > > > > > > >> > > >
> > > > > > > >> > > > --
> > > > > > > >> > > > Denis
> > > > > > > >> > > >
> > > > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko <
> > > > > > > >> > > > [hidden email]> wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Denis,
> > > > > > > >> > > > >
> > > > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a
> class
> > is
> > > > > > > deployed
> > > > > > > >> or
> > > > > > > >> > > > > redeployed. If this doesn't happen in some cases, I
> > > > believe
> > > > > > this
> > > > > > > >> > would
> > > > > > > >> > > > be a
> > > > > > > >> > > > > bug. I don't think we need to add any new events.
> > > > > > > >> > > > >
> > > > > > > >> > > > > -Val
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Denis,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I would encourage us to persist a service
> > > configuration
> > > > in
> > > > > > the
> > > > > > > >> meta
> > > > > > > >> > > > store
> > > > > > > >> > > > > > and have this capability enabled by default.
> That's
> > > > > > essential
> > > > > > > >> for
> > > > > > > >> > > > > services
> > > > > > > >> > > > > > started dynamically. Moreover, we support similar
> > > > behavior
> > > > > > for
> > > > > > > >> > > caches,
> > > > > > > >> > > > > > indexes, and other DDL changes happened at
> runtime.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > --
> > > > > > > >> > > > > > Denis
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov <
> > > > > > > >> > > > [hidden email]>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Another question, that I would like to discuss
> is
> > > > > whether
> > > > > > > >> > services
> > > > > > > >> > > > > should
> > > > > > > >> > > > > > > be preserved on cluster restarts.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Currently it depends on persistence
> configuration.
> > > If
> > > > > > > >> persistence
> > > > > > > >> > > for
> > > > > > > >> > > > > any
> > > > > > > >> > > > > > > data region is enabled, then services will be
> > > > persisted
> > > > > as
> > > > > > > >> well.
> > > > > > > >> > > This
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > a
> > > > > > > >> > > > > > > pretty strange way of configuring this
> behaviour.
> > > > > > > >> > > > > > > I'm not sure, if anybody relies on this
> > > functionality
> > > > > > right
> > > > > > > >> now.
> > > > > > > >> > > > Should
> > > > > > > >> > > > > > we
> > > > > > > >> > > > > > > support it at all? If yes, should we make it
> > > > > configurable?
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Denis
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > > > > > > >> > > [hidden email]
> > > > > > > >> > > > >:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Val,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Sounds reasonable. I just think, that user
> > should
> > > > have
> > > > > > > some
> > > > > > > >> way
> > > > > > > >> > > to
> > > > > > > >> > > > > > know,
> > > > > > > >> > > > > > > > that new version of a service class was
> > deployed.
> > > > > > > >> > > > > > > > One way to do it is to listen to
> > > > *EVT_CLASS_DEPLOYED.
> > > > > > *I'm
> > > > > > > >> not
> > > > > > > >> > > > sure,
> > > > > > > >> > > > > > > > whether it is triggered on class redeployment,
> > > > though.
> > > > > > If
> > > > > > > >> not,
> > > > > > > >> > > then
> > > > > > > >> > > > > > > another
> > > > > > > >> > > > > > > > event type should be added.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > I don't think, that a lot of people will
> > implement
> > > > > their
> > > > > > > own
> > > > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make work with
> > > > > > > >> > *UriDeploymentSpi*
> > > > > > > >> > > > as
> > > > > > > >> > > > > > > > comfortable as possible.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Denis
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin
> Kulichenko
> > <
> > > > > > > >> > > > > > > > [hidden email]>:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >> Yes, the class deployment itself has to be
> > > > explicit.
> > > > > > > I.e.,
> > > > > > > >> > there
> > > > > > > >> > > > has
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > be
> > > > > > > >> > > > > > > >> a manual step where user updates the class,
> and
> > > the
> > > > > > exact
> > > > > > > >> step
> > > > > > > >> > > > > > required
> > > > > > > >> > > > > > > >> would depend on DeploymentSpi implementation.
> > But
> > > > > then
> > > > > > > >> Ignite
> > > > > > > >> > > > takes
> > > > > > > >> > > > > > care
> > > > > > > >> > > > > > > >> of
> > > > > > > >> > > > > > > >> everything else - service redeployment and
> > > restart
> > > > is
> > > > > > > >> > automatic.
> > > > > > > >> > > > > > > >>
> > > > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be
> > disabled
> > > if
> > > > > > > >> > > DeploymentSpi
> > > > > > > >> > > > is
> > > > > > > >> > > > > > not
> > > > > > > >> > > > > > > >> configured. In this case service class
> > > definitions
> > > > > have
> > > > > > > to
> > > > > > > >> be
> > > > > > > >> > > > > deployed
> > > > > > > >> > > > > > > on
> > > > > > > >> > > > > > > >> local classpath and can't be updated in
> > runtime.
> > > > Just
> > > > > > > like
> > > > > > > >> it
> > > > > > > >> > > > works
> > > > > > > >> > > > > > > right
> > > > > > > >> > > > > > > >> now.
> > > > > > > >> > > > > > > >>
> > > > > > > >> > > > > > > >> -Val
> > > > > > > >> > > > > > > >>
> > > > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy
> > > Setrakyan
> > > > <
> > > > > > > >> > > > > > > [hidden email]
> > > > > > > >> > > > > > > >> >
> > > > > > > >> > > > > > > >> wrote:
> > > > > > > >> > > > > > > >>
> > > > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry
> > Pavlov <
> > > > > > > >> > > > > > [hidden email]>
> > > > > > > >> > > > > > > >> > wrote:
> > > > > > > >> > > > > > > >> >
> > > > > > > >> > > > > > > >> > > Hi Igniters,
> > > > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > > > >> > > I like automatic redeploy which can be
> > > disabled
> > > > > by
> > > > > > > >> config
> > > > > > > >> > if
> > > > > > > >> > > > > user
> > > > > > > >> > > > > > > >> wants
> > > > > > > >> > > > > > > >> > to
> > > > > > > >> > > > > > > >> > > control this process. What do you think?
> > > > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > > > >> >
> > > > > > > >> > > > > > > >> > I do not think we should have anything
> > > automatic
> > > > > when
> > > > > > > it
> > > > > > > >> > comes
> > > > > > > >> > > > to
> > > > > > > >> > > > > > > >> > deployment, everything should be explicit.
> > > > However,
> > > > > > if
> > > > > > > we
> > > > > > > >> > use
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > >> > deployment SPI, then a user should be able
> to
> > > do
> > > > > > "hot"
> > > > > > > >> > > redeploy,
> > > > > > > >> > > > > > > where a
> > > > > > > >> > > > > > > >> > new service will be deployed if the user
> > drops
> > > an
> > > > > > > updated
> > > > > > > >> > jar.
> > > > > > > >> > > > > > > >> >
> > > > > > > >> > > > > > > >> > We should not create anything new here.
> > Ignite
> > > > > > already
> > > > > > > >> has a
> > > > > > > >> > > > > > > deployment
> > > > > > > >> > > > > > > >> SPI
> > > > > > > >> > > > > > > >> > and it already works in a certain way.
> Let's
> > > not
> > > > > > change
> > > > > > > >> it.
> > > > > > > >> > > > > > > >> >
> > > > > > > >> > > > > > > >> > D.
> > > > > > > >> > > > > > > >> >
> > > > > > > >> > > > > > > >>
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Denis Mekhanikov]

Val,

*init()* method is executed before a service is considered deployed.
If any exception is thrown from it, then it will be handled as deployment
failure.

*execute() *method is run after the service is deployed, and it can keep
running until the service is cancelled.
This method has its own thread, so it can perform some background work.

Suppose you want to deploy HTTP server as a service on one of your nodes.
You can place HTTP server creation logic in the *init() *method.
If some nodes don't have a permission to listen to needed ports, then a
corresponding exception will be propagated to the user code.
On the other hand, if exception is thrown from the *execute() *method, then
service won't be undeployed.

Denis

пт, 20 апр. 2018 г. в 2:35, Valentin Kulichenko <
[hidden email]>:

> Denis,
>
> I totally agree with you. I'm just not sure why do we need two methods
> (init and execute) that have virtually same semantics. With the new design,
> what would be the difference between these methods from user point of view,
> and how one would determine what exactly should go in each of them? Unless
> I'm missing something, it looks like unnecessary complication.
>
> -Val
>
> On Tue, Apr 17, 2018 at 1:00 AM, Denis Mekhanikov <[hidden email]>
> wrote:
>
> > Val,
> >
> > Service initialisation is not going to happen in the discovery thread.
> > It should be done asynchronously, and initialisation results should be
> sent
> > to the coordinator over communication.
> > This is described in the IEP:
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> > Successfulscenario
> >
> > *init()* method is a validation step, making sure, that service is ready
> > for work.
> > And deployment shouldn't be considered successful until *init()* methods
> > finish their work on the assigned nodes.
> > Also *cancel() *and *init() *methods may be useful if we decide to
> > implement moving existing services to new nodes
> > <https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> > Movingexistingservicestonewnodes>
> >  in
> > future.
> > These methods can be used to save and restore service's state from cache,
> > when it is rebalanced to another node.
> >
> > As Denis said, if we are not going to prevent nodes from starting on
> > service failures, then we should at least generate corresponding events.
> > Otherwise there won't be any way to react to service initialization
> > failures during nodes startup.
> >
> > Denis
> >
> > вт, 17 апр. 2018 г. в 6:59, Valentin Kulichenko <
> > [hidden email]>:
> >
> > > I agree we shouldn't do anything synchronously within discovery
> threads.
> > If
> > > something goes wrong, we just need to properly notify the user, logging
> > and
> > > events seem to be right options to achieve that.
> > >
> > > BTW, with this design I'm not sure init() method makes sense, probably
> we
> > > should deprecate it.
> > >
> > > -Val
> > >
> > > On Mon, Apr 16, 2018 at 12:03 PM, Denis Magda <[hidden email]>
> wrote:
> > >
> > > > Denis,
> > > >
> > > > In general, service initialization shouldn't prevent a node from
> > joining
> > > > the cluster or slowing down that process. Thus, I would start the
> > > > initialization routines only after the node is accepted by the
> cluster.
> > > If
> > > > the initialization fails then we need to report a respective message
> to
> > > the
> > > > logs and, probably, generate a system event the user can be
> subscribed
> > > to.
> > > >
> > > > Regardless, of the service initialization time, I think we still need
> > to
> > > > utilize discovery SPI to avoid problems discussed later.
> > > >
> > > > Val, others, what do you think about that?
> > > >
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Basically, my question is: at which moment services should be
> > deployed
> > > on
> > > > > connecting nodes?
> > > > > Should we reject a node from being included into a topology, if
> > > services,
> > > > > that are assigned to it, fail to deploy?
> > > > >
> > > > > It would be good to be sure, that all assigned services are
> > initialised
> > > > and
> > > > > working, when node start is finished.
> > > > > Otherwise it's unclear, how to notify a user about failures in
> > service
> > > > > initialisation on new nodes.
> > > > >
> > > > > If we decide to provide such guarantee, then how are we going to do
> > > that?
> > > > > Is procedure, that I described, viable?
> > > > > It requires hacking through the discovery protocol, which is a
> thing,
> > > > that
> > > > > should be avoided.
> > > > > So, maybe there is another way to achieve the same thing?
> > > > >
> > > > > Denis
> > > > >
> > > > > сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:
> > > > >
> > > > > > It sounds like it's not a trivial thing to support the automatic
> > > > services
> > > > > > redeployment after a restart. Let's postpone it for now, guys
> > > > > concentrating
> > > > > > on more urgent things related to the services.
> > > > > >
> > > > > > Alex, Vladimir,
> > > > > >
> > > > > > Could you have a look at Denis question about the discovery-based
> > > > > > deployment? Guess it's the only one thing that prevents us from
> the
> > > IEP
> > > > > > finalization.
> > > > > >
> > > > > > --
> > > > > > Denis
> > > > > >
> > > > > > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <
> > > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Vladimir,
> > > > > > >
> > > > > > > Currently we don't save binary metadata to disk, when
> persistence
> > > is
> > > > > > > disabled.
> > > > > > > But we still persist marshaller mappings for some reason, and I
> > > > > > personally
> > > > > > > believe, that we shouldn't.
> > > > > > >
> > > > > > > But I agree, that we should separate data and service
> persistence
> > > > > > > configuration.
> > > > > > > Right now persistence of services is configured in a pretty
> > > > non-obvious
> > > > > > > manner.
> > > > > > > It should be a clear way to tell Ignite, whether you want
> > services
> > > to
> > > > > be
> > > > > > > persisted or not.
> > > > > > >
> > > > > > > I'm not sure, that we should make "statefullness" in general
> > > > > > configurable.
> > > > > > > Users don't care much, whether metadata is preserved on
> restarts,
> > > or
> > > > > not.
> > > > > > >
> > > > > > > Denis
> > > > > > >
> > > > > > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <
> > [hidden email]
> > > >:
> > > > > > >
> > > > > > > > Alex,
> > > > > > > >
> > > > > > > > I would say that we've already had this behavior for years -
> > > > > marshaller
> > > > > > > > cache. I think it is time to agree that "in-memory" !=
> > stateless.
> > > > > > Instead
> > > > > > > > "in-memory" means "data is not stored on disk".
> > > > > > > > May be we can have a flag which will wipe out all metadata on
> > > node
> > > > > > > restart
> > > > > > > > (e.g. it could make sense for embedded clients)?
> > > > > > > >
> > > > > > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > > > > > > [hidden email]> wrote:
> > > > > > > >
> > > > > > > > > Denis,
> > > > > > > > >
> > > > > > > > > This is a subtle question. It looks like we have now a
> number
> > > of
> > > > > > > > use-cases
> > > > > > > > > when persistent storage is required even for a pure
> in-memory
> > > > mode.
> > > > > > One
> > > > > > > > of
> > > > > > > > > the use-cases is thin client authentication, the other is
> > > service
> > > > > > grid
> > > > > > > > > configuration persistence.
> > > > > > > > >
> > > > > > > > > Generally, I would agree that this is an expected behavior.
> > > > > However,
> > > > > > > this
> > > > > > > > > means that a user cannot simply start and stop nodes
> randomly
> > > > > > anymore.
> > > > > > > > > Ignite start will require some sort of installation or work
> > > > folder
> > > > > > > > > initialization (sort of initdb in postgres) which is ok for
> > > > > > > > > persistence-enabled modes, but I am not sure if this is
> > > expected
> > > > > for
> > > > > > > > > in-memory. Of course, we can run this initialization
> > > > automatically,
> > > > > > but
> > > > > > > > it
> > > > > > > > > is not always a good idea.
> > > > > > > > >
> > > > > > > > > If we are ok to have this restrictions for in-memory mode,
> > then
> > > > > > service
> > > > > > > > > persistence makes sense.
> > > > > > > > >
> > > > > > > > > --AG
> > > > > > > > >
> > > > > > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <[hidden email]
> >:
> > > > > > > > >
> > > > > > > > >> Denis,
> > > > > > > > >>
> > > > > > > > >> I think that the service deployment state needs be
> persisted
> > > > > > > > cluster-wide.
> > > > > > > > >> I guess that our meta-store is capable of doing so. Alex
> G.,
> > > > > > Vladimir,
> > > > > > > > >> could you confirm?
> > > > > > > > >>
> > > > > > > > >> As for the split-brain scenarios, I would put them aside
> for
> > > now
> > > > > > > > because,
> > > > > > > > >> anyway, they have to be solved at lower levels (meta
> store,
> > > > > > discovery,
> > > > > > > > >> etc.).
> > > > > > > > >>
> > > > > > > > >> Also, I heard that presently we store a service
> > configuration
> > > in
> > > > > the
> > > > > > > > >> system
> > > > > > > > >> cache that doesn't give us a way to deploy a new version
> of
> > a
> > > > > > service
> > > > > > > > >> without undeployment of the previous one. Will this issue
> be
> > > > > > addressed
> > > > > > > > by
> > > > > > > > >> the new deployment approach?
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Denis
> > > > > > > > >>
> > > > > > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > > > > > > [hidden email]>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Denis,
> > > > > > > > >> >
> > > > > > > > >> > Sounds reasonable. It's not clear, though, what should
> > > happen,
> > > > > if
> > > > > > a
> > > > > > > > >> joining
> > > > > > > > >> > node has some services persisted, that are missing on
> > other
> > > > > nodes.
> > > > > > > > >> > Should we deploy them?
> > > > > > > > >> > If we do so, it could lead to surprising behaviour. For
> > > > example
> > > > > > you
> > > > > > > > >> could
> > > > > > > > >> > kill a node, undeploy a service, then bring back an old
> > > node,
> > > > > and
> > > > > > it
> > > > > > > > >> would
> > > > > > > > >> > make the service resurrect.
> > > > > > > > >> > We could store some deployment counter along with the
> > > service
> > > > > > > > >> > configurations on all nodes, that would show how many
> > times
> > > > the
> > > > > > > > service
> > > > > > > > >> > state has changed, i.e. it has been
> undeployed/redeployed.
> > > It
> > > > > > should
> > > > > > > > be
> > > > > > > > >> > kept for undeployed services as well to avoid situations
> > > like
> > > > I
> > > > > > > > >> described.
> > > > > > > > >> >
> > > > > > > > >> > But it still leaves a possibility of incorrect
> behaviour,
> > if
> > > > > there
> > > > > > > > was a
> > > > > > > > >> > split-brain situation at some point. I don't think we
> > should
> > > > > > precess
> > > > > > > > it
> > > > > > > > >> > somehow, though. If we choose to tackle it, it will
> > > > > overcomplicate
> > > > > > > > >> things
> > > > > > > > >> > for a sake of a minor improvement.
> > > > > > > > >> >
> > > > > > > > >> > Denis
> > > > > > > > >> >
> > > > > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > > > > > >> > [hidden email]>:
> > > > > > > > >> >
> > > > > > > > >> > > I was responding to another Denis :) Agree with you on
> > > your
> > > > > > point
> > > > > > > > >> though.
> > > > > > > > >> > >
> > > > > > > > >> > > -Val
> > > > > > > > >> > >
> > > > > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> > > > > [hidden email]>
> > > > > > > > >> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Val,
> > > > > > > > >> > > >
> > > > > > > > >> > > > Guess we're talking about other situations. I'm
> > bringing
> > > > up
> > > > > > the
> > > > > > > > case
> > > > > > > > >> > > when a
> > > > > > > > >> > > > service was deployed dynamically and has to be
> brought
> > > up
> > > > > > after
> > > > > > > a
> > > > > > > > >> full
> > > > > > > > >> > > > cluster restart w/o user intervention. To achieve
> this
> > > we
> > > > > need
> > > > > > > to
> > > > > > > > >> > persist
> > > > > > > > >> > > > the service's configuration somewhere.
> > > > > > > > >> > > >
> > > > > > > > >> > > > --
> > > > > > > > >> > > > Denis
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko
> <
> > > > > > > > >> > > > [hidden email]> wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Denis,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a
> > class
> > > is
> > > > > > > > deployed
> > > > > > > > >> or
> > > > > > > > >> > > > > redeployed. If this doesn't happen in some cases,
> I
> > > > > believe
> > > > > > > this
> > > > > > > > >> > would
> > > > > > > > >> > > > be a
> > > > > > > > >> > > > > bug. I don't think we need to add any new events.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > -Val
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Denis,
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > I would encourage us to persist a service
> > > > configuration
> > > > > in
> > > > > > > the
> > > > > > > > >> meta
> > > > > > > > >> > > > store
> > > > > > > > >> > > > > > and have this capability enabled by default.
> > That's
> > > > > > > essential
> > > > > > > > >> for
> > > > > > > > >> > > > > services
> > > > > > > > >> > > > > > started dynamically. Moreover, we support
> similar
> > > > > behavior
> > > > > > > for
> > > > > > > > >> > > caches,
> > > > > > > > >> > > > > > indexes, and other DDL changes happened at
> > runtime.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > --
> > > > > > > > >> > > > > > Denis
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis
> Mekhanikov <
> > > > > > > > >> > > > [hidden email]>
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Another question, that I would like to discuss
> > is
> > > > > > whether
> > > > > > > > >> > services
> > > > > > > > >> > > > > should
> > > > > > > > >> > > > > > > be preserved on cluster restarts.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Currently it depends on persistence
> > configuration.
> > > > If
> > > > > > > > >> persistence
> > > > > > > > >> > > for
> > > > > > > > >> > > > > any
> > > > > > > > >> > > > > > > data region is enabled, then services will be
> > > > > persisted
> > > > > > as
> > > > > > > > >> well.
> > > > > > > > >> > > This
> > > > > > > > >> > > > > is
> > > > > > > > >> > > > > > a
> > > > > > > > >> > > > > > > pretty strange way of configuring this
> > behaviour.
> > > > > > > > >> > > > > > > I'm not sure, if anybody relies on this
> > > > functionality
> > > > > > > right
> > > > > > > > >> now.
> > > > > > > > >> > > > Should
> > > > > > > > >> > > > > > we
> > > > > > > > >> > > > > > > support it at all? If yes, should we make it
> > > > > > configurable?
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Denis
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov <
> > > > > > > > >> > > [hidden email]
> > > > > > > > >> > > > >:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > Val,
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Sounds reasonable. I just think, that user
> > > should
> > > > > have
> > > > > > > > some
> > > > > > > > >> way
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > know,
> > > > > > > > >> > > > > > > > that new version of a service class was
> > > deployed.
> > > > > > > > >> > > > > > > > One way to do it is to listen to
> > > > > *EVT_CLASS_DEPLOYED.
> > > > > > > *I'm
> > > > > > > > >> not
> > > > > > > > >> > > > sure,
> > > > > > > > >> > > > > > > > whether it is triggered on class
> redeployment,
> > > > > though.
> > > > > > > If
> > > > > > > > >> not,
> > > > > > > > >> > > then
> > > > > > > > >> > > > > > > another
> > > > > > > > >> > > > > > > > event type should be added.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > I don't think, that a lot of people will
> > > implement
> > > > > > their
> > > > > > > > own
> > > > > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make work
> with
> > > > > > > > >> > *UriDeploymentSpi*
> > > > > > > > >> > > > as
> > > > > > > > >> > > > > > > > comfortable as possible.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Denis
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin
> > Kulichenko
> > > <
> > > > > > > > >> > > > > > > > [hidden email]>:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >> Yes, the class deployment itself has to be
> > > > > explicit.
> > > > > > > > I.e.,
> > > > > > > > >> > there
> > > > > > > > >> > > > has
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > be
> > > > > > > > >> > > > > > > >> a manual step where user updates the class,
> > and
> > > > the
> > > > > > > exact
> > > > > > > > >> step
> > > > > > > > >> > > > > > required
> > > > > > > > >> > > > > > > >> would depend on DeploymentSpi
> implementation.
> > > But
> > > > > > then
> > > > > > > > >> Ignite
> > > > > > > > >> > > > takes
> > > > > > > > >> > > > > > care
> > > > > > > > >> > > > > > > >> of
> > > > > > > > >> > > > > > > >> everything else - service redeployment and
> > > > restart
> > > > > is
> > > > > > > > >> > automatic.
> > > > > > > > >> > > > > > > >>
> > > > > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be
> > > disabled
> > > > if
> > > > > > > > >> > > DeploymentSpi
> > > > > > > > >> > > > is
> > > > > > > > >> > > > > > not
> > > > > > > > >> > > > > > > >> configured. In this case service class
> > > > definitions
> > > > > > have
> > > > > > > > to
> > > > > > > > >> be
> > > > > > > > >> > > > > deployed
> > > > > > > > >> > > > > > > on
> > > > > > > > >> > > > > > > >> local classpath and can't be updated in
> > > runtime.
> > > > > Just
> > > > > > > > like
> > > > > > > > >> it
> > > > > > > > >> > > > works
> > > > > > > > >> > > > > > > right
> > > > > > > > >> > > > > > > >> now.
> > > > > > > > >> > > > > > > >>
> > > > > > > > >> > > > > > > >> -Val
> > > > > > > > >> > > > > > > >>
> > > > > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy
> > > > Setrakyan
> > > > > <
> > > > > > > > >> > > > > > > [hidden email]
> > > > > > > > >> > > > > > > >> >
> > > > > > > > >> > > > > > > >> wrote:
> > > > > > > > >> > > > > > > >>
> > > > > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry
> > > Pavlov <
> > > > > > > > >> > > > > > [hidden email]>
> > > > > > > > >> > > > > > > >> > wrote:
> > > > > > > > >> > > > > > > >> >
> > > > > > > > >> > > > > > > >> > > Hi Igniters,
> > > > > > > > >> > > > > > > >> > >
> > > > > > > > >> > > > > > > >> > > I like automatic redeploy which can be
> > > > disabled
> > > > > > by
> > > > > > > > >> config
> > > > > > > > >> > if
> > > > > > > > >> > > > > user
> > > > > > > > >> > > > > > > >> wants
> > > > > > > > >> > > > > > > >> > to
> > > > > > > > >> > > > > > > >> > > control this process. What do you
> think?
> > > > > > > > >> > > > > > > >> > >
> > > > > > > > >> > > > > > > >> >
> > > > > > > > >> > > > > > > >> > I do not think we should have anything
> > > > automatic
> > > > > > when
> > > > > > > > it
> > > > > > > > >> > comes
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > > >> > deployment, everything should be
> explicit.
> > > > > However,
> > > > > > > if
> > > > > > > > we
> > > > > > > > >> > use
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > >> > deployment SPI, then a user should be
> able
> > to
> > > > do
> > > > > > > "hot"
> > > > > > > > >> > > redeploy,
> > > > > > > > >> > > > > > > where a
> > > > > > > > >> > > > > > > >> > new service will be deployed if the user
> > > drops
> > > > an
> > > > > > > > updated
> > > > > > > > >> > jar.
> > > > > > > > >> > > > > > > >> >
> > > > > > > > >> > > > > > > >> > We should not create anything new here.
> > > Ignite
> > > > > > > already
> > > > > > > > >> has a
> > > > > > > > >> > > > > > > deployment
> > > > > > > > >> > > > > > > >> SPI
> > > > > > > > >> > > > > > > >> > and it already works in a certain way.
> > Let's
> > > > not
> > > > > > > change
> > > > > > > > >> it.
> > > > > > > > >> > > > > > > >> >
> > > > > > > > >> > > > > > > >> > D.
> > > > > > > > >> > > > > > > >> >
> > > > > > > > >> > > > > > > >>
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Valentin Kulichenko]

Denis,

> On the other hand, if exception is thrown from the *execute() *method,
then service won't be undeployed.

This is actually weird... What is going to happen in this case and how user
would handle this?

-Val

On Fri, Apr 20, 2018 at 1:10 AM, Denis Mekhanikov <[hidden email]>
wrote:

> Val,
>
> *init()* method is executed before a service is considered deployed.
> If any exception is thrown from it, then it will be handled as deployment
> failure.
>
> *execute() *method is run after the service is deployed, and it can keep
> running until the service is cancelled.
> This method has its own thread, so it can perform some background work.
>
> Suppose you want to deploy HTTP server as a service on one of your nodes.
> You can place HTTP server creation logic in the *init() *method.
> If some nodes don't have a permission to listen to needed ports, then a
> corresponding exception will be propagated to the user code.
> On the other hand, if exception is thrown from the *execute() *method, then
> service won't be undeployed.
>
> Denis
>
> пт, 20 апр. 2018 г. в 2:35, Valentin Kulichenko <
> [hidden email]>:
>
> > Denis,
> >
> > I totally agree with you. I'm just not sure why do we need two methods
> > (init and execute) that have virtually same semantics. With the new
> design,
> > what would be the difference between these methods from user point of
> view,
> > and how one would determine what exactly should go in each of them?
> Unless
> > I'm missing something, it looks like unnecessary complication.
> >
> > -Val
> >
> > On Tue, Apr 17, 2018 at 1:00 AM, Denis Mekhanikov <[hidden email]
> >
> > wrote:
> >
> > > Val,
> > >
> > > Service initialisation is not going to happen in the discovery thread.
> > > It should be done asynchronously, and initialisation results should be
> > sent
> > > to the coordinator over communication.
> > > This is described in the IEP:
> > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> > > Successfulscenario
> > >
> > > *init()* method is a validation step, making sure, that service is
> ready
> > > for work.
> > > And deployment shouldn't be considered successful until *init()*
> methods
> > > finish their work on the assigned nodes.
> > > Also *cancel() *and *init() *methods may be useful if we decide to
> > > implement moving existing services to new nodes
> > > <https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> > > Movingexistingservicestonewnodes>
> > >  in
> > > future.
> > > These methods can be used to save and restore service's state from
> cache,
> > > when it is rebalanced to another node.
> > >
> > > As Denis said, if we are not going to prevent nodes from starting on
> > > service failures, then we should at least generate corresponding
> events.
> > > Otherwise there won't be any way to react to service initialization
> > > failures during nodes startup.
> > >
> > > Denis
> > >
> > > вт, 17 апр. 2018 г. в 6:59, Valentin Kulichenko <
> > > [hidden email]>:
> > >
> > > > I agree we shouldn't do anything synchronously within discovery
> > threads.
> > > If
> > > > something goes wrong, we just need to properly notify the user,
> logging
> > > and
> > > > events seem to be right options to achieve that.
> > > >
> > > > BTW, with this design I'm not sure init() method makes sense,
> probably
> > we
> > > > should deprecate it.
> > > >
> > > > -Val
> > > >
> > > > On Mon, Apr 16, 2018 at 12:03 PM, Denis Magda <[hidden email]>
> > wrote:
> > > >
> > > > > Denis,
> > > > >
> > > > > In general, service initialization shouldn't prevent a node from
> > > joining
> > > > > the cluster or slowing down that process. Thus, I would start the
> > > > > initialization routines only after the node is accepted by the
> > cluster.
> > > > If
> > > > > the initialization fails then we need to report a respective
> message
> > to
> > > > the
> > > > > logs and, probably, generate a system event the user can be
> > subscribed
> > > > to.
> > > > >
> > > > > Regardless, of the service initialization time, I think we still
> need
> > > to
> > > > > utilize discovery SPI to avoid problems discussed later.
> > > > >
> > > > > Val, others, what do you think about that?
> > > > >
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <
> > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Basically, my question is: at which moment services should be
> > > deployed
> > > > on
> > > > > > connecting nodes?
> > > > > > Should we reject a node from being included into a topology, if
> > > > services,
> > > > > > that are assigned to it, fail to deploy?
> > > > > >
> > > > > > It would be good to be sure, that all assigned services are
> > > initialised
> > > > > and
> > > > > > working, when node start is finished.
> > > > > > Otherwise it's unclear, how to notify a user about failures in
> > > service
> > > > > > initialisation on new nodes.
> > > > > >
> > > > > > If we decide to provide such guarantee, then how are we going to
> do
> > > > that?
> > > > > > Is procedure, that I described, viable?
> > > > > > It requires hacking through the discovery protocol, which is a
> > thing,
> > > > > that
> > > > > > should be avoided.
> > > > > > So, maybe there is another way to achieve the same thing?
> > > > > >
> > > > > > Denis
> > > > > >
> > > > > > сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:
> > > > > >
> > > > > > > It sounds like it's not a trivial thing to support the
> automatic
> > > > > services
> > > > > > > redeployment after a restart. Let's postpone it for now, guys
> > > > > > concentrating
> > > > > > > on more urgent things related to the services.
> > > > > > >
> > > > > > > Alex, Vladimir,
> > > > > > >
> > > > > > > Could you have a look at Denis question about the
> discovery-based
> > > > > > > deployment? Guess it's the only one thing that prevents us from
> > the
> > > > IEP
> > > > > > > finalization.
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <
> > > > > [hidden email]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Vladimir,
> > > > > > > >
> > > > > > > > Currently we don't save binary metadata to disk, when
> > persistence
> > > > is
> > > > > > > > disabled.
> > > > > > > > But we still persist marshaller mappings for some reason,
> and I
> > > > > > > personally
> > > > > > > > believe, that we shouldn't.
> > > > > > > >
> > > > > > > > But I agree, that we should separate data and service
> > persistence
> > > > > > > > configuration.
> > > > > > > > Right now persistence of services is configured in a pretty
> > > > > non-obvious
> > > > > > > > manner.
> > > > > > > > It should be a clear way to tell Ignite, whether you want
> > > services
> > > > to
> > > > > > be
> > > > > > > > persisted or not.
> > > > > > > >
> > > > > > > > I'm not sure, that we should make "statefullness" in general
> > > > > > > configurable.
> > > > > > > > Users don't care much, whether metadata is preserved on
> > restarts,
> > > > or
> > > > > > not.
> > > > > > > >
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <
> > > [hidden email]
> > > > >:
> > > > > > > >
> > > > > > > > > Alex,
> > > > > > > > >
> > > > > > > > > I would say that we've already had this behavior for years
> -
> > > > > > marshaller
> > > > > > > > > cache. I think it is time to agree that "in-memory" !=
> > > stateless.
> > > > > > > Instead
> > > > > > > > > "in-memory" means "data is not stored on disk".
> > > > > > > > > May be we can have a flag which will wipe out all metadata
> on
> > > > node
> > > > > > > > restart
> > > > > > > > > (e.g. it could make sense for embedded clients)?
> > > > > > > > >
> > > > > > > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > > > > > > > [hidden email]> wrote:
> > > > > > > > >
> > > > > > > > > > Denis,
> > > > > > > > > >
> > > > > > > > > > This is a subtle question. It looks like we have now a
> > number
> > > > of
> > > > > > > > > use-cases
> > > > > > > > > > when persistent storage is required even for a pure
> > in-memory
> > > > > mode.
> > > > > > > One
> > > > > > > > > of
> > > > > > > > > > the use-cases is thin client authentication, the other is
> > > > service
> > > > > > > grid
> > > > > > > > > > configuration persistence.
> > > > > > > > > >
> > > > > > > > > > Generally, I would agree that this is an expected
> behavior.
> > > > > > However,
> > > > > > > > this
> > > > > > > > > > means that a user cannot simply start and stop nodes
> > randomly
> > > > > > > anymore.
> > > > > > > > > > Ignite start will require some sort of installation or
> work
> > > > > folder
> > > > > > > > > > initialization (sort of initdb in postgres) which is ok
> for
> > > > > > > > > > persistence-enabled modes, but I am not sure if this is
> > > > expected
> > > > > > for
> > > > > > > > > > in-memory. Of course, we can run this initialization
> > > > > automatically,
> > > > > > > but
> > > > > > > > > it
> > > > > > > > > > is not always a good idea.
> > > > > > > > > >
> > > > > > > > > > If we are ok to have this restrictions for in-memory
> mode,
> > > then
> > > > > > > service
> > > > > > > > > > persistence makes sense.
> > > > > > > > > >
> > > > > > > > > > --AG
> > > > > > > > > >
> > > > > > > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <
> [hidden email]
> > >:
> > > > > > > > > >
> > > > > > > > > >> Denis,
> > > > > > > > > >>
> > > > > > > > > >> I think that the service deployment state needs be
> > persisted
> > > > > > > > > cluster-wide.
> > > > > > > > > >> I guess that our meta-store is capable of doing so. Alex
> > G.,
> > > > > > > Vladimir,
> > > > > > > > > >> could you confirm?
> > > > > > > > > >>
> > > > > > > > > >> As for the split-brain scenarios, I would put them aside
> > for
> > > > now
> > > > > > > > > because,
> > > > > > > > > >> anyway, they have to be solved at lower levels (meta
> > store,
> > > > > > > discovery,
> > > > > > > > > >> etc.).
> > > > > > > > > >>
> > > > > > > > > >> Also, I heard that presently we store a service
> > > configuration
> > > > in
> > > > > > the
> > > > > > > > > >> system
> > > > > > > > > >> cache that doesn't give us a way to deploy a new version
> > of
> > > a
> > > > > > > service
> > > > > > > > > >> without undeployment of the previous one. Will this
> issue
> > be
> > > > > > > addressed
> > > > > > > > > by
> > > > > > > > > >> the new deployment approach?
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> Denis
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > > > > > > > [hidden email]>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Denis,
> > > > > > > > > >> >
> > > > > > > > > >> > Sounds reasonable. It's not clear, though, what should
> > > > happen,
> > > > > > if
> > > > > > > a
> > > > > > > > > >> joining
> > > > > > > > > >> > node has some services persisted, that are missing on
> > > other
> > > > > > nodes.
> > > > > > > > > >> > Should we deploy them?
> > > > > > > > > >> > If we do so, it could lead to surprising behaviour.
> For
> > > > > example
> > > > > > > you
> > > > > > > > > >> could
> > > > > > > > > >> > kill a node, undeploy a service, then bring back an
> old
> > > > node,
> > > > > > and
> > > > > > > it
> > > > > > > > > >> would
> > > > > > > > > >> > make the service resurrect.
> > > > > > > > > >> > We could store some deployment counter along with the
> > > > service
> > > > > > > > > >> > configurations on all nodes, that would show how many
> > > times
> > > > > the
> > > > > > > > > service
> > > > > > > > > >> > state has changed, i.e. it has been
> > undeployed/redeployed.
> > > > It
> > > > > > > should
> > > > > > > > > be
> > > > > > > > > >> > kept for undeployed services as well to avoid
> situations
> > > > like
> > > > > I
> > > > > > > > > >> described.
> > > > > > > > > >> >
> > > > > > > > > >> > But it still leaves a possibility of incorrect
> > behaviour,
> > > if
> > > > > > there
> > > > > > > > > was a
> > > > > > > > > >> > split-brain situation at some point. I don't think we
> > > should
> > > > > > > precess
> > > > > > > > > it
> > > > > > > > > >> > somehow, though. If we choose to tackle it, it will
> > > > > > overcomplicate
> > > > > > > > > >> things
> > > > > > > > > >> > for a sake of a minor improvement.
> > > > > > > > > >> >
> > > > > > > > > >> > Denis
> > > > > > > > > >> >
> > > > > > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > > > > > > >> > [hidden email]>:
> > > > > > > > > >> >
> > > > > > > > > >> > > I was responding to another Denis :) Agree with you
> on
> > > > your
> > > > > > > point
> > > > > > > > > >> though.
> > > > > > > > > >> > >
> > > > > > > > > >> > > -Val
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> > > > > > [hidden email]>
> > > > > > > > > >> wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Val,
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Guess we're talking about other situations. I'm
> > > bringing
> > > > > up
> > > > > > > the
> > > > > > > > > case
> > > > > > > > > >> > > when a
> > > > > > > > > >> > > > service was deployed dynamically and has to be
> > brought
> > > > up
> > > > > > > after
> > > > > > > > a
> > > > > > > > > >> full
> > > > > > > > > >> > > > cluster restart w/o user intervention. To achieve
> > this
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > > >> > persist
> > > > > > > > > >> > > > the service's configuration somewhere.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > --
> > > > > > > > > >> > > > Denis
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin
> Kulichenko
> > <
> > > > > > > > > >> > > > [hidden email]> wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Denis,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a
> > > class
> > > > is
> > > > > > > > > deployed
> > > > > > > > > >> or
> > > > > > > > > >> > > > > redeployed. If this doesn't happen in some
> cases,
> > I
> > > > > > believe
> > > > > > > > this
> > > > > > > > > >> > would
> > > > > > > > > >> > > > be a
> > > > > > > > > >> > > > > bug. I don't think we need to add any new
> events.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > -Val
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > Denis,
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > I would encourage us to persist a service
> > > > > configuration
> > > > > > in
> > > > > > > > the
> > > > > > > > > >> meta
> > > > > > > > > >> > > > store
> > > > > > > > > >> > > > > > and have this capability enabled by default.
> > > That's
> > > > > > > > essential
> > > > > > > > > >> for
> > > > > > > > > >> > > > > services
> > > > > > > > > >> > > > > > started dynamically. Moreover, we support
> > similar
> > > > > > behavior
> > > > > > > > for
> > > > > > > > > >> > > caches,
> > > > > > > > > >> > > > > > indexes, and other DDL changes happened at
> > > runtime.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > --
> > > > > > > > > >> > > > > > Denis
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis
> > Mekhanikov <
> > > > > > > > > >> > > > [hidden email]>
> > > > > > > > > >> > > > > > wrote:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > Another question, that I would like to
> discuss
> > > is
> > > > > > > whether
> > > > > > > > > >> > services
> > > > > > > > > >> > > > > should
> > > > > > > > > >> > > > > > > be preserved on cluster restarts.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Currently it depends on persistence
> > > configuration.
> > > > > If
> > > > > > > > > >> persistence
> > > > > > > > > >> > > for
> > > > > > > > > >> > > > > any
> > > > > > > > > >> > > > > > > data region is enabled, then services will
> be
> > > > > > persisted
> > > > > > > as
> > > > > > > > > >> well.
> > > > > > > > > >> > > This
> > > > > > > > > >> > > > > is
> > > > > > > > > >> > > > > > a
> > > > > > > > > >> > > > > > > pretty strange way of configuring this
> > > behaviour.
> > > > > > > > > >> > > > > > > I'm not sure, if anybody relies on this
> > > > > functionality
> > > > > > > > right
> > > > > > > > > >> now.
> > > > > > > > > >> > > > Should
> > > > > > > > > >> > > > > > we
> > > > > > > > > >> > > > > > > support it at all? If yes, should we make it
> > > > > > > configurable?
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Denis
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis
> Mekhanikov <
> > > > > > > > > >> > > [hidden email]
> > > > > > > > > >> > > > >:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > > Val,
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Sounds reasonable. I just think, that user
> > > > should
> > > > > > have
> > > > > > > > > some
> > > > > > > > > >> way
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > know,
> > > > > > > > > >> > > > > > > > that new version of a service class was
> > > > deployed.
> > > > > > > > > >> > > > > > > > One way to do it is to listen to
> > > > > > *EVT_CLASS_DEPLOYED.
> > > > > > > > *I'm
> > > > > > > > > >> not
> > > > > > > > > >> > > > sure,
> > > > > > > > > >> > > > > > > > whether it is triggered on class
> > redeployment,
> > > > > > though.
> > > > > > > > If
> > > > > > > > > >> not,
> > > > > > > > > >> > > then
> > > > > > > > > >> > > > > > > another
> > > > > > > > > >> > > > > > > > event type should be added.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > I don't think, that a lot of people will
> > > > implement
> > > > > > > their
> > > > > > > > > own
> > > > > > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make work
> > with
> > > > > > > > > >> > *UriDeploymentSpi*
> > > > > > > > > >> > > > as
> > > > > > > > > >> > > > > > > > comfortable as possible.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Denis
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin
> > > Kulichenko
> > > > <
> > > > > > > > > >> > > > > > > > [hidden email]>:
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > >> Yes, the class deployment itself has to
> be
> > > > > > explicit.
> > > > > > > > > I.e.,
> > > > > > > > > >> > there
> > > > > > > > > >> > > > has
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > be
> > > > > > > > > >> > > > > > > >> a manual step where user updates the
> class,
> > > and
> > > > > the
> > > > > > > > exact
> > > > > > > > > >> step
> > > > > > > > > >> > > > > > required
> > > > > > > > > >> > > > > > > >> would depend on DeploymentSpi
> > implementation.
> > > > But
> > > > > > > then
> > > > > > > > > >> Ignite
> > > > > > > > > >> > > > takes
> > > > > > > > > >> > > > > > care
> > > > > > > > > >> > > > > > > >> of
> > > > > > > > > >> > > > > > > >> everything else - service redeployment
> and
> > > > > restart
> > > > > > is
> > > > > > > > > >> > automatic.
> > > > > > > > > >> > > > > > > >>
> > > > > > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be
> > > > disabled
> > > > > if
> > > > > > > > > >> > > DeploymentSpi
> > > > > > > > > >> > > > is
> > > > > > > > > >> > > > > > not
> > > > > > > > > >> > > > > > > >> configured. In this case service class
> > > > > definitions
> > > > > > > have
> > > > > > > > > to
> > > > > > > > > >> be
> > > > > > > > > >> > > > > deployed
> > > > > > > > > >> > > > > > > on
> > > > > > > > > >> > > > > > > >> local classpath and can't be updated in
> > > > runtime.
> > > > > > Just
> > > > > > > > > like
> > > > > > > > > >> it
> > > > > > > > > >> > > > works
> > > > > > > > > >> > > > > > > right
> > > > > > > > > >> > > > > > > >> now.
> > > > > > > > > >> > > > > > > >>
> > > > > > > > > >> > > > > > > >> -Val
> > > > > > > > > >> > > > > > > >>
> > > > > > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy
> > > > > Setrakyan
> > > > > > <
> > > > > > > > > >> > > > > > > [hidden email]
> > > > > > > > > >> > > > > > > >> >
> > > > > > > > > >> > > > > > > >> wrote:
> > > > > > > > > >> > > > > > > >>
> > > > > > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry
> > > > Pavlov <
> > > > > > > > > >> > > > > > [hidden email]>
> > > > > > > > > >> > > > > > > >> > wrote:
> > > > > > > > > >> > > > > > > >> >
> > > > > > > > > >> > > > > > > >> > > Hi Igniters,
> > > > > > > > > >> > > > > > > >> > >
> > > > > > > > > >> > > > > > > >> > > I like automatic redeploy which can
> be
> > > > > disabled
> > > > > > > by
> > > > > > > > > >> config
> > > > > > > > > >> > if
> > > > > > > > > >> > > > > user
> > > > > > > > > >> > > > > > > >> wants
> > > > > > > > > >> > > > > > > >> > to
> > > > > > > > > >> > > > > > > >> > > control this process. What do you
> > think?
> > > > > > > > > >> > > > > > > >> > >
> > > > > > > > > >> > > > > > > >> >
> > > > > > > > > >> > > > > > > >> > I do not think we should have anything
> > > > > automatic
> > > > > > > when
> > > > > > > > > it
> > > > > > > > > >> > comes
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > > >> > deployment, everything should be
> > explicit.
> > > > > > However,
> > > > > > > > if
> > > > > > > > > we
> > > > > > > > > >> > use
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > >> > deployment SPI, then a user should be
> > able
> > > to
> > > > > do
> > > > > > > > "hot"
> > > > > > > > > >> > > redeploy,
> > > > > > > > > >> > > > > > > where a
> > > > > > > > > >> > > > > > > >> > new service will be deployed if the
> user
> > > > drops
> > > > > an
> > > > > > > > > updated
> > > > > > > > > >> > jar.
> > > > > > > > > >> > > > > > > >> >
> > > > > > > > > >> > > > > > > >> > We should not create anything new here.
> > > > Ignite
> > > > > > > > already
> > > > > > > > > >> has a
> > > > > > > > > >> > > > > > > deployment
> > > > > > > > > >> > > > > > > >> SPI
> > > > > > > > > >> > > > > > > >> > and it already works in a certain way.
> > > Let's
> > > > > not
> > > > > > > > change
> > > > > > > > > >> it.
> > > > > > > > > >> > > > > > > >> >
> > > > > > > > > >> > > > > > > >> > D.
> > > > > > > > > >> > > > > > > >> >
> > > > > > > > > >> > > > > > > >>
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

I am not sure why we are discussing a potential removal of the "init"
method. I think it is useful, as the service may have to do some
initialization before it goes online. I do not think this method is hurting
anyone.

This thread is getting too long, and I am sure that most readers are
already getting lost in the proposed design. I would start a new thread
with a summary of all proposed changes.

D.

On Fri, Apr 20, 2018 at 5:25 PM, Valentin Kulichenko <
[hidden email]> wrote:

> Denis,
>
> > On the other hand, if exception is thrown from the *execute() *method,
> then service won't be undeployed.
>
> This is actually weird... What is going to happen in this case and how user
> would handle this?
>
> -Val
>
> On Fri, Apr 20, 2018 at 1:10 AM, Denis Mekhanikov <[hidden email]>
> wrote:
>
> > Val,
> >
> > *init()* method is executed before a service is considered deployed.
> > If any exception is thrown from it, then it will be handled as deployment
> > failure.
> >
> > *execute() *method is run after the service is deployed, and it can keep
> > running until the service is cancelled.
> > This method has its own thread, so it can perform some background work.
> >
> > Suppose you want to deploy HTTP server as a service on one of your nodes.
> > You can place HTTP server creation logic in the *init() *method.
> > If some nodes don't have a permission to listen to needed ports, then a
> > corresponding exception will be propagated to the user code.
> > On the other hand, if exception is thrown from the *execute() *method,
> then
> > service won't be undeployed.
> >
> > Denis
> >
> > пт, 20 апр. 2018 г. в 2:35, Valentin Kulichenko <
> > [hidden email]>:
> >
> > > Denis,
> > >
> > > I totally agree with you. I'm just not sure why do we need two methods
> > > (init and execute) that have virtually same semantics. With the new
> > design,
> > > what would be the difference between these methods from user point of
> > view,
> > > and how one would determine what exactly should go in each of them?
> > Unless
> > > I'm missing something, it looks like unnecessary complication.
> > >
> > > -Val
> > >
> > > On Tue, Apr 17, 2018 at 1:00 AM, Denis Mekhanikov <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Val,
> > > >
> > > > Service initialisation is not going to happen in the discovery
> thread.
> > > > It should be done asynchronously, and initialisation results should
> be
> > > sent
> > > > to the coordinator over communication.
> > > > This is described in the IEP:
> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > > 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> > > > Successfulscenario
> > > >
> > > > *init()* method is a validation step, making sure, that service is
> > ready
> > > > for work.
> > > > And deployment shouldn't be considered successful until *init()*
> > methods
> > > > finish their work on the assigned nodes.
> > > > Also *cancel() *and *init() *methods may be useful if we decide to
> > > > implement moving existing services to new nodes
> > > > <https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > > 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> > > > Movingexistingservicestonewnodes>
> > > >  in
> > > > future.
> > > > These methods can be used to save and restore service's state from
> > cache,
> > > > when it is rebalanced to another node.
> > > >
> > > > As Denis said, if we are not going to prevent nodes from starting on
> > > > service failures, then we should at least generate corresponding
> > events.
> > > > Otherwise there won't be any way to react to service initialization
> > > > failures during nodes startup.
> > > >
> > > > Denis
> > > >
> > > > вт, 17 апр. 2018 г. в 6:59, Valentin Kulichenko <
> > > > [hidden email]>:
> > > >
> > > > > I agree we shouldn't do anything synchronously within discovery
> > > threads.
> > > > If
> > > > > something goes wrong, we just need to properly notify the user,
> > logging
> > > > and
> > > > > events seem to be right options to achieve that.
> > > > >
> > > > > BTW, with this design I'm not sure init() method makes sense,
> > probably
> > > we
> > > > > should deprecate it.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Mon, Apr 16, 2018 at 12:03 PM, Denis Magda <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Denis,
> > > > > >
> > > > > > In general, service initialization shouldn't prevent a node from
> > > > joining
> > > > > > the cluster or slowing down that process. Thus, I would start the
> > > > > > initialization routines only after the node is accepted by the
> > > cluster.
> > > > > If
> > > > > > the initialization fails then we need to report a respective
> > message
> > > to
> > > > > the
> > > > > > logs and, probably, generate a system event the user can be
> > > subscribed
> > > > > to.
> > > > > >
> > > > > > Regardless, of the service initialization time, I think we still
> > need
> > > > to
> > > > > > utilize discovery SPI to avoid problems discussed later.
> > > > > >
> > > > > > Val, others, what do you think about that?
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Denis
> > > > > >
> > > > > > On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <
> > > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Basically, my question is: at which moment services should be
> > > > deployed
> > > > > on
> > > > > > > connecting nodes?
> > > > > > > Should we reject a node from being included into a topology, if
> > > > > services,
> > > > > > > that are assigned to it, fail to deploy?
> > > > > > >
> > > > > > > It would be good to be sure, that all assigned services are
> > > > initialised
> > > > > > and
> > > > > > > working, when node start is finished.
> > > > > > > Otherwise it's unclear, how to notify a user about failures in
> > > > service
> > > > > > > initialisation on new nodes.
> > > > > > >
> > > > > > > If we decide to provide such guarantee, then how are we going
> to
> > do
> > > > > that?
> > > > > > > Is procedure, that I described, viable?
> > > > > > > It requires hacking through the discovery protocol, which is a
> > > thing,
> > > > > > that
> > > > > > > should be avoided.
> > > > > > > So, maybe there is another way to achieve the same thing?
> > > > > > >
> > > > > > > Denis
> > > > > > >
> > > > > > > сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:
> > > > > > >
> > > > > > > > It sounds like it's not a trivial thing to support the
> > automatic
> > > > > > services
> > > > > > > > redeployment after a restart. Let's postpone it for now, guys
> > > > > > > concentrating
> > > > > > > > on more urgent things related to the services.
> > > > > > > >
> > > > > > > > Alex, Vladimir,
> > > > > > > >
> > > > > > > > Could you have a look at Denis question about the
> > discovery-based
> > > > > > > > deployment? Guess it's the only one thing that prevents us
> from
> > > the
> > > > > IEP
> > > > > > > > finalization.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Vladimir,
> > > > > > > > >
> > > > > > > > > Currently we don't save binary metadata to disk, when
> > > persistence
> > > > > is
> > > > > > > > > disabled.
> > > > > > > > > But we still persist marshaller mappings for some reason,
> > and I
> > > > > > > > personally
> > > > > > > > > believe, that we shouldn't.
> > > > > > > > >
> > > > > > > > > But I agree, that we should separate data and service
> > > persistence
> > > > > > > > > configuration.
> > > > > > > > > Right now persistence of services is configured in a pretty
> > > > > > non-obvious
> > > > > > > > > manner.
> > > > > > > > > It should be a clear way to tell Ignite, whether you want
> > > > services
> > > > > to
> > > > > > > be
> > > > > > > > > persisted or not.
> > > > > > > > >
> > > > > > > > > I'm not sure, that we should make "statefullness" in
> general
> > > > > > > > configurable.
> > > > > > > > > Users don't care much, whether metadata is preserved on
> > > restarts,
> > > > > or
> > > > > > > not.
> > > > > > > > >
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <
> > > > [hidden email]
> > > > > >:
> > > > > > > > >
> > > > > > > > > > Alex,
> > > > > > > > > >
> > > > > > > > > > I would say that we've already had this behavior for
> years
> > -
> > > > > > > marshaller
> > > > > > > > > > cache. I think it is time to agree that "in-memory" !=
> > > > stateless.
> > > > > > > > Instead
> > > > > > > > > > "in-memory" means "data is not stored on disk".
> > > > > > > > > > May be we can have a flag which will wipe out all
> metadata
> > on
> > > > > node
> > > > > > > > > restart
> > > > > > > > > > (e.g. it could make sense for embedded clients)?
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > > > > > > > > [hidden email]> wrote:
> > > > > > > > > >
> > > > > > > > > > > Denis,
> > > > > > > > > > >
> > > > > > > > > > > This is a subtle question. It looks like we have now a
> > > number
> > > > > of
> > > > > > > > > > use-cases
> > > > > > > > > > > when persistent storage is required even for a pure
> > > in-memory
> > > > > > mode.
> > > > > > > > One
> > > > > > > > > > of
> > > > > > > > > > > the use-cases is thin client authentication, the other
> is
> > > > > service
> > > > > > > > grid
> > > > > > > > > > > configuration persistence.
> > > > > > > > > > >
> > > > > > > > > > > Generally, I would agree that this is an expected
> > behavior.
> > > > > > > However,
> > > > > > > > > this
> > > > > > > > > > > means that a user cannot simply start and stop nodes
> > > randomly
> > > > > > > > anymore.
> > > > > > > > > > > Ignite start will require some sort of installation or
> > work
> > > > > > folder
> > > > > > > > > > > initialization (sort of initdb in postgres) which is ok
> > for
> > > > > > > > > > > persistence-enabled modes, but I am not sure if this is
> > > > > expected
> > > > > > > for
> > > > > > > > > > > in-memory. Of course, we can run this initialization
> > > > > > automatically,
> > > > > > > > but
> > > > > > > > > > it
> > > > > > > > > > > is not always a good idea.
> > > > > > > > > > >
> > > > > > > > > > > If we are ok to have this restrictions for in-memory
> > mode,
> > > > then
> > > > > > > > service
> > > > > > > > > > > persistence makes sense.
> > > > > > > > > > >
> > > > > > > > > > > --AG
> > > > > > > > > > >
> > > > > > > > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <
> > [hidden email]
> > > >:
> > > > > > > > > > >
> > > > > > > > > > >> Denis,
> > > > > > > > > > >>
> > > > > > > > > > >> I think that the service deployment state needs be
> > > persisted
> > > > > > > > > > cluster-wide.
> > > > > > > > > > >> I guess that our meta-store is capable of doing so.
> Alex
> > > G.,
> > > > > > > > Vladimir,
> > > > > > > > > > >> could you confirm?
> > > > > > > > > > >>
> > > > > > > > > > >> As for the split-brain scenarios, I would put them
> aside
> > > for
> > > > > now
> > > > > > > > > > because,
> > > > > > > > > > >> anyway, they have to be solved at lower levels (meta
> > > store,
> > > > > > > > discovery,
> > > > > > > > > > >> etc.).
> > > > > > > > > > >>
> > > > > > > > > > >> Also, I heard that presently we store a service
> > > > configuration
> > > > > in
> > > > > > > the
> > > > > > > > > > >> system
> > > > > > > > > > >> cache that doesn't give us a way to deploy a new
> version
> > > of
> > > > a
> > > > > > > > service
> > > > > > > > > > >> without undeployment of the previous one. Will this
> > issue
> > > be
> > > > > > > > addressed
> > > > > > > > > > by
> > > > > > > > > > >> the new deployment approach?
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> Denis
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Denis,
> > > > > > > > > > >> >
> > > > > > > > > > >> > Sounds reasonable. It's not clear, though, what
> should
> > > > > happen,
> > > > > > > if
> > > > > > > > a
> > > > > > > > > > >> joining
> > > > > > > > > > >> > node has some services persisted, that are missing
> on
> > > > other
> > > > > > > nodes.
> > > > > > > > > > >> > Should we deploy them?
> > > > > > > > > > >> > If we do so, it could lead to surprising behaviour.
> > For
> > > > > > example
> > > > > > > > you
> > > > > > > > > > >> could
> > > > > > > > > > >> > kill a node, undeploy a service, then bring back an
> > old
> > > > > node,
> > > > > > > and
> > > > > > > > it
> > > > > > > > > > >> would
> > > > > > > > > > >> > make the service resurrect.
> > > > > > > > > > >> > We could store some deployment counter along with
> the
> > > > > service
> > > > > > > > > > >> > configurations on all nodes, that would show how
> many
> > > > times
> > > > > > the
> > > > > > > > > > service
> > > > > > > > > > >> > state has changed, i.e. it has been
> > > undeployed/redeployed.
> > > > > It
> > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > >> > kept for undeployed services as well to avoid
> > situations
> > > > > like
> > > > > > I
> > > > > > > > > > >> described.
> > > > > > > > > > >> >
> > > > > > > > > > >> > But it still leaves a possibility of incorrect
> > > behaviour,
> > > > if
> > > > > > > there
> > > > > > > > > > was a
> > > > > > > > > > >> > split-brain situation at some point. I don't think
> we
> > > > should
> > > > > > > > precess
> > > > > > > > > > it
> > > > > > > > > > >> > somehow, though. If we choose to tackle it, it will
> > > > > > > overcomplicate
> > > > > > > > > > >> things
> > > > > > > > > > >> > for a sake of a minor improvement.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Denis
> > > > > > > > > > >> >
> > > > > > > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > > > > > > > >> > [hidden email]>:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > I was responding to another Denis :) Agree with
> you
> > on
> > > > > your
> > > > > > > > point
> > > > > > > > > > >> though.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > -Val
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> > > > > > > [hidden email]>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Val,
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Guess we're talking about other situations. I'm
> > > > bringing
> > > > > > up
> > > > > > > > the
> > > > > > > > > > case
> > > > > > > > > > >> > > when a
> > > > > > > > > > >> > > > service was deployed dynamically and has to be
> > > brought
> > > > > up
> > > > > > > > after
> > > > > > > > > a
> > > > > > > > > > >> full
> > > > > > > > > > >> > > > cluster restart w/o user intervention. To
> achieve
> > > this
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > >> > persist
> > > > > > > > > > >> > > > the service's configuration somewhere.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > --
> > > > > > > > > > >> > > > Denis
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin
> > Kulichenko
> > > <
> > > > > > > > > > >> > > > [hidden email]> wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Denis,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time
> a
> > > > class
> > > > > is
> > > > > > > > > > deployed
> > > > > > > > > > >> or
> > > > > > > > > > >> > > > > redeployed. If this doesn't happen in some
> > cases,
> > > I
> > > > > > > believe
> > > > > > > > > this
> > > > > > > > > > >> > would
> > > > > > > > > > >> > > > be a
> > > > > > > > > > >> > > > > bug. I don't think we need to add any new
> > events.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > -Val
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > Denis,
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > I would encourage us to persist a service
> > > > > > configuration
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >> meta
> > > > > > > > > > >> > > > store
> > > > > > > > > > >> > > > > > and have this capability enabled by default.
> > > > That's
> > > > > > > > > essential
> > > > > > > > > > >> for
> > > > > > > > > > >> > > > > services
> > > > > > > > > > >> > > > > > started dynamically. Moreover, we support
> > > similar
> > > > > > > behavior
> > > > > > > > > for
> > > > > > > > > > >> > > caches,
> > > > > > > > > > >> > > > > > indexes, and other DDL changes happened at
> > > > runtime.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > --
> > > > > > > > > > >> > > > > > Denis
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis
> > > Mekhanikov <
> > > > > > > > > > >> > > > [hidden email]>
> > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > Another question, that I would like to
> > discuss
> > > > is
> > > > > > > > whether
> > > > > > > > > > >> > services
> > > > > > > > > > >> > > > > should
> > > > > > > > > > >> > > > > > > be preserved on cluster restarts.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Currently it depends on persistence
> > > > configuration.
> > > > > > If
> > > > > > > > > > >> persistence
> > > > > > > > > > >> > > for
> > > > > > > > > > >> > > > > any
> > > > > > > > > > >> > > > > > > data region is enabled, then services will
> > be
> > > > > > > persisted
> > > > > > > > as
> > > > > > > > > > >> well.
> > > > > > > > > > >> > > This
> > > > > > > > > > >> > > > > is
> > > > > > > > > > >> > > > > > a
> > > > > > > > > > >> > > > > > > pretty strange way of configuring this
> > > > behaviour.
> > > > > > > > > > >> > > > > > > I'm not sure, if anybody relies on this
> > > > > > functionality
> > > > > > > > > right
> > > > > > > > > > >> now.
> > > > > > > > > > >> > > > Should
> > > > > > > > > > >> > > > > > we
> > > > > > > > > > >> > > > > > > support it at all? If yes, should we make
> it
> > > > > > > > configurable?
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Denis
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis
> > Mekhanikov <
> > > > > > > > > > >> > > [hidden email]
> > > > > > > > > > >> > > > >:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > > Val,
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > Sounds reasonable. I just think, that
> user
> > > > > should
> > > > > > > have
> > > > > > > > > > some
> > > > > > > > > > >> way
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > know,
> > > > > > > > > > >> > > > > > > > that new version of a service class was
> > > > > deployed.
> > > > > > > > > > >> > > > > > > > One way to do it is to listen to
> > > > > > > *EVT_CLASS_DEPLOYED.
> > > > > > > > > *I'm
> > > > > > > > > > >> not
> > > > > > > > > > >> > > > sure,
> > > > > > > > > > >> > > > > > > > whether it is triggered on class
> > > redeployment,
> > > > > > > though.
> > > > > > > > > If
> > > > > > > > > > >> not,
> > > > > > > > > > >> > > then
> > > > > > > > > > >> > > > > > > another
> > > > > > > > > > >> > > > > > > > event type should be added.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > I don't think, that a lot of people will
> > > > > implement
> > > > > > > > their
> > > > > > > > > > own
> > > > > > > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make
> work
> > > with
> > > > > > > > > > >> > *UriDeploymentSpi*
> > > > > > > > > > >> > > > as
> > > > > > > > > > >> > > > > > > > comfortable as possible.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > Denis
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin
> > > > Kulichenko
> > > > > <
> > > > > > > > > > >> > > > > > > > [hidden email]>:
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > >> Yes, the class deployment itself has to
> > be
> > > > > > > explicit.
> > > > > > > > > > I.e.,
> > > > > > > > > > >> > there
> > > > > > > > > > >> > > > has
> > > > > > > > > > >> > > > > > to
> > > > > > > > > > >> > > > > > > be
> > > > > > > > > > >> > > > > > > >> a manual step where user updates the
> > class,
> > > > and
> > > > > > the
> > > > > > > > > exact
> > > > > > > > > > >> step
> > > > > > > > > > >> > > > > > required
> > > > > > > > > > >> > > > > > > >> would depend on DeploymentSpi
> > > implementation.
> > > > > But
> > > > > > > > then
> > > > > > > > > > >> Ignite
> > > > > > > > > > >> > > > takes
> > > > > > > > > > >> > > > > > care
> > > > > > > > > > >> > > > > > > >> of
> > > > > > > > > > >> > > > > > > >> everything else - service redeployment
> > and
> > > > > > restart
> > > > > > > is
> > > > > > > > > > >> > automatic.
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be
> > > > > disabled
> > > > > > if
> > > > > > > > > > >> > > DeploymentSpi
> > > > > > > > > > >> > > > is
> > > > > > > > > > >> > > > > > not
> > > > > > > > > > >> > > > > > > >> configured. In this case service class
> > > > > > definitions
> > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > >> be
> > > > > > > > > > >> > > > > deployed
> > > > > > > > > > >> > > > > > > on
> > > > > > > > > > >> > > > > > > >> local classpath and can't be updated in
> > > > > runtime.
> > > > > > > Just
> > > > > > > > > > like
> > > > > > > > > > >> it
> > > > > > > > > > >> > > > works
> > > > > > > > > > >> > > > > > > right
> > > > > > > > > > >> > > > > > > >> now.
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >> -Val
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM,
> Dmitriy
> > > > > > Setrakyan
> > > > > > > <
> > > > > > > > > > >> > > > > > > [hidden email]
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> wrote:
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM,
> Dmitry
> > > > > Pavlov <
> > > > > > > > > > >> > > > > > [hidden email]>
> > > > > > > > > > >> > > > > > > >> > wrote:
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> > > Hi Igniters,
> > > > > > > > > > >> > > > > > > >> > >
> > > > > > > > > > >> > > > > > > >> > > I like automatic redeploy which can
> > be
> > > > > > disabled
> > > > > > > > by
> > > > > > > > > > >> config
> > > > > > > > > > >> > if
> > > > > > > > > > >> > > > > user
> > > > > > > > > > >> > > > > > > >> wants
> > > > > > > > > > >> > > > > > > >> > to
> > > > > > > > > > >> > > > > > > >> > > control this process. What do you
> > > think?
> > > > > > > > > > >> > > > > > > >> > >
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> > I do not think we should have
> anything
> > > > > > automatic
> > > > > > > > when
> > > > > > > > > > it
> > > > > > > > > > >> > comes
> > > > > > > > > > >> > > > to
> > > > > > > > > > >> > > > > > > >> > deployment, everything should be
> > > explicit.
> > > > > > > However,
> > > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > >> > use
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > >> > deployment SPI, then a user should be
> > > able
> > > > to
> > > > > > do
> > > > > > > > > "hot"
> > > > > > > > > > >> > > redeploy,
> > > > > > > > > > >> > > > > > > where a
> > > > > > > > > > >> > > > > > > >> > new service will be deployed if the
> > user
> > > > > drops
> > > > > > an
> > > > > > > > > > updated
> > > > > > > > > > >> > jar.
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> > We should not create anything new
> here.
> > > > > Ignite
> > > > > > > > > already
> > > > > > > > > > >> has a
> > > > > > > > > > >> > > > > > > deployment
> > > > > > > > > > >> > > > > > > >> SPI
> > > > > > > > > > >> > > > > > > >> > and it already works in a certain
> way.
> > > > Let's
> > > > > > not
> > > > > > > > > change
> > > > > > > > > > >> it.
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> > D.
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dmagda]

Dmitriy,

Consider IEP page as a summary that was updated along the way:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-17%3A+Oil+Change+in+Service+Grid

As far as I understand, Denis is going to create JIRA tickets basing on the
discussion results.

--
Denis

On Sat, Apr 21, 2018 at 3:04 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> I am not sure why we are discussing a potential removal of the "init"
> method. I think it is useful, as the service may have to do some
> initialization before it goes online. I do not think this method is hurting
> anyone.
>
> This thread is getting too long, and I am sure that most readers are
> already getting lost in the proposed design. I would start a new thread
> with a summary of all proposed changes.
>
> D.
>
> On Fri, Apr 20, 2018 at 5:25 PM, Valentin Kulichenko <
> [hidden email]> wrote:
>
> > Denis,
> >
> > > On the other hand, if exception is thrown from the *execute() *method,
> > then service won't be undeployed.
> >
> > This is actually weird... What is going to happen in this case and how
> user
> > would handle this?
> >
> > -Val
> >
> > On Fri, Apr 20, 2018 at 1:10 AM, Denis Mekhanikov <[hidden email]
> >
> > wrote:
> >
> > > Val,
> > >
> > > *init()* method is executed before a service is considered deployed.
> > > If any exception is thrown from it, then it will be handled as
> deployment
> > > failure.
> > >
> > > *execute() *method is run after the service is deployed, and it can
> keep
> > > running until the service is cancelled.
> > > This method has its own thread, so it can perform some background work.
> > >
> > > Suppose you want to deploy HTTP server as a service on one of your
> nodes.
> > > You can place HTTP server creation logic in the *init() *method.
> > > If some nodes don't have a permission to listen to needed ports, then a
> > > corresponding exception will be propagated to the user code.
> > > On the other hand, if exception is thrown from the *execute() *method,
> > then
> > > service won't be undeployed.
> > >
> > > Denis
> > >
> > > пт, 20 апр. 2018 г. в 2:35, Valentin Kulichenko <
> > > [hidden email]>:
> > >
> > > > Denis,
> > > >
> > > > I totally agree with you. I'm just not sure why do we need two
> methods
> > > > (init and execute) that have virtually same semantics. With the new
> > > design,
> > > > what would be the difference between these methods from user point of
> > > view,
> > > > and how one would determine what exactly should go in each of them?
> > > Unless
> > > > I'm missing something, it looks like unnecessary complication.
> > > >
> > > > -Val
> > > >
> > > > On Tue, Apr 17, 2018 at 1:00 AM, Denis Mekhanikov <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Val,
> > > > >
> > > > > Service initialisation is not going to happen in the discovery
> > thread.
> > > > > It should be done asynchronously, and initialisation results should
> > be
> > > > sent
> > > > > to the coordinator over communication.
> > > > > This is described in the IEP:
> > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > > > 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> > > > > Successfulscenario
> > > > >
> > > > > *init()* method is a validation step, making sure, that service is
> > > ready
> > > > > for work.
> > > > > And deployment shouldn't be considered successful until *init()*
> > > methods
> > > > > finish their work on the assigned nodes.
> > > > > Also *cancel() *and *init() *methods may be useful if we decide to
> > > > > implement moving existing services to new nodes
> > > > > <https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > > > 17%3A+Oil+Change+in+Service+Grid#IEP-17:OilChangeinServiceGrid-
> > > > > Movingexistingservicestonewnodes>
> > > > >  in
> > > > > future.
> > > > > These methods can be used to save and restore service's state from
> > > cache,
> > > > > when it is rebalanced to another node.
> > > > >
> > > > > As Denis said, if we are not going to prevent nodes from starting
> on
> > > > > service failures, then we should at least generate corresponding
> > > events.
> > > > > Otherwise there won't be any way to react to service initialization
> > > > > failures during nodes startup.
> > > > >
> > > > > Denis
> > > > >
> > > > > вт, 17 апр. 2018 г. в 6:59, Valentin Kulichenko <
> > > > > [hidden email]>:
> > > > >
> > > > > > I agree we shouldn't do anything synchronously within discovery
> > > > threads.
> > > > > If
> > > > > > something goes wrong, we just need to properly notify the user,
> > > logging
> > > > > and
> > > > > > events seem to be right options to achieve that.
> > > > > >
> > > > > > BTW, with this design I'm not sure init() method makes sense,
> > > probably
> > > > we
> > > > > > should deprecate it.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Mon, Apr 16, 2018 at 12:03 PM, Denis Magda <[hidden email]
> >
> > > > wrote:
> > > > > >
> > > > > > > Denis,
> > > > > > >
> > > > > > > In general, service initialization shouldn't prevent a node
> from
> > > > > joining
> > > > > > > the cluster or slowing down that process. Thus, I would start
> the
> > > > > > > initialization routines only after the node is accepted by the
> > > > cluster.
> > > > > > If
> > > > > > > the initialization fails then we need to report a respective
> > > message
> > > > to
> > > > > > the
> > > > > > > logs and, probably, generate a system event the user can be
> > > > subscribed
> > > > > > to.
> > > > > > >
> > > > > > > Regardless, of the service initialization time, I think we
> still
> > > need
> > > > > to
> > > > > > > utilize discovery SPI to avoid problems discussed later.
> > > > > > >
> > > > > > > Val, others, what do you think about that?
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > > On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <
> > > > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Basically, my question is: at which moment services should be
> > > > > deployed
> > > > > > on
> > > > > > > > connecting nodes?
> > > > > > > > Should we reject a node from being included into a topology,
> if
> > > > > > services,
> > > > > > > > that are assigned to it, fail to deploy?
> > > > > > > >
> > > > > > > > It would be good to be sure, that all assigned services are
> > > > > initialised
> > > > > > > and
> > > > > > > > working, when node start is finished.
> > > > > > > > Otherwise it's unclear, how to notify a user about failures
> in
> > > > > service
> > > > > > > > initialisation on new nodes.
> > > > > > > >
> > > > > > > > If we decide to provide such guarantee, then how are we going
> > to
> > > do
> > > > > > that?
> > > > > > > > Is procedure, that I described, viable?
> > > > > > > > It requires hacking through the discovery protocol, which is
> a
> > > > thing,
> > > > > > > that
> > > > > > > > should be avoided.
> > > > > > > > So, maybe there is another way to achieve the same thing?
> > > > > > > >
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > сб, 14 апр. 2018 г. в 1:48, Denis Magda <[hidden email]>:
> > > > > > > >
> > > > > > > > > It sounds like it's not a trivial thing to support the
> > > automatic
> > > > > > > services
> > > > > > > > > redeployment after a restart. Let's postpone it for now,
> guys
> > > > > > > > concentrating
> > > > > > > > > on more urgent things related to the services.
> > > > > > > > >
> > > > > > > > > Alex, Vladimir,
> > > > > > > > >
> > > > > > > > > Could you have a look at Denis question about the
> > > discovery-based
> > > > > > > > > deployment? Guess it's the only one thing that prevents us
> > from
> > > > the
> > > > > > IEP
> > > > > > > > > finalization.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Vladimir,
> > > > > > > > > >
> > > > > > > > > > Currently we don't save binary metadata to disk, when
> > > > persistence
> > > > > > is
> > > > > > > > > > disabled.
> > > > > > > > > > But we still persist marshaller mappings for some reason,
> > > and I
> > > > > > > > > personally
> > > > > > > > > > believe, that we shouldn't.
> > > > > > > > > >
> > > > > > > > > > But I agree, that we should separate data and service
> > > > persistence
> > > > > > > > > > configuration.
> > > > > > > > > > Right now persistence of services is configured in a
> pretty
> > > > > > > non-obvious
> > > > > > > > > > manner.
> > > > > > > > > > It should be a clear way to tell Ignite, whether you want
> > > > > services
> > > > > > to
> > > > > > > > be
> > > > > > > > > > persisted or not.
> > > > > > > > > >
> > > > > > > > > > I'm not sure, that we should make "statefullness" in
> > general
> > > > > > > > > configurable.
> > > > > > > > > > Users don't care much, whether metadata is preserved on
> > > > restarts,
> > > > > > or
> > > > > > > > not.
> > > > > > > > > >
> > > > > > > > > > Denis
> > > > > > > > > >
> > > > > > > > > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <
> > > > > [hidden email]
> > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Alex,
> > > > > > > > > > >
> > > > > > > > > > > I would say that we've already had this behavior for
> > years
> > > -
> > > > > > > > marshaller
> > > > > > > > > > > cache. I think it is time to agree that "in-memory" !=
> > > > > stateless.
> > > > > > > > > Instead
> > > > > > > > > > > "in-memory" means "data is not stored on disk".
> > > > > > > > > > > May be we can have a flag which will wipe out all
> > metadata
> > > on
> > > > > > node
> > > > > > > > > > restart
> > > > > > > > > > > (e.g. it could make sense for embedded clients)?
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk <
> > > > > > > > > > > [hidden email]> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Denis,
> > > > > > > > > > > >
> > > > > > > > > > > > This is a subtle question. It looks like we have now
> a
> > > > number
> > > > > > of
> > > > > > > > > > > use-cases
> > > > > > > > > > > > when persistent storage is required even for a pure
> > > > in-memory
> > > > > > > mode.
> > > > > > > > > One
> > > > > > > > > > > of
> > > > > > > > > > > > the use-cases is thin client authentication, the
> other
> > is
> > > > > > service
> > > > > > > > > grid
> > > > > > > > > > > > configuration persistence.
> > > > > > > > > > > >
> > > > > > > > > > > > Generally, I would agree that this is an expected
> > > behavior.
> > > > > > > > However,
> > > > > > > > > > this
> > > > > > > > > > > > means that a user cannot simply start and stop nodes
> > > > randomly
> > > > > > > > > anymore.
> > > > > > > > > > > > Ignite start will require some sort of installation
> or
> > > work
> > > > > > > folder
> > > > > > > > > > > > initialization (sort of initdb in postgres) which is
> ok
> > > for
> > > > > > > > > > > > persistence-enabled modes, but I am not sure if this
> is
> > > > > > expected
> > > > > > > > for
> > > > > > > > > > > > in-memory. Of course, we can run this initialization
> > > > > > > automatically,
> > > > > > > > > but
> > > > > > > > > > > it
> > > > > > > > > > > > is not always a good idea.
> > > > > > > > > > > >
> > > > > > > > > > > > If we are ok to have this restrictions for in-memory
> > > mode,
> > > > > then
> > > > > > > > > service
> > > > > > > > > > > > persistence makes sense.
> > > > > > > > > > > >
> > > > > > > > > > > > --AG
> > > > > > > > > > > >
> > > > > > > > > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <
> > > [hidden email]
> > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > >> Denis,
> > > > > > > > > > > >>
> > > > > > > > > > > >> I think that the service deployment state needs be
> > > > persisted
> > > > > > > > > > > cluster-wide.
> > > > > > > > > > > >> I guess that our meta-store is capable of doing so.
> > Alex
> > > > G.,
> > > > > > > > > Vladimir,
> > > > > > > > > > > >> could you confirm?
> > > > > > > > > > > >>
> > > > > > > > > > > >> As for the split-brain scenarios, I would put them
> > aside
> > > > for
> > > > > > now
> > > > > > > > > > > because,
> > > > > > > > > > > >> anyway, they have to be solved at lower levels (meta
> > > > store,
> > > > > > > > > discovery,
> > > > > > > > > > > >> etc.).
> > > > > > > > > > > >>
> > > > > > > > > > > >> Also, I heard that presently we store a service
> > > > > configuration
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > >> system
> > > > > > > > > > > >> cache that doesn't give us a way to deploy a new
> > version
> > > > of
> > > > > a
> > > > > > > > > service
> > > > > > > > > > > >> without undeployment of the previous one. Will this
> > > issue
> > > > be
> > > > > > > > > addressed
> > > > > > > > > > > by
> > > > > > > > > > > >> the new deployment approach?
> > > > > > > > > > > >>
> > > > > > > > > > > >> --
> > > > > > > > > > > >> Denis
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Denis,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Sounds reasonable. It's not clear, though, what
> > should
> > > > > > happen,
> > > > > > > > if
> > > > > > > > > a
> > > > > > > > > > > >> joining
> > > > > > > > > > > >> > node has some services persisted, that are missing
> > on
> > > > > other
> > > > > > > > nodes.
> > > > > > > > > > > >> > Should we deploy them?
> > > > > > > > > > > >> > If we do so, it could lead to surprising
> behaviour.
> > > For
> > > > > > > example
> > > > > > > > > you
> > > > > > > > > > > >> could
> > > > > > > > > > > >> > kill a node, undeploy a service, then bring back
> an
> > > old
> > > > > > node,
> > > > > > > > and
> > > > > > > > > it
> > > > > > > > > > > >> would
> > > > > > > > > > > >> > make the service resurrect.
> > > > > > > > > > > >> > We could store some deployment counter along with
> > the
> > > > > > service
> > > > > > > > > > > >> > configurations on all nodes, that would show how
> > many
> > > > > times
> > > > > > > the
> > > > > > > > > > > service
> > > > > > > > > > > >> > state has changed, i.e. it has been
> > > > undeployed/redeployed.
> > > > > > It
> > > > > > > > > should
> > > > > > > > > > > be
> > > > > > > > > > > >> > kept for undeployed services as well to avoid
> > > situations
> > > > > > like
> > > > > > > I
> > > > > > > > > > > >> described.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > But it still leaves a possibility of incorrect
> > > > behaviour,
> > > > > if
> > > > > > > > there
> > > > > > > > > > > was a
> > > > > > > > > > > >> > split-brain situation at some point. I don't think
> > we
> > > > > should
> > > > > > > > > precess
> > > > > > > > > > > it
> > > > > > > > > > > >> > somehow, though. If we choose to tackle it, it
> will
> > > > > > > > overcomplicate
> > > > > > > > > > > >> things
> > > > > > > > > > > >> > for a sake of a minor improvement.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Denis
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > > > > > > > > >> > [hidden email]>:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > I was responding to another Denis :) Agree with
> > you
> > > on
> > > > > > your
> > > > > > > > > point
> > > > > > > > > > > >> though.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > -Val
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> > > > > > > > [hidden email]>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > Val,
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Guess we're talking about other situations.
> I'm
> > > > > bringing
> > > > > > > up
> > > > > > > > > the
> > > > > > > > > > > case
> > > > > > > > > > > >> > > when a
> > > > > > > > > > > >> > > > service was deployed dynamically and has to be
> > > > brought
> > > > > > up
> > > > > > > > > after
> > > > > > > > > > a
> > > > > > > > > > > >> full
> > > > > > > > > > > >> > > > cluster restart w/o user intervention. To
> > achieve
> > > > this
> > > > > > we
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > >> > persist
> > > > > > > > > > > >> > > > the service's configuration somewhere.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > --
> > > > > > > > > > > >> > > > Denis
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin
> > > Kulichenko
> > > > <
> > > > > > > > > > > >> > > > [hidden email]> wrote:
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > Denis,
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every
> time
> > a
> > > > > class
> > > > > > is
> > > > > > > > > > > deployed
> > > > > > > > > > > >> or
> > > > > > > > > > > >> > > > > redeployed. If this doesn't happen in some
> > > cases,
> > > > I
> > > > > > > > believe
> > > > > > > > > > this
> > > > > > > > > > > >> > would
> > > > > > > > > > > >> > > > be a
> > > > > > > > > > > >> > > > > bug. I don't think we need to add any new
> > > events.
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > -Val
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis
> Magda <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > Denis,
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > I would encourage us to persist a service
> > > > > > > configuration
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > >> meta
> > > > > > > > > > > >> > > > store
> > > > > > > > > > > >> > > > > > and have this capability enabled by
> default.
> > > > > That's
> > > > > > > > > > essential
> > > > > > > > > > > >> for
> > > > > > > > > > > >> > > > > services
> > > > > > > > > > > >> > > > > > started dynamically. Moreover, we support
> > > > similar
> > > > > > > > behavior
> > > > > > > > > > for
> > > > > > > > > > > >> > > caches,
> > > > > > > > > > > >> > > > > > indexes, and other DDL changes happened at
> > > > > runtime.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > --
> > > > > > > > > > > >> > > > > > Denis
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis
> > > > Mekhanikov <
> > > > > > > > > > > >> > > > [hidden email]>
> > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > Another question, that I would like to
> > > discuss
> > > > > is
> > > > > > > > > whether
> > > > > > > > > > > >> > services
> > > > > > > > > > > >> > > > > should
> > > > > > > > > > > >> > > > > > > be preserved on cluster restarts.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Currently it depends on persistence
> > > > > configuration.
> > > > > > > If
> > > > > > > > > > > >> persistence
> > > > > > > > > > > >> > > for
> > > > > > > > > > > >> > > > > any
> > > > > > > > > > > >> > > > > > > data region is enabled, then services
> will
> > > be
> > > > > > > > persisted
> > > > > > > > > as
> > > > > > > > > > > >> well.
> > > > > > > > > > > >> > > This
> > > > > > > > > > > >> > > > > is
> > > > > > > > > > > >> > > > > > a
> > > > > > > > > > > >> > > > > > > pretty strange way of configuring this
> > > > > behaviour.
> > > > > > > > > > > >> > > > > > > I'm not sure, if anybody relies on this
> > > > > > > functionality
> > > > > > > > > > right
> > > > > > > > > > > >> now.
> > > > > > > > > > > >> > > > Should
> > > > > > > > > > > >> > > > > > we
> > > > > > > > > > > >> > > > > > > support it at all? If yes, should we
> make
> > it
> > > > > > > > > configurable?
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Denis
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis
> > > Mekhanikov <
> > > > > > > > > > > >> > > [hidden email]
> > > > > > > > > > > >> > > > >:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > > Val,
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > Sounds reasonable. I just think, that
> > user
> > > > > > should
> > > > > > > > have
> > > > > > > > > > > some
> > > > > > > > > > > >> way
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > know,
> > > > > > > > > > > >> > > > > > > > that new version of a service class
> was
> > > > > > deployed.
> > > > > > > > > > > >> > > > > > > > One way to do it is to listen to
> > > > > > > > *EVT_CLASS_DEPLOYED.
> > > > > > > > > > *I'm
> > > > > > > > > > > >> not
> > > > > > > > > > > >> > > > sure,
> > > > > > > > > > > >> > > > > > > > whether it is triggered on class
> > > > redeployment,
> > > > > > > > though.
> > > > > > > > > > If
> > > > > > > > > > > >> not,
> > > > > > > > > > > >> > > then
> > > > > > > > > > > >> > > > > > > another
> > > > > > > > > > > >> > > > > > > > event type should be added.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > I don't think, that a lot of people
> will
> > > > > > implement
> > > > > > > > > their
> > > > > > > > > > > own
> > > > > > > > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make
> > work
> > > > with
> > > > > > > > > > > >> > *UriDeploymentSpi*
> > > > > > > > > > > >> > > > as
> > > > > > > > > > > >> > > > > > > > comfortable as possible.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > Denis
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin
> > > > > Kulichenko
> > > > > > <
> > > > > > > > > > > >> > > > > > > > [hidden email]>:
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > >> Yes, the class deployment itself has
> to
> > > be
> > > > > > > > explicit.
> > > > > > > > > > > I.e.,
> > > > > > > > > > > >> > there
> > > > > > > > > > > >> > > > has
> > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > >> > > > > > > be
> > > > > > > > > > > >> > > > > > > >> a manual step where user updates the
> > > class,
> > > > > and
> > > > > > > the
> > > > > > > > > > exact
> > > > > > > > > > > >> step
> > > > > > > > > > > >> > > > > > required
> > > > > > > > > > > >> > > > > > > >> would depend on DeploymentSpi
> > > > implementation.
> > > > > > But
> > > > > > > > > then
> > > > > > > > > > > >> Ignite
> > > > > > > > > > > >> > > > takes
> > > > > > > > > > > >> > > > > > care
> > > > > > > > > > > >> > > > > > > >> of
> > > > > > > > > > > >> > > > > > > >> everything else - service
> redeployment
> > > and
> > > > > > > restart
> > > > > > > > is
> > > > > > > > > > > >> > automatic.
> > > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to
> be
> > > > > > disabled
> > > > > > > if
> > > > > > > > > > > >> > > DeploymentSpi
> > > > > > > > > > > >> > > > is
> > > > > > > > > > > >> > > > > > not
> > > > > > > > > > > >> > > > > > > >> configured. In this case service
> class
> > > > > > > definitions
> > > > > > > > > have
> > > > > > > > > > > to
> > > > > > > > > > > >> be
> > > > > > > > > > > >> > > > > deployed
> > > > > > > > > > > >> > > > > > > on
> > > > > > > > > > > >> > > > > > > >> local classpath and can't be updated
> in
> > > > > > runtime.
> > > > > > > > Just
> > > > > > > > > > > like
> > > > > > > > > > > >> it
> > > > > > > > > > > >> > > > works
> > > > > > > > > > > >> > > > > > > right
> > > > > > > > > > > >> > > > > > > >> now.
> > > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > > >> > > > > > > >> -Val
> > > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM,
> > Dmitriy
> > > > > > > Setrakyan
> > > > > > > > <
> > > > > > > > > > > >> > > > > > > [hidden email]
> > > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > > >> > > > > > > >> wrote:
> > > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM,
> > Dmitry
> > > > > > Pavlov <
> > > > > > > > > > > >> > > > > > [hidden email]>
> > > > > > > > > > > >> > > > > > > >> > wrote:
> > > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > > >> > > > > > > >> > > Hi Igniters,
> > > > > > > > > > > >> > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > >> > > I like automatic redeploy which
> can
> > > be
> > > > > > > disabled
> > > > > > > > > by
> > > > > > > > > > > >> config
> > > > > > > > > > > >> > if
> > > > > > > > > > > >> > > > > user
> > > > > > > > > > > >> > > > > > > >> wants
> > > > > > > > > > > >> > > > > > > >> > to
> > > > > > > > > > > >> > > > > > > >> > > control this process. What do you
> > > > think?
> > > > > > > > > > > >> > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > > >> > > > > > > >> > I do not think we should have
> > anything
> > > > > > > automatic
> > > > > > > > > when
> > > > > > > > > > > it
> > > > > > > > > > > >> > comes
> > > > > > > > > > > >> > > > to
> > > > > > > > > > > >> > > > > > > >> > deployment, everything should be
> > > > explicit.
> > > > > > > > However,
> > > > > > > > > > if
> > > > > > > > > > > we
> > > > > > > > > > > >> > use
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > >> > deployment SPI, then a user should
> be
> > > > able
> > > > > to
> > > > > > > do
> > > > > > > > > > "hot"
> > > > > > > > > > > >> > > redeploy,
> > > > > > > > > > > >> > > > > > > where a
> > > > > > > > > > > >> > > > > > > >> > new service will be deployed if the
> > > user
> > > > > > drops
> > > > > > > an
> > > > > > > > > > > updated
> > > > > > > > > > > >> > jar.
> > > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > > >> > > > > > > >> > We should not create anything new
> > here.
> > > > > > Ignite
> > > > > > > > > > already
> > > > > > > > > > > >> has a
> > > > > > > > > > > >> > > > > > > deployment
> > > > > > > > > > > >> > > > > > > >> SPI
> > > > > > > > > > > >> > > > > > > >> > and it already works in a certain
> > way.
> > > > > Let's
> > > > > > > not
> > > > > > > > > > change
> > > > > > > > > > > >> it.
> > > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > > >> > > > > > > >> > D.
> > > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>