automatically deploying user libraries

[dsetrakyan]

Igniters,

So far Ignite deployment process left it up to a user to deploy all the
libraries on all the nodes manually, before a node can even be started up.

What about adding *Ignite.deploy(...)* method that could do the deployment
automatically? As parameters to this method, it could receive either a URI
of the user archives, a GIT repository, or Maven repositories and artifacts.

I have described the design here:
https://issues.apache.org/jira/browse/IGNITE-1160

Please provide comments on whether you think this functionality is useful
or sufficient.

D,

[Valentin Kulichenko]

+1 for the feature. Looks like really good usability enhancement.

-Val

On Sun, Jul 26, 2015 at 10:31 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Igniters,
>
> So far Ignite deployment process left it up to a user to deploy all the
> libraries on all the nodes manually, before a node can even be started up.
>
> What about adding *Ignite.deploy(...)* method that could do the deployment
> automatically? As parameters to this method, it could receive either a URI
> of the user archives, a GIT repository, or Maven repositories and
> artifacts.
>
> I have described the design here:
> https://issues.apache.org/jira/browse/IGNITE-1160
>
> Please provide comments on whether you think this functionality is useful
> or sufficient.
>
> D,
>

[Alexey Goncharuk]

I like the idea, however I do not like the API.

Why do we need to limit deployment process to a list of files or GIT
repositories? What if I want to build artifacts from Mercurial using
gradle? I think the entity that provides artifacts should be an interface
and Ignite should provide an out-of-the-box set of builders that can fetch
files or Maven artifacts or build GIT repos with pom files.

Besides the API, we should describe how the deployment process interacts
with other Ignite subsystems. For example, once artifacts are deployed,
they should be made available on joining nodes on early stages of
discovery, so that dynamic cache start feature can use those classes to
instantiate cache store. Should we provide a method to undeploy artifacts?
If yes, we should gracefully clean up all components that may have used
deployed code: stop caches, distributed services, tasks. Looks like
deploy() should return an instance of Ignite which has a deployment
context, and deployed classes should be made available only for method
invocations made on that particular Ignite instance.

--AG

2015-07-27 11:49 GMT-07:00 Valentin Kulichenko <
[hidden email]>:

> +1 for the feature. Looks like really good usability enhancement.
>
> -Val
>
> On Sun, Jul 26, 2015 at 10:31 PM, Dmitriy Setrakyan <[hidden email]
> >
> wrote:
>
> > Igniters,
> >
> > So far Ignite deployment process left it up to a user to deploy all the
> > libraries on all the nodes manually, before a node can even be started
> up.
> >
> > What about adding *Ignite.deploy(...)* method that could do the
> deployment
> > automatically? As parameters to this method, it could receive either a
> URI
> > of the user archives, a GIT repository, or Maven repositories and
> > artifacts.
> >
> > I have described the design here:
> > https://issues.apache.org/jira/browse/IGNITE-1160
> >
> > Please provide comments on whether you think this functionality is useful
> > or sufficient.
> >
> > D,
> >
>

[Valentin Kulichenko]

On Mon, Jul 27, 2015 at 12:32 PM, Alexey Goncharuk <
[hidden email]> wrote:

> I like the idea, however I do not like the API.
>
> Why do we need to limit deployment process to a list of files or GIT
> repositories? What if I want to build artifacts from Mercurial using
> gradle? I think the entity that provides artifacts should be an interface
> and Ignite should provide an out-of-the-box set of builders that can fetch
> files or Maven artifacts or build GIT repos with pom files.
>
>
Agree. deploy() method should take smth like DeploymentProvider interface
with getArtifacts() method. We can also add it as a configuration property
to apply automatically on startup.

I'm also not sure about -deploy option for ignite.sh. It seems to me that
deployment should be used by an application that deployed it. What can a
standalone node do with it?

> Besides the API, we should describe how the deployment process interacts
> with other Ignite subsystems. For example, once artifacts are deployed,
> they should be made available on joining nodes on early stages of
> discovery, so that dynamic cache start feature can use those classes to
> instantiate cache store. Should we provide a method to undeploy artifacts?
> If yes, we should gracefully clean up all components that may have used
> deployed code: stop caches, distributed services, tasks. Looks like
> deploy() should return an instance of Ignite which has a deployment
> context, and deployed classes should be made available only for method
> invocations made on that particular Ignite instance.
>

I think ability to redeploy (and therefore undeploy) is a must here.
Otherwise you will need to restart the cluster each time you have changes.
This makes the feature useless.

>
> --AG
>
> 2015-07-27 11:49 GMT-07:00 Valentin Kulichenko <
> [hidden email]>:
>
> > +1 for the feature. Looks like really good usability enhancement.
> >
> > -Val
> >
> > On Sun, Jul 26, 2015 at 10:31 PM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > wrote:
> >
> > > Igniters,
> > >
> > > So far Ignite deployment process left it up to a user to deploy all the
> > > libraries on all the nodes manually, before a node can even be started
> > up.
> > >
> > > What about adding *Ignite.deploy(...)* method that could do the
> > deployment
> > > automatically? As parameters to this method, it could receive either a
> > URI
> > > of the user archives, a GIT repository, or Maven repositories and
> > > artifacts.
> > >
> > > I have described the design here:
> > > https://issues.apache.org/jira/browse/IGNITE-1160
> > >
> > > Please provide comments on whether you think this functionality is
> useful
> > > or sufficient.
> > >
> > > D,
> > >
> >
>

[dsetrakyan]

Hm...

I actually don't like the complexity of creating a DeploymentProvider for
every deployment source. How about we just use URI-like approach?

git://my.git.repository/master
svn://...
file:///....
mvn:my.maven.artifact
etc...

This way we can simply have a collection of URIs as a parameter to the
deploy method. At implementation level we can have a different
DeploymentProvider for each deployment protocol we support.

D.

On Mon, Jul 27, 2015 at 5:43 PM, Valentin Kulichenko <
[hidden email]> wrote:

> On Mon, Jul 27, 2015 at 12:32 PM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > I like the idea, however I do not like the API.
> >
> > Why do we need to limit deployment process to a list of files or GIT
> > repositories? What if I want to build artifacts from Mercurial using
> > gradle? I think the entity that provides artifacts should be an interface
> > and Ignite should provide an out-of-the-box set of builders that can
> fetch
> > files or Maven artifacts or build GIT repos with pom files.
> >
> >
> Agree. deploy() method should take smth like DeploymentProvider interface
> with getArtifacts() method. We can also add it as a configuration property
> to apply automatically on startup.
>
> I'm also not sure about -deploy option for ignite.sh. It seems to me that
> deployment should be used by an application that deployed it. What can a
> standalone node do with it?
>
>
> > Besides the API, we should describe how the deployment process interacts
> > with other Ignite subsystems. For example, once artifacts are deployed,
> > they should be made available on joining nodes on early stages of
> > discovery, so that dynamic cache start feature can use those classes to
> > instantiate cache store. Should we provide a method to undeploy
> artifacts?
> > If yes, we should gracefully clean up all components that may have used
> > deployed code: stop caches, distributed services, tasks. Looks like
> > deploy() should return an instance of Ignite which has a deployment
> > context, and deployed classes should be made available only for method
> > invocations made on that particular Ignite instance.
> >
>
> I think ability to redeploy (and therefore undeploy) is a must here.
> Otherwise you will need to restart the cluster each time you have changes.
> This makes the feature useless.
>
>
> >
> > --AG
> >
> > 2015-07-27 11:49 GMT-07:00 Valentin Kulichenko <
> > [hidden email]>:
> >
> > > +1 for the feature. Looks like really good usability enhancement.
> > >
> > > -Val
> > >
> > > On Sun, Jul 26, 2015 at 10:31 PM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > So far Ignite deployment process left it up to a user to deploy all
> the
> > > > libraries on all the nodes manually, before a node can even be
> started
> > > up.
> > > >
> > > > What about adding *Ignite.deploy(...)* method that could do the
> > > deployment
> > > > automatically? As parameters to this method, it could receive either
> a
> > > URI
> > > > of the user archives, a GIT repository, or Maven repositories and
> > > > artifacts.
> > > >
> > > > I have described the design here:
> > > > https://issues.apache.org/jira/browse/IGNITE-1160
> > > >
> > > > Please provide comments on whether you think this functionality is
> > useful
> > > > or sufficient.
> > > >
> > > > D,
> > > >
> > >
> >
>

[Alexey Goncharuk]

I still do not like this approach. Maven artifacts may require another URL
for an external repository, now we would have to encode a URI into another
URI. The git repository may require some settings for a build system, which
also may not always be expressed in terms of URI parameters.

Besides, if I understood correctly, we should still leave a room for users
to implement their own provider that will parse an URI and provide
artifacts, which is the exact interface suggested by Val, but
getArtifacts() method now has an URI as an argument. But instead of passing
these providers to deploy() method, we need to set them in configuration.

2015-07-28 1:47 GMT-07:00 Dmitriy Setrakyan <[hidden email]>:

> Hm...
>
> I actually don't like the complexity of creating a DeploymentProvider for
> every deployment source. How about we just use URI-like approach?
>
> git://my.git.repository/master
> svn://...
> file:///....
> mvn:my.maven.artifact
> etc...
>
> This way we can simply have a collection of URIs as a parameter to the
> deploy method. At implementation level we can have a different
> DeploymentProvider for each deployment protocol we support.
>
> D.
>
> On Mon, Jul 27, 2015 at 5:43 PM, Valentin Kulichenko <
> [hidden email]> wrote:
>
> > On Mon, Jul 27, 2015 at 12:32 PM, Alexey Goncharuk <
> > [hidden email]> wrote:
> >
> > > I like the idea, however I do not like the API.
> > >
> > > Why do we need to limit deployment process to a list of files or GIT
> > > repositories? What if I want to build artifacts from Mercurial using
> > > gradle? I think the entity that provides artifacts should be an
> interface
> > > and Ignite should provide an out-of-the-box set of builders that can
> > fetch
> > > files or Maven artifacts or build GIT repos with pom files.
> > >
> > >
> > Agree. deploy() method should take smth like DeploymentProvider interface
> > with getArtifacts() method. We can also add it as a configuration
> property
> > to apply automatically on startup.
> >
> > I'm also not sure about -deploy option for ignite.sh. It seems to me that
> > deployment should be used by an application that deployed it. What can a
> > standalone node do with it?
> >
> >
> > > Besides the API, we should describe how the deployment process
> interacts
> > > with other Ignite subsystems. For example, once artifacts are deployed,
> > > they should be made available on joining nodes on early stages of
> > > discovery, so that dynamic cache start feature can use those classes to
> > > instantiate cache store. Should we provide a method to undeploy
> > artifacts?
> > > If yes, we should gracefully clean up all components that may have used
> > > deployed code: stop caches, distributed services, tasks. Looks like
> > > deploy() should return an instance of Ignite which has a deployment
> > > context, and deployed classes should be made available only for method
> > > invocations made on that particular Ignite instance.
> > >
> >
> > I think ability to redeploy (and therefore undeploy) is a must here.
> > Otherwise you will need to restart the cluster each time you have
> changes.
> > This makes the feature useless.
> >
> >
> > >
> > > --AG
> > >
> > > 2015-07-27 11:49 GMT-07:00 Valentin Kulichenko <
> > > [hidden email]>:
> > >
> > > > +1 for the feature. Looks like really good usability enhancement.
> > > >
> > > > -Val
> > > >
> > > > On Sun, Jul 26, 2015 at 10:31 PM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > So far Ignite deployment process left it up to a user to deploy all
> > the
> > > > > libraries on all the nodes manually, before a node can even be
> > started
> > > > up.
> > > > >
> > > > > What about adding *Ignite.deploy(...)* method that could do the
> > > > deployment
> > > > > automatically? As parameters to this method, it could receive
> either
> > a
> > > > URI
> > > > > of the user archives, a GIT repository, or Maven repositories and
> > > > > artifacts.
> > > > >
> > > > > I have described the design here:
> > > > > https://issues.apache.org/jira/browse/IGNITE-1160
> > > > >
> > > > > Please provide comments on whether you think this functionality is
> > > useful
> > > > > or sufficient.
> > > > >
> > > > > D,
> > > > >
> > > >
> > >
> >
>

[Valentin Kulichenko]

Agree. URIs are not flexible enough. But we still can have
UriDeploymentProvider for protocols that fit there (file, HTTP, FTP, etc.).
Definitely no need to create a separate provider for each of them.

-Val

On Tue, Jul 28, 2015 at 10:19 AM, Alexey Goncharuk <
[hidden email]> wrote:

> I still do not like this approach. Maven artifacts may require another URL
> for an external repository, now we would have to encode a URI into another
> URI. The git repository may require some settings for a build system, which
> also may not always be expressed in terms of URI parameters.
>
> Besides, if I understood correctly, we should still leave a room for users
> to implement their own provider that will parse an URI and provide
> artifacts, which is the exact interface suggested by Val, but
> getArtifacts() method now has an URI as an argument. But instead of passing
> these providers to deploy() method, we need to set them in configuration.
>
> 2015-07-28 1:47 GMT-07:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Hm...
> >
> > I actually don't like the complexity of creating a DeploymentProvider for
> > every deployment source. How about we just use URI-like approach?
> >
> > git://my.git.repository/master
> > svn://...
> > file:///....
> > mvn:my.maven.artifact
> > etc...
> >
> > This way we can simply have a collection of URIs as a parameter to the
> > deploy method. At implementation level we can have a different
> > DeploymentProvider for each deployment protocol we support.
> >
> > D.
> >
> > On Mon, Jul 27, 2015 at 5:43 PM, Valentin Kulichenko <
> > [hidden email]> wrote:
> >
> > > On Mon, Jul 27, 2015 at 12:32 PM, Alexey Goncharuk <
> > > [hidden email]> wrote:
> > >
> > > > I like the idea, however I do not like the API.
> > > >
> > > > Why do we need to limit deployment process to a list of files or GIT
> > > > repositories? What if I want to build artifacts from Mercurial using
> > > > gradle? I think the entity that provides artifacts should be an
> > interface
> > > > and Ignite should provide an out-of-the-box set of builders that can
> > > fetch
> > > > files or Maven artifacts or build GIT repos with pom files.
> > > >
> > > >
> > > Agree. deploy() method should take smth like DeploymentProvider
> interface
> > > with getArtifacts() method. We can also add it as a configuration
> > property
> > > to apply automatically on startup.
> > >
> > > I'm also not sure about -deploy option for ignite.sh. It seems to me
> that
> > > deployment should be used by an application that deployed it. What can
> a
> > > standalone node do with it?
> > >
> > >
> > > > Besides the API, we should describe how the deployment process
> > interacts
> > > > with other Ignite subsystems. For example, once artifacts are
> deployed,
> > > > they should be made available on joining nodes on early stages of
> > > > discovery, so that dynamic cache start feature can use those classes
> to
> > > > instantiate cache store. Should we provide a method to undeploy
> > > artifacts?
> > > > If yes, we should gracefully clean up all components that may have
> used
> > > > deployed code: stop caches, distributed services, tasks. Looks like
> > > > deploy() should return an instance of Ignite which has a deployment
> > > > context, and deployed classes should be made available only for
> method
> > > > invocations made on that particular Ignite instance.
> > > >
> > >
> > > I think ability to redeploy (and therefore undeploy) is a must here.
> > > Otherwise you will need to restart the cluster each time you have
> > changes.
> > > This makes the feature useless.
> > >
> > >
> > > >
> > > > --AG
> > > >
> > > > 2015-07-27 11:49 GMT-07:00 Valentin Kulichenko <
> > > > [hidden email]>:
> > > >
> > > > > +1 for the feature. Looks like really good usability enhancement.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Sun, Jul 26, 2015 at 10:31 PM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > So far Ignite deployment process left it up to a user to deploy
> all
> > > the
> > > > > > libraries on all the nodes manually, before a node can even be
> > > started
> > > > > up.
> > > > > >
> > > > > > What about adding *Ignite.deploy(...)* method that could do the
> > > > > deployment
> > > > > > automatically? As parameters to this method, it could receive
> > either
> > > a
> > > > > URI
> > > > > > of the user archives, a GIT repository, or Maven repositories and
> > > > > > artifacts.
> > > > > >
> > > > > > I have described the design here:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1160
> > > > > >
> > > > > > Please provide comments on whether you think this functionality
> is
> > > > useful
> > > > > > or sufficient.
> > > > > >
> > > > > > D,
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

Alexey,

I probably was not clear in my email earlier. I think that
DeploymentProvider is a good abstraction and we should have it. My comment
was that I would like it to be exposed as a URI string vs. "new
DeploymentProvider(...)".

For example, GitDeploymentProvider would be configured as
"git://some.repo?param1=1&param2=2"

As far as our point about Maven repositories, we can think of a way to
expose it artifacts and repo coordinates in a URI string. For example, we
can have "mvn://..." and "mvnrepo://" URIs.

Does this make sense?

D.


On Tue, Jul 28, 2015 at 10:19 AM, Alexey Goncharuk <
[hidden email]> wrote:

> I still do not like this approach. Maven artifacts may require another URL
> for an external repository, now we would have to encode a URI into another
> URI. The git repository may require some settings for a build system, which
> also may not always be expressed in terms of URI parameters.
>
> Besides, if I understood correctly, we should still leave a room for users
> to implement their own provider that will parse an URI and provide
> artifacts, which is the exact interface suggested by Val, but
> getArtifacts() method now has an URI as an argument. But instead of passing
> these providers to deploy() method, we need to set them in configuration.
>
> 2015-07-28 1:47 GMT-07:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Hm...
> >
> > I actually don't like the complexity of creating a DeploymentProvider for
> > every deployment source. How about we just use URI-like approach?
> >
> > git://my.git.repository/master
> > svn://...
> > file:///....
> > mvn:my.maven.artifact
> > etc...
> >
> > This way we can simply have a collection of URIs as a parameter to the
> > deploy method. At implementation level we can have a different
> > DeploymentProvider for each deployment protocol we support.
> >
> > D.
> >
> > On Mon, Jul 27, 2015 at 5:43 PM, Valentin Kulichenko <
> > [hidden email]> wrote:
> >
> > > On Mon, Jul 27, 2015 at 12:32 PM, Alexey Goncharuk <
> > > [hidden email]> wrote:
> > >
> > > > I like the idea, however I do not like the API.
> > > >
> > > > Why do we need to limit deployment process to a list of files or GIT
> > > > repositories? What if I want to build artifacts from Mercurial using
> > > > gradle? I think the entity that provides artifacts should be an
> > interface
> > > > and Ignite should provide an out-of-the-box set of builders that can
> > > fetch
> > > > files or Maven artifacts or build GIT repos with pom files.
> > > >
> > > >
> > > Agree. deploy() method should take smth like DeploymentProvider
> interface
> > > with getArtifacts() method. We can also add it as a configuration
> > property
> > > to apply automatically on startup.
> > >
> > > I'm also not sure about -deploy option for ignite.sh. It seems to me
> that
> > > deployment should be used by an application that deployed it. What can
> a
> > > standalone node do with it?
> > >
> > >
> > > > Besides the API, we should describe how the deployment process
> > interacts
> > > > with other Ignite subsystems. For example, once artifacts are
> deployed,
> > > > they should be made available on joining nodes on early stages of
> > > > discovery, so that dynamic cache start feature can use those classes
> to
> > > > instantiate cache store. Should we provide a method to undeploy
> > > artifacts?
> > > > If yes, we should gracefully clean up all components that may have
> used
> > > > deployed code: stop caches, distributed services, tasks. Looks like
> > > > deploy() should return an instance of Ignite which has a deployment
> > > > context, and deployed classes should be made available only for
> method
> > > > invocations made on that particular Ignite instance.
> > > >
> > >
> > > I think ability to redeploy (and therefore undeploy) is a must here.
> > > Otherwise you will need to restart the cluster each time you have
> > changes.
> > > This makes the feature useless.
> > >
> > >
> > > >
> > > > --AG
> > > >
> > > > 2015-07-27 11:49 GMT-07:00 Valentin Kulichenko <
> > > > [hidden email]>:
> > > >
> > > > > +1 for the feature. Looks like really good usability enhancement.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Sun, Jul 26, 2015 at 10:31 PM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > So far Ignite deployment process left it up to a user to deploy
> all
> > > the
> > > > > > libraries on all the nodes manually, before a node can even be
> > > started
> > > > > up.
> > > > > >
> > > > > > What about adding *Ignite.deploy(...)* method that could do the
> > > > > deployment
> > > > > > automatically? As parameters to this method, it could receive
> > either
> > > a
> > > > > URI
> > > > > > of the user archives, a GIT repository, or Maven repositories and
> > > > > > artifacts.
> > > > > >
> > > > > > I have described the design here:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1160
> > > > > >
> > > > > > Please provide comments on whether you think this functionality
> is
> > > > useful
> > > > > > or sufficient.
> > > > > >
> > > > > > D,
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Konstantin Boudnik-2]

On Tue, Jul 28, 2015 at 10:29AM, Dmitriy Setrakyan wrote:
> Alexey,
>
> I probably was not clear in my email earlier. I think that
> DeploymentProvider is a good abstraction and we should have it. My comment
> was that I would like it to be exposed as a URI string vs. "new
> DeploymentProvider(...)".
>
> For example, GitDeploymentProvider would be configured as
> "git://some.repo?param1=1&param2=2"

I am missing a point of doing the git:, svn:, or any other vcs: URI, really.
Checking binaries into a VCS system is very problematic and unclean thing to
do, actually. Why would you encourage that with the proposed API?

> As far as our point about Maven repositories, we can think of a way to
> expose it artifacts and repo coordinates in a URI string. For example, we
> can have "mvn://..." and "mvnrepo://" URIs.
>
> Does this make sense?

I'd leave this job to the systems that already doing this and perhaps do it
quite well. Why would you spend time developing your own mvn provider? Esp in
the world that has gradle APIs? I have expressed this pow in the JIRA, so I
will step out now ;)

Cos

> On Tue, Jul 28, 2015 at 10:19 AM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > I still do not like this approach. Maven artifacts may require another URL
> > for an external repository, now we would have to encode a URI into another
> > URI. The git repository may require some settings for a build system, which
> > also may not always be expressed in terms of URI parameters.
> >
> > Besides, if I understood correctly, we should still leave a room for users
> > to implement their own provider that will parse an URI and provide
> > artifacts, which is the exact interface suggested by Val, but
> > getArtifacts() method now has an URI as an argument. But instead of passing
> > these providers to deploy() method, we need to set them in configuration.
> >
> > 2015-07-28 1:47 GMT-07:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > Hm...
> > >
> > > I actually don't like the complexity of creating a DeploymentProvider for
> > > every deployment source. How about we just use URI-like approach?
> > >
> > > git://my.git.repository/master
> > > svn://...
> > > file:///....
> > > mvn:my.maven.artifact
> > > etc...
> > >
> > > This way we can simply have a collection of URIs as a parameter to the
> > > deploy method. At implementation level we can have a different
> > > DeploymentProvider for each deployment protocol we support.
> > >
> > > D.
> > >
> > > On Mon, Jul 27, 2015 at 5:43 PM, Valentin Kulichenko <
> > > [hidden email]> wrote:
> > >
> > > > On Mon, Jul 27, 2015 at 12:32 PM, Alexey Goncharuk <
> > > > [hidden email]> wrote:
> > > >
> > > > > I like the idea, however I do not like the API.
> > > > >
> > > > > Why do we need to limit deployment process to a list of files or GIT
> > > > > repositories? What if I want to build artifacts from Mercurial using
> > > > > gradle? I think the entity that provides artifacts should be an
> > > interface
> > > > > and Ignite should provide an out-of-the-box set of builders that can
> > > > fetch
> > > > > files or Maven artifacts or build GIT repos with pom files.
> > > > >
> > > > >
> > > > Agree. deploy() method should take smth like DeploymentProvider
> > interface
> > > > with getArtifacts() method. We can also add it as a configuration
> > > property
> > > > to apply automatically on startup.
> > > >
> > > > I'm also not sure about -deploy option for ignite.sh. It seems to me
> > that
> > > > deployment should be used by an application that deployed it. What can
> > a
> > > > standalone node do with it?
> > > >
> > > >
> > > > > Besides the API, we should describe how the deployment process
> > > interacts
> > > > > with other Ignite subsystems. For example, once artifacts are
> > deployed,
> > > > > they should be made available on joining nodes on early stages of
> > > > > discovery, so that dynamic cache start feature can use those classes
> > to
> > > > > instantiate cache store. Should we provide a method to undeploy
> > > > artifacts?
> > > > > If yes, we should gracefully clean up all components that may have
> > used
> > > > > deployed code: stop caches, distributed services, tasks. Looks like
> > > > > deploy() should return an instance of Ignite which has a deployment
> > > > > context, and deployed classes should be made available only for
> > method
> > > > > invocations made on that particular Ignite instance.
> > > > >
> > > >
> > > > I think ability to redeploy (and therefore undeploy) is a must here.
> > > > Otherwise you will need to restart the cluster each time you have
> > > changes.
> > > > This makes the feature useless.
> > > >
> > > >
> > > > >
> > > > > --AG
> > > > >
> > > > > 2015-07-27 11:49 GMT-07:00 Valentin Kulichenko <
> > > > > [hidden email]>:
> > > > >
> > > > > > +1 for the feature. Looks like really good usability enhancement.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Sun, Jul 26, 2015 at 10:31 PM, Dmitriy Setrakyan <
> > > > > [hidden email]
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > So far Ignite deployment process left it up to a user to deploy
> > all
> > > > the
> > > > > > > libraries on all the nodes manually, before a node can even be
> > > > started
> > > > > > up.
> > > > > > >
> > > > > > > What about adding *Ignite.deploy(...)* method that could do the
> > > > > > deployment
> > > > > > > automatically? As parameters to this method, it could receive
> > > either
> > > > a
> > > > > > URI
> > > > > > > of the user archives, a GIT repository, or Maven repositories and
> > > > > > > artifacts.
> > > > > > >
> > > > > > > I have described the design here:
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-1160
> > > > > > >
> > > > > > > Please provide comments on whether you think this functionality
> > is
> > > > > useful
> > > > > > > or sufficient.
> > > > > > >
> > > > > > > D,
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

[dsetrakyan]

On Tue, Jul 28, 2015 at 11:20 AM, Konstantin Boudnik <[hidden email]> wrote:

> On Tue, Jul 28, 2015 at 10:29AM, Dmitriy Setrakyan wrote:
> > Alexey,
> >
> > I probably was not clear in my email earlier. I think that
> > DeploymentProvider is a good abstraction and we should have it. My
> comment
> > was that I would like it to be exposed as a URI string vs. "new
> > DeploymentProvider(...)".
> >
> > For example, GitDeploymentProvider would be configured as
> > "git://some.repo?param1=1&param2=2"
>
> I am missing a point of doing the git:, svn:, or any other vcs: URI,
> really.
> Checking binaries into a VCS system is very problematic and unclean thing
> to do, actually. Why would you encourage that with the proposed API?
>

Cos, we are not talking about checking binaries. We are planning to support
 GIT/SVN/etc repositories with a POM file. This way we simply build it
using maven ourselves and deploy it.

> > As far as our point about Maven repositories, we can think of a way to
> > expose it artifacts and repo coordinates in a URI string. For example, we
> > can have "mvn://..." and "mvnrepo://" URIs.
> >
> > Does this make sense?
>
> I'd leave this job to the systems that already doing this and perhaps do it
> quite well. Why would you spend time developing your own mvn provider? Esp
> in the world that has gradle APIs? I have expressed this pow in the JIRA,
> so I will step out now ;)
>

The implementation of MvnDeploymentProvider may very well delegate to some
very well known library.

DeploymentProvider should be a generic Ignite abstraction that has
different implementations for different deployment sources, mainly to
decouple Ignite from specific 3rd party library implementations.

[Konstantin Boudnik-2]

On Tue, Jul 28, 2015 at 11:31AM, Dmitriy Setrakyan wrote:

> On Tue, Jul 28, 2015 at 11:20 AM, Konstantin Boudnik <[hidden email]> wrote:
>
> > On Tue, Jul 28, 2015 at 10:29AM, Dmitriy Setrakyan wrote:
> > > Alexey,
> > >
> > > I probably was not clear in my email earlier. I think that
> > > DeploymentProvider is a good abstraction and we should have it. My
> > comment
> > > was that I would like it to be exposed as a URI string vs. "new
> > > DeploymentProvider(...)".
> > >
> > > For example, GitDeploymentProvider would be configured as
> > > "git://some.repo?param1=1&param2=2"
> >
> > I am missing a point of doing the git:, svn:, or any other vcs: URI,
> > really.
> > Checking binaries into a VCS system is very problematic and unclean thing
> > to do, actually. Why would you encourage that with the proposed API?
> >
>
> Cos, we are not talking about checking binaries. We are planning to support
>  GIT/SVN/etc repositories with a POM file. This way we simply build it
> using maven ourselves and deploy it.

Well, even worst IMO. Why would you want to run an external build process
as a part of the nodes deployment? This will pose a security risk in the
production deployment and you will have to find a way to disable this
in some case and allow it in the others, no?

> > > As far as our point about Maven repositories, we can think of a way to
> > > expose it artifacts and repo coordinates in a URI string. For example, we
> > > can have "mvn://..." and "mvnrepo://" URIs.
> > >
> > > Does this make sense?
> >
> > I'd leave this job to the systems that already doing this and perhaps do it
> > quite well. Why would you spend time developing your own mvn provider? Esp
> > in the world that has gradle APIs? I have expressed this pow in the JIRA,
> > so I will step out now ;)
> >
>
> The implementation of MvnDeploymentProvider may very well delegate to some
> very well known library.
>
> DeploymentProvider should be a generic Ignite abstraction that has
> different implementations for different deployment sources, mainly to
> decouple Ignite from specific 3rd party library implementations.

Good point, agree. Still don't like the the git://, svn:// stuff though ;)

  Cos

[dsetrakyan]

> Cos, we are not talking about checking binaries. We are planning to
support

> >  GIT/SVN/etc repositories with a POM file. This way we simply build it
> > using maven ourselves and deploy it.
>
> Well, even worst IMO. Why would you want to run an external build
> process as a part of the nodes deployment? This will pose a security risk
> in the production deployment and you will have to find a way to disable this
> in some case and allow it in the others, no?
>

Our Docker container right now does exactly that. The workflow is as
following:

   1. user changes code in a GIT repo
   2. the GIT repo is provided as a parameter at docker container startup
   3. docker builds the code and deploys it into Ignite
   4. docker starts an Ignite server

This is very convenient for a user, especially during development. What I
wanted to do with "deploy(...)" method, is add the same level of
convenience directly from client code.

Why do you think it will be a security risk?

[Konstantin Boudnik-2]

On Tue, Jul 28, 2015 at 12:36PM, Dmitriy Setrakyan wrote:

> > Cos, we are not talking about checking binaries. We are planning to
> support
>
> > >  GIT/SVN/etc repositories with a POM file. This way we simply build it
> > > using maven ourselves and deploy it.
> >
> > Well, even worst IMO. Why would you want to run an external build
> > process as a part of the nodes deployment? This will pose a security risk
> > in the production deployment and you will have to find a way to disable this
> > in some case and allow it in the others, no?
> >
>
> Our Docker container right now does exactly that. The workflow is as
> following:

I am an old dog and I don't think it is a real good idea to use Docker in any
sensible production. So, I don't buy that argument ;)

>
>    1. user changes code in a GIT repo
>    2. the GIT repo is provided as a parameter at docker container startup
>    3. docker builds the code and deploys it into Ignite
>    4. docker starts an Ignite server
>
> This is very convenient for a user, especially during development. What I
> wanted to do with "deploy(...)" method, is add the same level of
> convenience directly from client code.
>
> Why do you think it will be a security risk?

Yes, I think so. I won't allow something like that to be happening in the
production data center: there are too many ways where it can backfire.

Cos

[Branko Čibej]

On 28.07.2015 21:36, Dmitriy Setrakyan wrote:

>> Cos, we are not talking about checking binaries. We are planning to
> support
>
>>>  GIT/SVN/etc repositories with a POM file. This way we simply build it
>>> using maven ourselves and deploy it.
>> Well, even worst IMO. Why would you want to run an external build
>> process as a part of the nodes deployment? This will pose a security risk
>> in the production deployment and you will have to find a way to disable this
>> in some case and allow it in the others, no?
>>
> Our Docker container right now does exactly that. The workflow is as
> following:
>
>    1. user changes code in a GIT repo
>    2. the GIT repo is provided as a parameter at docker container startup
>    3. docker builds the code and deploys it into Ignite
>    4. docker starts an Ignite server
>
> This is very convenient for a user, especially during development. What I
> wanted to do with "deploy(...)" method, is add the same level of
> convenience directly from client code.

It's very convenient for users not to set a password on their computer
or mobile device, too.

> Why do you think it will be a security risk?

It's called a remote code execution exploit. Anyone who has write access
to the repo (i.e., anyone who can hack in) can change the deployed code
and DOS your whole cluster.

-- Brane

[Konstantin Boudnik-2]

On Tue, Jul 28, 2015 at 10:16PM, Branko Čibej wrote:

> On 28.07.2015 21:36, Dmitriy Setrakyan wrote:
> >> Cos, we are not talking about checking binaries. We are planning to
> > support
> >
> >>>  GIT/SVN/etc repositories with a POM file. This way we simply build it
> >>> using maven ourselves and deploy it.
> >> Well, even worst IMO. Why would you want to run an external build
> >> process as a part of the nodes deployment? This will pose a security risk
> >> in the production deployment and you will have to find a way to disable this
> >> in some case and allow it in the others, no?
> >>
> > Our Docker container right now does exactly that. The workflow is as
> > following:
> >
> >    1. user changes code in a GIT repo
> >    2. the GIT repo is provided as a parameter at docker container startup
> >    3. docker builds the code and deploys it into Ignite
> >    4. docker starts an Ignite server
> >
> > This is very convenient for a user, especially during development. What I
> > wanted to do with "deploy(...)" method, is add the same level of
> > convenience directly from client code.
>
> It's very convenient for users not to set a password on their computer
> or mobile device, too.
>
> > Why do you think it will be a security risk?
>
> It's called a remote code execution exploit. Anyone who has write access
> to the repo (i.e., anyone who can hack in) can change the deployed code
> and DOS your whole cluster.

The situation gets even worst when you allow the deployment from a build;
worst - running the build inside of node's deployment logic.

Cos

[Ognen Duzlevski]

>
> It's called a remote code execution exploit. Anyone who has write access
> to the repo (i.e., anyone who can hack in) can change the deployed code
> and DOS your whole cluster.
>

I believe these decisions are best left to the end user, the mechanism
proposed may or may not have valid uses for you but it may have valid uses
to others. I am an old dog too and I don't practice or agree with some of
the stuff "software engineers" pass along as good practice these days but
that doesn't mean other people should be deprived of a possibility. At the
end of the day, so long as the source code to something is online or on a
machine connected to the Internet, there is risk. Ever since that one paper
got us thinking (
https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf) ...

[Valentin Kulichenko]

Guys,

I'm not sure I understand the purpose of this discussion. Maven-based and
Git-based providers are just examples of pluggable implementations that we
can provide out of the box - user is never forced to use them.

I see two main use cases for the feature:

   - Dynamic deployment and redeployment of user's code during development.
   From my experience it can noticeably speed up the development, so I really
   like it. And Git+Maven approach fits here very well, BTW.
   - Interactive apps where users are allowed to deploy their custom stuff.
   Security concerns are valid here, of course. But it's up to developer to
   choose proper mechanisms of deployment and security.

I think we should discuss the feature itself, its API and how to provide
good abstraction that will allow the user to customize it in the way he
needs to achieve his goals and ensure required level security.

-Val

On Tue, Jul 28, 2015 at 3:29 PM, Ognen Duzlevski <[hidden email]>
wrote:

> >
> > It's called a remote code execution exploit. Anyone who has write access
> > to the repo (i.e., anyone who can hack in) can change the deployed code
> > and DOS your whole cluster.
> >
>
> I believe these decisions are best left to the end user, the mechanism
> proposed may or may not have valid uses for you but it may have valid uses
> to others. I am an old dog too and I don't practice or agree with some of
> the stuff "software engineers" pass along as good practice these days but
> that doesn't mean other people should be deprived of a possibility. At the
> end of the day, so long as the source code to something is online or on a
> machine connected to the Internet, there is risk. Ever since that one paper
> got us thinking (
> https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf) ...
>