Ignite and non-Java languages

[dsetrakyan]

In continuation of previous discussions on multi-language support in
Ignite, I have been thinking on how to make Ignite available to other
languages with minimal effort and think there is a very elegant way to do
it.

Even though Ignite tasks are written in Java, they can be used only as a
distribution engine for tasks that are written in Python, Ruby, or any
other language. For example, Ignite task can receive job coordinates for
Python task (task name, class name, etc.) and spawn of jobs. When jobs
arrive to remote nodes, they can start another process which in turn will
execute a Python, Ruby, or any other program.

The way we can make Ignite data available to Python or Ruby jobs is via
Memcached API that we already support and that is available on every
language out there.

Here is a Jira ticket for this (I will cross link it with this discussion):
https://issues.apache.org/jira/browse/IGNITE-859

Thoughts?

D.

[Vladimir Ozerov]

Dima,

The most complex part of task execution is master-node side. We assume that
user starts job execution from some Python/Ruby/etc. application. To
implement the same API as we have in Java, we must establish a link between
Java master node and external platform so that not only other platform
invokes Java, but Java is able to invoke the platform as well (e.g. to pass
job results). This is where we have to start JVM in the same process (or
use some other mechanisms like shmem).

On the other hand, remote job execution and cache operations are trivial to
implement given that "JVM-in-the-same-process" is done - these are single
"Java -> platform" or "platform -> Java" calls
Memcached and external process start could make our live easier for a while
(at the cost of more complex and less consistent API), but we will have to
implement the most difficult part still.

Vladimir.


On Tue, May 5, 2015 at 4:14 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> In continuation of previous discussions on multi-language support in
> Ignite, I have been thinking on how to make Ignite available to other
> languages with minimal effort and think there is a very elegant way to do
> it.
>
> Even though Ignite tasks are written in Java, they can be used only as a
> distribution engine for tasks that are written in Python, Ruby, or any
> other language. For example, Ignite task can receive job coordinates for
> Python task (task name, class name, etc.) and spawn of jobs. When jobs
> arrive to remote nodes, they can start another process which in turn will
> execute a Python, Ruby, or any other program.
>
> The way we can make Ignite data available to Python or Ruby jobs is via
> Memcached API that we already support and that is available on every
> language out there.
>
> Here is a Jira ticket for this (I will cross link it with this discussion):
> https://issues.apache.org/jira/browse/IGNITE-859
>
> Thoughts?
>
> D.
>

[dsetrakyan]

Vladimir,

I think you misunderstood me. I was not suggesting that users access our
API from Python. I was suggesting that users use our Java API to execute
Python tasks. For Python, we should have a PythonTask which will have
enough information to execute an appropriate Python program out of process.

D.

On Tue, May 5, 2015 at 1:23 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Dima,
>
> The most complex part of task execution is master-node side. We assume that
> user starts job execution from some Python/Ruby/etc. application. To
> implement the same API as we have in Java, we must establish a link between
> Java master node and external platform so that not only other platform
> invokes Java, but Java is able to invoke the platform as well (e.g. to pass
> job results). This is where we have to start JVM in the same process (or
> use some other mechanisms like shmem).
>
> On the other hand, remote job execution and cache operations are trivial to
> implement given that "JVM-in-the-same-process" is done - these are single
> "Java -> platform" or "platform -> Java" calls
> Memcached and external process start could make our live easier for a while
> (at the cost of more complex and less consistent API), but we will have to
> implement the most difficult part still.
>
> Vladimir.
>
>
> On Tue, May 5, 2015 at 4:14 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > In continuation of previous discussions on multi-language support in
> > Ignite, I have been thinking on how to make Ignite available to other
> > languages with minimal effort and think there is a very elegant way to do
> > it.
> >
> > Even though Ignite tasks are written in Java, they can be used only as a
> > distribution engine for tasks that are written in Python, Ruby, or any
> > other language. For example, Ignite task can receive job coordinates for
> > Python task (task name, class name, etc.) and spawn of jobs. When jobs
> > arrive to remote nodes, they can start another process which in turn will
> > execute a Python, Ruby, or any other program.
> >
> > The way we can make Ignite data available to Python or Ruby jobs is via
> > Memcached API that we already support and that is available on every
> > language out there.
> >
> > Here is a Jira ticket for this (I will cross link it with this
> discussion):
> > https://issues.apache.org/jira/browse/IGNITE-859
> >
> > Thoughts?
> >
> > D.
> >
>

[Vladimir Ozerov]

Got it.
+1, nice idea.

On Tue, May 5, 2015 at 11:33 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Vladimir,
>
> I think you misunderstood me. I was not suggesting that users access our
> API from Python. I was suggesting that users use our Java API to execute
> Python tasks. For Python, we should have a PythonTask which will have
> enough information to execute an appropriate Python program out of process.
>
> D.
>
> On Tue, May 5, 2015 at 1:23 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Dima,
> >
> > The most complex part of task execution is master-node side. We assume
> that
> > user starts job execution from some Python/Ruby/etc. application. To
> > implement the same API as we have in Java, we must establish a link
> between
> > Java master node and external platform so that not only other platform
> > invokes Java, but Java is able to invoke the platform as well (e.g. to
> pass
> > job results). This is where we have to start JVM in the same process (or
> > use some other mechanisms like shmem).
> >
> > On the other hand, remote job execution and cache operations are trivial
> to
> > implement given that "JVM-in-the-same-process" is done - these are single
> > "Java -> platform" or "platform -> Java" calls
> > Memcached and external process start could make our live easier for a
> while
> > (at the cost of more complex and less consistent API), but we will have
> to
> > implement the most difficult part still.
> >
> > Vladimir.
> >
> >
> > On Tue, May 5, 2015 at 4:14 AM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> > > In continuation of previous discussions on multi-language support in
> > > Ignite, I have been thinking on how to make Ignite available to other
> > > languages with minimal effort and think there is a very elegant way to
> do
> > > it.
> > >
> > > Even though Ignite tasks are written in Java, they can be used only as
> a
> > > distribution engine for tasks that are written in Python, Ruby, or any
> > > other language. For example, Ignite task can receive job coordinates
> for
> > > Python task (task name, class name, etc.) and spawn of jobs. When jobs
> > > arrive to remote nodes, they can start another process which in turn
> will
> > > execute a Python, Ruby, or any other program.
> > >
> > > The way we can make Ignite data available to Python or Ruby jobs is via
> > > Memcached API that we already support and that is available on every
> > > language out there.
> > >
> > > Here is a Jira ticket for this (I will cross link it with this
> > discussion):
> > > https://issues.apache.org/jira/browse/IGNITE-859
> > >
> > > Thoughts?
> > >
> > > D.
> > >
> >
>

[Konstantin Boudnik-2]

In other words, it is quite different from what we were discussing earlier,
right? IDLs (or else) would give Ignite a way to communicate to-n-from 3rd
party applications written in non-JVM languages.

What's proposed here is a way to reuse existing code written in non-JVM
languages from Ignite. it's a different and smaller scope if I reading it
right?

Cos

On Tue, May 05, 2015 at 01:33AM, Dmitriy Setrakyan wrote:

> Vladimir,
>
> I think you misunderstood me. I was not suggesting that users access our
> API from Python. I was suggesting that users use our Java API to execute
> Python tasks. For Python, we should have a PythonTask which will have
> enough information to execute an appropriate Python program out of process.
>
> D.
>
> On Tue, May 5, 2015 at 1:23 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Dima,
> >
> > The most complex part of task execution is master-node side. We assume that
> > user starts job execution from some Python/Ruby/etc. application. To
> > implement the same API as we have in Java, we must establish a link between
> > Java master node and external platform so that not only other platform
> > invokes Java, but Java is able to invoke the platform as well (e.g. to pass
> > job results). This is where we have to start JVM in the same process (or
> > use some other mechanisms like shmem).
> >
> > On the other hand, remote job execution and cache operations are trivial to
> > implement given that "JVM-in-the-same-process" is done - these are single
> > "Java -> platform" or "platform -> Java" calls
> > Memcached and external process start could make our live easier for a while
> > (at the cost of more complex and less consistent API), but we will have to
> > implement the most difficult part still.
> >
> > Vladimir.
> >
> >
> > On Tue, May 5, 2015 at 4:14 AM, Dmitriy Setrakyan <[hidden email]>
> > wrote:
> >
> > > In continuation of previous discussions on multi-language support in
> > > Ignite, I have been thinking on how to make Ignite available to other
> > > languages with minimal effort and think there is a very elegant way to do
> > > it.
> > >
> > > Even though Ignite tasks are written in Java, they can be used only as a
> > > distribution engine for tasks that are written in Python, Ruby, or any
> > > other language. For example, Ignite task can receive job coordinates for
> > > Python task (task name, class name, etc.) and spawn of jobs. When jobs
> > > arrive to remote nodes, they can start another process which in turn will
> > > execute a Python, Ruby, or any other program.
> > >
> > > The way we can make Ignite data available to Python or Ruby jobs is via
> > > Memcached API that we already support and that is available on every
> > > language out there.
> > >
> > > Here is a Jira ticket for this (I will cross link it with this
> > discussion):
> > > https://issues.apache.org/jira/browse/IGNITE-859
> > >
> > > Thoughts?
> > >
> > > D.
> > >
> >

[dsetrakyan]

On Tue, May 5, 2015 at 12:48 PM, Konstantin Boudnik <[hidden email]> wrote:

> In other words, it is quite different from what we were discussing earlier,
> right? IDLs (or else) would give Ignite a way to communicate to-n-from 3rd
> party applications written in non-JVM languages.
>
> What's proposed here is a way to reuse existing code written in non-JVM
> languages from Ignite. it's a different and smaller scope if I reading it
> right?
>

Correct.

>
> Cos
>
> On Tue, May 05, 2015 at 01:33AM, Dmitriy Setrakyan wrote:
> > Vladimir,
> >
> > I think you misunderstood me. I was not suggesting that users access our
> > API from Python. I was suggesting that users use our Java API to execute
> > Python tasks. For Python, we should have a PythonTask which will have
> > enough information to execute an appropriate Python program out of
> process.
> >
> > D.
> >
> > On Tue, May 5, 2015 at 1:23 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Dima,
> > >
> > > The most complex part of task execution is master-node side. We assume
> that
> > > user starts job execution from some Python/Ruby/etc. application. To
> > > implement the same API as we have in Java, we must establish a link
> between
> > > Java master node and external platform so that not only other platform
> > > invokes Java, but Java is able to invoke the platform as well (e.g. to
> pass
> > > job results). This is where we have to start JVM in the same process
> (or
> > > use some other mechanisms like shmem).
> > >
> > > On the other hand, remote job execution and cache operations are
> trivial to
> > > implement given that "JVM-in-the-same-process" is done - these are
> single
> > > "Java -> platform" or "platform -> Java" calls
> > > Memcached and external process start could make our live easier for a
> while
> > > (at the cost of more complex and less consistent API), but we will
> have to
> > > implement the most difficult part still.
> > >
> > > Vladimir.
> > >
> > >
> > > On Tue, May 5, 2015 at 4:14 AM, Dmitriy Setrakyan <
> [hidden email]>
> > > wrote:
> > >
> > > > In continuation of previous discussions on multi-language support in
> > > > Ignite, I have been thinking on how to make Ignite available to other
> > > > languages with minimal effort and think there is a very elegant way
> to do
> > > > it.
> > > >
> > > > Even though Ignite tasks are written in Java, they can be used only
> as a
> > > > distribution engine for tasks that are written in Python, Ruby, or
> any
> > > > other language. For example, Ignite task can receive job coordinates
> for
> > > > Python task (task name, class name, etc.) and spawn of jobs. When
> jobs
> > > > arrive to remote nodes, they can start another process which in turn
> will
> > > > execute a Python, Ruby, or any other program.
> > > >
> > > > The way we can make Ignite data available to Python or Ruby jobs is
> via
> > > > Memcached API that we already support and that is available on every
> > > > language out there.
> > > >
> > > > Here is a Jira ticket for this (I will cross link it with this
> > > discussion):
> > > > https://issues.apache.org/jira/browse/IGNITE-859
> > > >
> > > > Thoughts?
> > > >
> > > > D.
> > > >
> > >
>