Read this if you want to integrate Ignite with other platforms (Python, R, etc.)

[Vladimir Ozerov]

Igniters,

I see growing community interest to implementation of so-called "clients"
to other platforms, such as R, Python, etc.. Let me briefly describe
architecture of our existing integrations - C++ and .NET.

*1) Not a client!*
We do not have clients in classical sense. Our C++ and .NET integration are
fully-fledged platforms. C++/.NET node can act both as a client and as a
server.

*2) JVM inside*
We didn't implement existing integrations form scratch. Instead we wrapped
Java into relevant platform's API. So when your .NET/C++ node is started,
it starts JVM in the same process and delegate most of the work to it.
We considering creation of another mode, something similar to thin client,
when platform will not have JVM inside, but instead will connect to
JVM-based node 1]. However, no design work have been started on it so far.

*3) Forward and reverse calls*
Ignite is complex product. We have methods with "forward" semantics, when
platform invoke JVM, e.g. Cache.get(). But we also have plenty of "reverse"
calls, when JVM need to perform JNI call to the platform, e.g. continuous
query notification.

*4) Binary everywhere*
We have full interoperability between platforms. Object created in .NET can
be accessed from Java and C++ and vice versa. To achieve this we
implemented special "binary protocol" on all platforms, so that every
platform serialize objects in the same format. If you are to implement
fully-fledged platform integration, you will have to implement binary
marshaller on your own.

*5) API layering*
We have two C++ API layers - "common" [2] and everything else [3]. Common
API is shared between C++ and .NET integrations. It is low-level interface
to communicate with JVM (i.e. to perform forward and reverse JNI calls). It
operates on pointers and JNI handles, and expects that you already
serialized your objects in binary format amd put it to some memory chunk
with certain layout. "Everything else" is high-level C++ API exposed to
users.

*6) Do not expect SWIG to help you a lot!*
Taking all previous paragraphs in count - it is impossible to create
valuable working implementation with SWIG. You will have to do a lot more
than simple codegen.

*7) Any other ways to integrate?*
Yes, you can rely on our REST API [4]. We were planning to integrate with
Node.JS this way - you just send HTTP requests to Ignite node(s). Every
request is mapped to certain operation. However, this approach has two
serious limitations:
- It will be hard to employ advantages of binary interoperability
- It will be very hard to implement server -> client callbacks which are of
great importance for Ignite.
As such you will have problems implementing compute API, continuous queries
and so forth.

Please let me know if you have any further questions.

Vladimir.

[1] https://issues.apache.org/jira/browse/IGNITE-3568
[2]
https://github.com/apache/ignite/tree/master/modules/platforms/cpp/common
[3] https://github.com/apache/ignite/tree/master/modules/platforms/cpp
[4] https://apacheignite.readme.io/docs/rest-api

[Shashank Gandham]

So if we have to make full-fledged python platform for ignite. We can use C++ common  API to integrate and have an high level API of Python. Object serialisation can be achieved in python using libraries like marshal, pickle, shelve or something similar. Theoretically speaking this should be possible, or am I missing something?

> On 25-Jan-2017, at 1:39 PM, Vladimir Ozerov <[hidden email]> wrote:
>
> Igniters,
>
> I see growing community interest to implementation of so-called "clients"
> to other platforms, such as R, Python, etc.. Let me briefly describe
> architecture of our existing integrations - C++ and .NET.
>
> *1) Not a client!*
> We do not have clients in classical sense. Our C++ and .NET integration are
> fully-fledged platforms. C++/.NET node can act both as a client and as a
> server.
>
> *2) JVM inside*
> We didn't implement existing integrations form scratch. Instead we wrapped
> Java into relevant platform's API. So when your .NET/C++ node is started,
> it starts JVM in the same process and delegate most of the work to it.
> We considering creation of another mode, something similar to thin client,
> when platform will not have JVM inside, but instead will connect to
> JVM-based node 1]. However, no design work have been started on it so far.
>
> *3) Forward and reverse calls*
> Ignite is complex product. We have methods with "forward" semantics, when
> platform invoke JVM, e.g. Cache.get(). But we also have plenty of "reverse"
> calls, when JVM need to perform JNI call to the platform, e.g. continuous
> query notification.
>
> *4) Binary everywhere*
> We have full interoperability between platforms. Object created in .NET can
> be accessed from Java and C++ and vice versa. To achieve this we
> implemented special "binary protocol" on all platforms, so that every
> platform serialize objects in the same format. If you are to implement
> fully-fledged platform integration, you will have to implement binary
> marshaller on your own.
>
> *5) API layering*
> We have two C++ API layers - "common" [2] and everything else [3]. Common
> API is shared between C++ and .NET integrations. It is low-level interface
> to communicate with JVM (i.e. to perform forward and reverse JNI calls). It
> operates on pointers and JNI handles, and expects that you already
> serialized your objects in binary format amd put it to some memory chunk
> with certain layout. "Everything else" is high-level C++ API exposed to
> users.
>
> *6) Do not expect SWIG to help you a lot!*
> Taking all previous paragraphs in count - it is impossible to create
> valuable working implementation with SWIG. You will have to do a lot more
> than simple codegen.
>
> *7) Any other ways to integrate?*
> Yes, you can rely on our REST API [4]. We were planning to integrate with
> Node.JS this way - you just send HTTP requests to Ignite node(s). Every
> request is mapped to certain operation. However, this approach has two
> serious limitations:
> - It will be hard to employ advantages of binary interoperability
> - It will be very hard to implement server -> client callbacks which are of
> great importance for Ignite.
> As such you will have problems implementing compute API, continuous queries
> and so forth.
>
> Please let me know if you have any further questions.
>
> Vladimir.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-3568
> [2]
> https://github.com/apache/ignite/tree/master/modules/platforms/cpp/common
> [3] https://github.com/apache/ignite/tree/master/modules/platforms/cpp
> [4] https://apacheignite.readme.io/docs/rest-api

[dsetrakyan]

On Wed, Jan 25, 2017 at 1:12 AM, Shashank Gandham <[hidden email]
> wrote:

> So if we have to make full-fledged python platform for ignite. We can use
> C++ common  API to integrate and have an high level API of Python. Object
> serialisation can be achieved in python using libraries like marshal,
> pickle, shelve or something similar. Theoretically speaking this should be
> possible, or am I missing something?
>

As Vladimir mention, to add a new interoperable client in Python, you will
need to implement the BinaryMarshaller in Python in the same way we already
do in C++ and .NET. As a suggestion, you may be able to use Ignite C++ APIs
for Python integration. This way you will not have to worry about
marshalling. Python does support C callbacks as far as I know.

D.

[dmagda]

> On Jan 25, 2017, at 12:09 AM, Vladimir Ozerov <[hidden email]> wrote:
>
> *1) Not a client!*
> We do not have clients in classical sense. Our C++ and .NET integration are
> fully-fledged platforms. C++/.NET node can act both as a client and as a
> server.

This is a good point. To avoid confusion I renamed respective JIRA tickets and tasks submitted for GSOC
https://goo.gl/4SpTzK <https://goo.gl/4SpTzK>

BTW, Vovan don’t you want to become a mentor for one of the tasks? You can take over Python or R or suggest something else.

—
Denis

[dsetrakyan]

Should we add this to our documentation somewhere?

On Wed, Jan 25, 2017 at 2:59 PM, Denis Magda <[hidden email]> wrote:

>
> > On Jan 25, 2017, at 12:09 AM, Vladimir Ozerov <[hidden email]>
> wrote:
> >
> > *1) Not a client!*
> > We do not have clients in classical sense. Our C++ and .NET integration
> are
> > fully-fledged platforms. C++/.NET node can act both as a client and as a
> > server.
>
> This is a good point. To avoid confusion I renamed respective JIRA tickets
> and tasks submitted for GSOC
> https://goo.gl/4SpTzK <https://goo.gl/4SpTzK>
>
> BTW, Vovan don’t you want to become a mentor for one of the tasks? You can
> take over Python or R or suggest something else.
>
> —
> Denis

[Vladimir Ozerov]

Shashank, Dmitriy,

In principle it should be possible to use 3rd party marshalling frameworks,
but with some limitations.

Java side expects objects in certain format. At the very high level it
looks like [HEADER][BODY][FOOTER]. This is enough for Java to process the
object. While you *MUST* marshal an object with proper header and footer
which follows our binary format, you are allowed to marshal body as you
want.

We already employ this trick in some edge cases in Java and .NET when
binary marshalling cannot be used. The only drawback is that you will loose
interoperability: if object body doesn't follow Ignite's binary format, it
will be impossible to read it's fields on other platforms. Also you will
not be able to use these objects in SQL queries.

To summarize: you can implement only part of Ignite's binary protocol at
the cost of some intrinsic limitations.

Vladimir.

On Thu, Jan 26, 2017 at 1:42 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Wed, Jan 25, 2017 at 1:12 AM, Shashank Gandham <
> [hidden email]
> > wrote:
>
> > So if we have to make full-fledged python platform for ignite. We can use
> > C++ common  API to integrate and have an high level API of Python. Object
> > serialisation can be achieved in python using libraries like marshal,
> > pickle, shelve or something similar. Theoretically speaking this should
> be
> > possible, or am I missing something?
> >
>
> As Vladimir mention, to add a new interoperable client in Python, you will
> need to implement the BinaryMarshaller in Python in the same way we already
> do in C++ and .NET. As a suggestion, you may be able to use Ignite C++ APIs
> for Python integration. This way you will not have to worry about
> marshalling. Python does support C callbacks as far as I know.
>
> D.
>