Thin client and binary metadata

[Vladimir Ozerov]

Igniters,

As you probably know, we are working on a new protocol for thin client. It
would allow users to create integrations with other languages and platforms
(e.g. Go, R, etc). This protocol should be simple enough, otherwise no-one
would ever implement anything on top of it.

There is a problem with binary object metadata. Currently when we serialize
objects we check if any new metadata is available, and if yes - we push it
to the cluster before serialization is finished. This way we always have
metadata for all objects in the cluster. Metadata allow us to implement
certain features - introspection (usability) and performance (so called
"compact footer" optimization). Bad thing is that metadata management would
make client implementation much more complex.

I feel that in some cases clients might decide not to implement metadata
management for the sake of simplicity. Technically this is very easy - just
disable "compactFooter" optimization and write field IDs during
serialization. But I remember that there were some ideas to deprecate
"compactFooter=false" at all.

My question is: are fine with idea that objects without metadata will be
normal case in Ignite?

Vladimir.

[Pavel Tupitsyn]

In my understanding such objects are completely fine.

Even typeId is not necessary, since we have
GridBinaryMarshaller.UNREGISTERED_TYPE_ID mode,
where full type name is included in binary object header.

My testing shows that Ignite works fine with such objects (including SQL).

Pavel

On Mon, Sep 4, 2017 at 10:40 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Igniters,
>
> As you probably know, we are working on a new protocol for thin client. It
> would allow users to create integrations with other languages and platforms
> (e.g. Go, R, etc). This protocol should be simple enough, otherwise no-one
> would ever implement anything on top of it.
>
> There is a problem with binary object metadata. Currently when we serialize
> objects we check if any new metadata is available, and if yes - we push it
> to the cluster before serialization is finished. This way we always have
> metadata for all objects in the cluster. Metadata allow us to implement
> certain features - introspection (usability) and performance (so called
> "compact footer" optimization). Bad thing is that metadata management would
> make client implementation much more complex.
>
> I feel that in some cases clients might decide not to implement metadata
> management for the sake of simplicity. Technically this is very easy - just
> disable "compactFooter" optimization and write field IDs during
> serialization. But I remember that there were some ideas to deprecate
> "compactFooter=false" at all.
>
> My question is: are fine with idea that objects without metadata will be
> normal case in Ignite?
>
> Vladimir.
>

[dsetrakyan]

I think we should support and clearly document both modes, with and without
metadata.

On Mon, Sep 4, 2017 at 2:06 AM, Pavel Tupitsyn <[hidden email]> wrote:

> In my understanding such objects are completely fine.
>
> Even typeId is not necessary, since we have
> GridBinaryMarshaller.UNREGISTERED_TYPE_ID mode,
> where full type name is included in binary object header.
>
> My testing shows that Ignite works fine with such objects (including SQL).
>
> Pavel
>
> On Mon, Sep 4, 2017 at 10:40 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > As you probably know, we are working on a new protocol for thin client.
> It
> > would allow users to create integrations with other languages and
> platforms
> > (e.g. Go, R, etc). This protocol should be simple enough, otherwise
> no-one
> > would ever implement anything on top of it.
> >
> > There is a problem with binary object metadata. Currently when we
> serialize
> > objects we check if any new metadata is available, and if yes - we push
> it
> > to the cluster before serialization is finished. This way we always have
> > metadata for all objects in the cluster. Metadata allow us to implement
> > certain features - introspection (usability) and performance (so called
> > "compact footer" optimization). Bad thing is that metadata management
> would
> > make client implementation much more complex.
> >
> > I feel that in some cases clients might decide not to implement metadata
> > management for the sake of simplicity. Technically this is very easy -
> just
> > disable "compactFooter" optimization and write field IDs during
> > serialization. But I remember that there were some ideas to deprecate
> > "compactFooter=false" at all.
> >
> > My question is: are fine with idea that objects without metadata will be
> > normal case in Ignite?
> >
> > Vladimir.
> >
>

[Pavel Tupitsyn]

Ok, I think we all agree on that.
Client implementors can decide whether they call metadata API or not.

Now onto the API itself: I've outlined the proposal in JIRA [1].
Let me know your thoughts.

[1] https://issues.apache.org/jira/browse/IGNITE-6258

On Mon, Sep 4, 2017 at 5:06 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> I think we should support and clearly document both modes, with and without
> metadata.
>
> On Mon, Sep 4, 2017 at 2:06 AM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > In my understanding such objects are completely fine.
> >
> > Even typeId is not necessary, since we have
> > GridBinaryMarshaller.UNREGISTERED_TYPE_ID mode,
> > where full type name is included in binary object header.
> >
> > My testing shows that Ignite works fine with such objects (including
> SQL).
> >
> > Pavel
> >
> > On Mon, Sep 4, 2017 at 10:40 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Igniters,
> > >
> > > As you probably know, we are working on a new protocol for thin client.
> > It
> > > would allow users to create integrations with other languages and
> > platforms
> > > (e.g. Go, R, etc). This protocol should be simple enough, otherwise
> > no-one
> > > would ever implement anything on top of it.
> > >
> > > There is a problem with binary object metadata. Currently when we
> > serialize
> > > objects we check if any new metadata is available, and if yes - we push
> > it
> > > to the cluster before serialization is finished. This way we always
> have
> > > metadata for all objects in the cluster. Metadata allow us to implement
> > > certain features - introspection (usability) and performance (so called
> > > "compact footer" optimization). Bad thing is that metadata management
> > would
> > > make client implementation much more complex.
> > >
> > > I feel that in some cases clients might decide not to implement
> metadata
> > > management for the sake of simplicity. Technically this is very easy -
> > just
> > > disable "compactFooter" optimization and write field IDs during
> > > serialization. But I remember that there were some ideas to deprecate
> > > "compactFooter=false" at all.
> > >
> > > My question is: are fine with idea that objects without metadata will
> be
> > > normal case in Ignite?
> > >
> > > Vladimir.
> > >
> >
>