Apache Ignite Developers - Legacy Mail Archive

Thin Client Protocol documentation

Classic

List

Threaded

27 messages Options

Pavel Tupitsyn

Thin Client Protocol documentation

Igniters,

I've put together a detailed description of our Thin Client protocol
in form of IEP on wiki:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol

To clarify:
- Protocol implementation is in master (see ClientMessageParser class)
- Protocol has not been released yet, so we are free to change anything
- Protocol is only used by .NET Thin Client for now, but is supposed to be
used from other languages by third party contributors
- More operations will be added in future, this is a first set of them,
cache-related

Please review the document and let me know your thoughts.
Is there anything missing or wrong?

We should make sure that the foundation is solid and extensible.

Thanks,
Pavel

Prachi Garg

Re: Thin Client Protocol documentation

Thanks Pavel! The document has good information. I'll create one on
readme.io; will also add some examples there.

On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <[hidden email]>
wrote:

> Igniters,
>
> I've put together a detailed description of our Thin Client protocol
> in form of IEP on wiki:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 9+Thin+Client+Protocol
>
>
> To clarify:
> - Protocol implementation is in master (see ClientMessageParser class)
> - Protocol has not been released yet, so we are free to change anything
> - Protocol is only used by .NET Thin Client for now, but is supposed to be
> used from other languages by third party contributors
> - More operations will be added in future, this is a first set of them,
> cache-related
>
>
> Please review the document and let me know your thoughts.
> Is there anything missing or wrong?
>
> We should make sure that the foundation is solid and extensible.
>
>
> Thanks,
> Pavel
>

Sergey Kozlov

Re: Thin Client Protocol documentation

Pavel

Thanks for the document and your efforts for new protocol. It was really
helpful for playing around the python thin client design.

Could you explain some things that were still not clear for binary object
format:

1. What a reason to introduce separated type codes for arrays? Why just we
can't use the following?
*<1 byte universal array code>*
*<1 byte primitive code>*
*<4 bytes length>*
*<N bytes data>*

We get 1 byte overhead but save 10 bytes in the code table. For arrays the
overhead is really insignificant:10 longs array takes now 1+4+4*10=45 bytes
vs 1+1+4+4*10=46 bytes for the approach
Moreover for that appoach a new primitive code will be available for using
for array immediately.

2. Why the arrays force to use a selected type? For python there's no
limitations to use different types across one array (list). Would be good
to introduce a new type that will allow that. It could be look like
following
*<1 byte universal array code>*
*<1 byte no common type code*> <-- this says that every item must provide
its date type code like it does regular primitive data
*<4 bytes length>*
*<1 byte item 0 type code>* <-- item provides its code
*<N byte item 0 data>* <-- item provides its data
*<1 byte item 1 type code>*
*<N byte item 1 data>*
etc

Also that allow to put nested arrays without changes in type code table!
For instance if we want to store 9 longs and 1 boolean it will take
now 1+1+4+(1+9)*4+(1+1)=48
bytes (vs 45 bytes to store as 10 longs as usual).

3. Ther's only one way to store a dictionary (key-value) structure as value
in the cache via Complex Object. But it looks like overcomplicated. I
suppose to introduce a code for that
*<1 byte key-value dictionary code>*

*<4 bytes length>*
*<1 byte key 1 **name **type code>*
*<N byte key 1 name data>*
*<1 byte value 1 type code>*
*<N byte value 1 value>*
*<1 byte key 2 **name **type code>*
*<N byte key 2 name data>*
*<1 byte value 2 type code>*
*<N byte value 2 value>*
etc

Also that allow to put nested dictionaries without changes in type code
table!
Of course for the appoach above we get significat overhead for key
storing. But I think it is acceptable for some cases and definitely ok for
Python

On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]> wrote:

> Thanks Pavel! The document has good information. I'll create one on
> readme.io; will also add some examples there.
>
> On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > I've put together a detailed description of our Thin Client protocol
> > in form of IEP on wiki:
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 9+Thin+Client+Protocol
> >
> >
> > To clarify:
> > - Protocol implementation is in master (see ClientMessageParser class)
> > - Protocol has not been released yet, so we are free to change anything
> > - Protocol is only used by .NET Thin Client for now, but is supposed to
> be
> > used from other languages by third party contributors
> > - More operations will be added in future, this is a first set of them,
> > cache-related
> >
> >
> > Please review the document and let me know your thoughts.
> > Is there anything missing or wrong?
> >
> > We should make sure that the foundation is solid and extensible.
> >
> >
> > Thanks,
> > Pavel
> >
>

--
Sergey Kozlov
GridGain Systems
www.gridgain.com

Pavel Tupitsyn

Re: Thin Client Protocol documentation

Sergey,

1. Code table size does not affect anything, as I understand, so there is
no reason to introduce an extra byte.
2. We have object arrays (code 23), I forgot to mention them, fixed.
3. Also forgot, see code 25 in the updated document.

Also note that operation codes have been updated (grouped by purpose) as
part of https://issues.apache.org/jira/browse/IGNITE-6989.

Thanks,
Pavel

On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <[hidden email]> wrote:

> Pavel
>
> Thanks for the document and your efforts for new protocol. It was really
> helpful for playing around the python thin client design.
>
> Could you explain some things that were still not clear for binary object
> format:
>
> 1. What a reason to introduce separated type codes for arrays? Why just we
> can't use the following?
> *<1 byte universal array code>*
> *<1 byte primitive code>*
> *<4 bytes length>*
> *<N bytes data>*
>
> We get 1 byte overhead but save 10 bytes in the code table. For arrays the
> overhead is really insignificant:10 longs array takes now 1+4+4*10=45 bytes
> vs 1+1+4+4*10=46 bytes for the approach
> Moreover for that appoach a new primitive code will be available for using
> for array immediately.
>
> 2. Why the arrays force to use a selected type? For python there's no
> limitations to use different types across one array (list). Would be good
> to introduce a new type that will allow that. It could be look like
> following
> *<1 byte universal array code>*
> *<1 byte no common type code*> <-- this says that every item must provide
> its date type code like it does regular primitive data
> *<4 bytes length>*
> *<1 byte item 0 type code>* <-- item provides its code
> *<N byte item 0 data>* <-- item provides its data
> *<1 byte item 1 type code>*
> *<N byte item 1 data>*
> etc
>
> Also that allow to put nested arrays without changes in type code table!
> For instance if we want to store 9 longs and 1 boolean it will take
> now 1+1+4+(1+9)*4+(1+1)=48
> bytes (vs 45 bytes to store as 10 longs as usual).
>
> 3. Ther's only one way to store a dictionary (key-value) structure as value
> in the cache via Complex Object. But it looks like overcomplicated. I
> suppose to introduce a code for that
> *<1 byte key-value dictionary code>*
>
> *<4 bytes length>*
> *<1 byte key 1 **name **type code>*
> *<N byte key 1 name data>*
> *<1 byte value 1 type code>*
> *<N byte value 1 value>*
> *<1 byte key 2 **name **type code>*
> *<N byte key 2 name data>*
> *<1 byte value 2 type code>*
> *<N byte value 2 value>*
> etc
>
> Also that allow to put nested dictionaries without changes in type code
> table!
> Of course for the appoach above we get significat overhead for key
> storing. But I think it is acceptable for some cases and definitely ok for
> Python
>
>
>
>
> On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]> wrote:
>
> > Thanks Pavel! The document has good information. I'll create one on
> > readme.io; will also add some examples there.
> >
> > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <[hidden email]>
> > wrote:
> >
> > > Igniters,
> > >
> > > I've put together a detailed description of our Thin Client protocol
> > > in form of IEP on wiki:
> > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > 9+Thin+Client+Protocol
> > >
> > >
> > > To clarify:
> > > - Protocol implementation is in master (see ClientMessageParser class)
> > > - Protocol has not been released yet, so we are free to change anything
> > > - Protocol is only used by .NET Thin Client for now, but is supposed to
> > be
> > > used from other languages by third party contributors
> > > - More operations will be added in future, this is a first set of them,
> > > cache-related
> > >
> > >
> > > Please review the document and let me know your thoughts.
> > > Is there anything missing or wrong?
> > >
> > > We should make sure that the foundation is solid and extensible.
> > >
> > >
> > > Thanks,
> > > Pavel
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Sergey Kozlov

Re: Thin Client Protocol documentation

Pavel

Thanks for explanations!

On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <[hidden email]>
wrote:

> Sergey,
>
> 1. Code table size does not affect anything, as I understand, so there is
> no reason to introduce an extra byte.
> 2. We have object arrays (code 23), I forgot to mention them, fixed.
> 3. Also forgot, see code 25 in the updated document.
>
> Also note that operation codes have been updated (grouped by purpose) as
> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>
> Thanks,
> Pavel
>
> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <[hidden email]>
> wrote:
>
> > Pavel
> >
> > Thanks for the document and your efforts for new protocol. It was really
> > helpful for playing around the python thin client design.
> >
> > Could you explain some things that were still not clear for binary object
> > format:
> >
> > 1. What a reason to introduce separated type codes for arrays? Why just
> we
> > can't use the following?
> > *<1 byte universal array code>*
> > *<1 byte primitive code>*
> > *<4 bytes length>*
> > *<N bytes data>*
> >
> > We get 1 byte overhead but save 10 bytes in the code table. For arrays
> the
> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45
> bytes
> > vs 1+1+4+4*10=46 bytes for the approach
> > Moreover for that appoach a new primitive code will be available for
> using
> > for array immediately.
> >
> > 2. Why the arrays force to use a selected type? For python there's no
> > limitations to use different types across one array (list). Would be good
> > to introduce a new type that will allow that. It could be look like
> > following
> > *<1 byte universal array code>*
> > *<1 byte no common type code*> <-- this says that every item must provide
> > its date type code like it does regular primitive data
> > *<4 bytes length>*
> > *<1 byte item 0 type code>* <-- item provides its code
> > *<N byte item 0 data>* <-- item provides its data
> > *<1 byte item 1 type code>*
> > *<N byte item 1 data>*
> > etc
> >
> > Also that allow to put nested arrays without changes in type code table!
> > For instance if we want to store 9 longs and 1 boolean it will take
> > now 1+1+4+(1+9)*4+(1+1)=48
> > bytes (vs 45 bytes to store as 10 longs as usual).
> >
> > 3. Ther's only one way to store a dictionary (key-value) structure as
> value
> > in the cache via Complex Object. But it looks like overcomplicated. I
> > suppose to introduce a code for that
> > *<1 byte key-value dictionary code>*
> >
> > *<4 bytes length>*
> > *<1 byte key 1 **name **type code>*
> > *<N byte key 1 name data>*
> > *<1 byte value 1 type code>*
> > *<N byte value 1 value>*
> > *<1 byte key 2 **name **type code>*
> > *<N byte key 2 name data>*
> > *<1 byte value 2 type code>*
> > *<N byte value 2 value>*
> > etc
> >
> > Also that allow to put nested dictionaries without changes in type code
> > table!
> > Of course for the appoach above we get significat overhead for key
> > storing. But I think it is acceptable for some cases and definitely ok
> for
> > Python
> >
> >
> >
> >
> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]> wrote:
> >
> > > Thanks Pavel! The document has good information. I'll create one on
> > > readme.io; will also add some examples there.
> > >
> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <[hidden email]>
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I've put together a detailed description of our Thin Client protocol
> > > > in form of IEP on wiki:
> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > > 9+Thin+Client+Protocol
> > > >
> > > >
> > > > To clarify:
> > > > - Protocol implementation is in master (see ClientMessageParser
> class)
> > > > - Protocol has not been released yet, so we are free to change
> anything
> > > > - Protocol is only used by .NET Thin Client for now, but is supposed
> to
> > > be
> > > > used from other languages by third party contributors
> > > > - More operations will be added in future, this is a first set of
> them,
> > > > cache-related
> > > >
> > > >
> > > > Please review the document and let me know your thoughts.
> > > > Is there anything missing or wrong?
> > > >
> > > > We should make sure that the foundation is solid and extensible.
> > > >
> > > >
> > > > Thanks,
> > > > Pavel
> > > >
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>

--
Sergey Kozlov
GridGain Systems
www.gridgain.com

Sergey Kozlov

Re: Thin Client Protocol documentation

Pavel

Could you update the page by following:

- String, date, UUID arrays allow to put NULL. Due to that every item in
the array written as type code byte (default item type or null type code) +
type data. It should be detailed explained (looks like that the table
should have an addtional column called for instance "nullable")

- UUID type takes 16 bytes length

thanks

On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <[hidden email]> wrote:

> Pavel
>
> Thanks for explanations!
>
> On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
>> Sergey,
>>
>> 1. Code table size does not affect anything, as I understand, so there is
>> no reason to introduce an extra byte.
>> 2. We have object arrays (code 23), I forgot to mention them, fixed.
>> 3. Also forgot, see code 25 in the updated document.
>>
>> Also note that operation codes have been updated (grouped by purpose) as
>> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>>
>> Thanks,
>> Pavel
>>
>> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <[hidden email]>
>> wrote:
>>
>> > Pavel
>> >
>> > Thanks for the document and your efforts for new protocol. It was really
>> > helpful for playing around the python thin client design.
>> >
>> > Could you explain some things that were still not clear for binary
>> object
>> > format:
>> >
>> > 1. What a reason to introduce separated type codes for arrays? Why just
>> we
>> > can't use the following?
>> > *<1 byte universal array code>*
>> > *<1 byte primitive code>*
>> > *<4 bytes length>*
>> > *<N bytes data>*
>> >
>> > We get 1 byte overhead but save 10 bytes in the code table. For arrays
>> the
>> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45
>> bytes
>> > vs 1+1+4+4*10=46 bytes for the approach
>> > Moreover for that appoach a new primitive code will be available for
>> using
>> > for array immediately.
>> >
>> > 2. Why the arrays force to use a selected type? For python there's no
>> > limitations to use different types across one array (list). Would be
>> good
>> > to introduce a new type that will allow that. It could be look like
>> > following
>> > *<1 byte universal array code>*
>> > *<1 byte no common type code*> <-- this says that every item must
>> provide
>> > its date type code like it does regular primitive data
>> > *<4 bytes length>*
>> > *<1 byte item 0 type code>* <-- item provides its code
>> > *<N byte item 0 data>* <-- item provides its data
>> > *<1 byte item 1 type code>*
>> > *<N byte item 1 data>*
>> > etc
>> >
>> > Also that allow to put nested arrays without changes in type code table!
>> > For instance if we want to store 9 longs and 1 boolean it will take
>> > now 1+1+4+(1+9)*4+(1+1)=48
>> > bytes (vs 45 bytes to store as 10 longs as usual).
>> >
>> > 3. Ther's only one way to store a dictionary (key-value) structure as
>> value
>> > in the cache via Complex Object. But it looks like overcomplicated. I
>> > suppose to introduce a code for that
>> > *<1 byte key-value dictionary code>*
>> >
>> > *<4 bytes length>*
>> > *<1 byte key 1 **name **type code>*
>> > *<N byte key 1 name data>*
>> > *<1 byte value 1 type code>*
>> > *<N byte value 1 value>*
>> > *<1 byte key 2 **name **type code>*
>> > *<N byte key 2 name data>*
>> > *<1 byte value 2 type code>*
>> > *<N byte value 2 value>*
>> > etc
>> >
>> > Also that allow to put nested dictionaries without changes in type code
>> > table!
>> > Of course for the appoach above we get significat overhead for key
>> > storing. But I think it is acceptable for some cases and definitely ok
>> for
>> > Python
>> >
>> >
>> >
>> >
>> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]>
>> wrote:
>> >
>> > > Thanks Pavel! The document has good information. I'll create one on
>> > > readme.io; will also add some examples there.
>> > >
>> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <[hidden email]
>> >
>> > > wrote:
>> > >
>> > > > Igniters,
>> > > >
>> > > > I've put together a detailed description of our Thin Client protocol
>> > > > in form of IEP on wiki:
>> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>> > > > 9+Thin+Client+Protocol
>> > > >
>> > > >
>> > > > To clarify:
>> > > > - Protocol implementation is in master (see ClientMessageParser
>> class)
>> > > > - Protocol has not been released yet, so we are free to change
>> anything
>> > > > - Protocol is only used by .NET Thin Client for now, but is
>> supposed to
>> > > be
>> > > > used from other languages by third party contributors
>> > > > - More operations will be added in future, this is a first set of
>> them,
>> > > > cache-related
>> > > >
>> > > >
>> > > > Please review the document and let me know your thoughts.
>> > > > Is there anything missing or wrong?
>> > > >
>> > > > We should make sure that the foundation is solid and extensible.
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Pavel
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Sergey Kozlov
>> > GridGain Systems
>> > www.gridgain.com
>> >
>>
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

--
Sergey Kozlov
GridGain Systems
www.gridgain.com

Pavel Tupitsyn

Re: Thin Client Protocol documentation

Sergey, good point, done.

On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <[hidden email]> wrote:

> Pavel
>
> Could you update the page by following:
>
> - String, date, UUID arrays allow to put NULL. Due to that every item in
> the array written as type code byte (default item type or null type code) +
> type data. It should be detailed explained (looks like that the table
> should have an addtional column called for instance "nullable")
>
> - UUID type takes 16 bytes length
>
> thanks
>
>
> On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <[hidden email]>
> wrote:
>
> > Pavel
> >
> > Thanks for explanations!
> >
> > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <[hidden email]>
> > wrote:
> >
> >> Sergey,
> >>
> >> 1. Code table size does not affect anything, as I understand, so there
> is
> >> no reason to introduce an extra byte.
> >> 2. We have object arrays (code 23), I forgot to mention them, fixed.
> >> 3. Also forgot, see code 25 in the updated document.
> >>
> >> Also note that operation codes have been updated (grouped by purpose) as
> >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
> >>
> >> Thanks,
> >> Pavel
> >>
> >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <[hidden email]>
> >> wrote:
> >>
> >> > Pavel
> >> >
> >> > Thanks for the document and your efforts for new protocol. It was
> really
> >> > helpful for playing around the python thin client design.
> >> >
> >> > Could you explain some things that were still not clear for binary
> >> object
> >> > format:
> >> >
> >> > 1. What a reason to introduce separated type codes for arrays? Why
> just
> >> we
> >> > can't use the following?
> >> > *<1 byte universal array code>*
> >> > *<1 byte primitive code>*
> >> > *<4 bytes length>*
> >> > *<N bytes data>*
> >> >
> >> > We get 1 byte overhead but save 10 bytes in the code table. For arrays
> >> the
> >> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45
> >> bytes
> >> > vs 1+1+4+4*10=46 bytes for the approach
> >> > Moreover for that appoach a new primitive code will be available for
> >> using
> >> > for array immediately.
> >> >
> >> > 2. Why the arrays force to use a selected type? For python there's no
> >> > limitations to use different types across one array (list). Would be
> >> good
> >> > to introduce a new type that will allow that. It could be look like
> >> > following
> >> > *<1 byte universal array code>*
> >> > *<1 byte no common type code*> <-- this says that every item must
> >> provide
> >> > its date type code like it does regular primitive data
> >> > *<4 bytes length>*
> >> > *<1 byte item 0 type code>* <-- item provides its code
> >> > *<N byte item 0 data>* <-- item provides its data
> >> > *<1 byte item 1 type code>*
> >> > *<N byte item 1 data>*
> >> > etc
> >> >
> >> > Also that allow to put nested arrays without changes in type code
> table!
> >> > For instance if we want to store 9 longs and 1 boolean it will take
> >> > now 1+1+4+(1+9)*4+(1+1)=48
> >> > bytes (vs 45 bytes to store as 10 longs as usual).
> >> >
> >> > 3. Ther's only one way to store a dictionary (key-value) structure as
> >> value
> >> > in the cache via Complex Object. But it looks like overcomplicated. I
> >> > suppose to introduce a code for that
> >> > *<1 byte key-value dictionary code>*
> >> >
> >> > *<4 bytes length>*
> >> > *<1 byte key 1 **name **type code>*
> >> > *<N byte key 1 name data>*
> >> > *<1 byte value 1 type code>*
> >> > *<N byte value 1 value>*
> >> > *<1 byte key 2 **name **type code>*
> >> > *<N byte key 2 name data>*
> >> > *<1 byte value 2 type code>*
> >> > *<N byte value 2 value>*
> >> > etc
> >> >
> >> > Also that allow to put nested dictionaries without changes in type
> code
> >> > table!
> >> > Of course for the appoach above we get significat overhead for key
> >> > storing. But I think it is acceptable for some cases and definitely ok
> >> for
> >> > Python
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]>
> >> wrote:
> >> >
> >> > > Thanks Pavel! The document has good information. I'll create one on
> >> > > readme.io; will also add some examples there.
> >> > >
> >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
> [hidden email]
> >> >
> >> > > wrote:
> >> > >
> >> > > > Igniters,
> >> > > >
> >> > > > I've put together a detailed description of our Thin Client
> protocol
> >> > > > in form of IEP on wiki:
> >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> >> > > > 9+Thin+Client+Protocol
> >> > > >
> >> > > >
> >> > > > To clarify:
> >> > > > - Protocol implementation is in master (see ClientMessageParser
> >> class)
> >> > > > - Protocol has not been released yet, so we are free to change
> >> anything
> >> > > > - Protocol is only used by .NET Thin Client for now, but is
> >> supposed to
> >> > > be
> >> > > > used from other languages by third party contributors
> >> > > > - More operations will be added in future, this is a first set of
> >> them,
> >> > > > cache-related
> >> > > >
> >> > > >
> >> > > > Please review the document and let me know your thoughts.
> >> > > > Is there anything missing or wrong?
> >> > > >
> >> > > > We should make sure that the foundation is solid and extensible.
> >> > > >
> >> > > >
> >> > > > Thanks,
> >> > > > Pavel
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Sergey Kozlov
> >> > GridGain Systems
> >> > www.gridgain.com
> >> >
> >>
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Prachi Garg

Re: Thin Client Protocol documentation

Hi Pavel,

In the operation request, what does the 'flags' parameter mean?

Thanks,
-Prachi

On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn <[hidden email]>
wrote:

> Sergey, good point, done.
>
> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <[hidden email]>
> wrote:
>
> > Pavel
> >
> > Could you update the page by following:
> >
> > - String, date, UUID arrays allow to put NULL. Due to that every item in
> > the array written as type code byte (default item type or null type
> code) +
> > type data. It should be detailed explained (looks like that the table
> > should have an addtional column called for instance "nullable")
> >
> > - UUID type takes 16 bytes length
> >
> > thanks
> >
> >
> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <[hidden email]>
> > wrote:
> >
> > > Pavel
> > >
> > > Thanks for explanations!
> > >
> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <[hidden email]>
> > > wrote:
> > >
> > >> Sergey,
> > >>
> > >> 1. Code table size does not affect anything, as I understand, so there
> > is
> > >> no reason to introduce an extra byte.
> > >> 2. We have object arrays (code 23), I forgot to mention them, fixed.
> > >> 3. Also forgot, see code 25 in the updated document.
> > >>
> > >> Also note that operation codes have been updated (grouped by purpose)
> as
> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
> > >>
> > >> Thanks,
> > >> Pavel
> > >>
> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <[hidden email]>
> > >> wrote:
> > >>
> > >> > Pavel
> > >> >
> > >> > Thanks for the document and your efforts for new protocol. It was
> > really
> > >> > helpful for playing around the python thin client design.
> > >> >
> > >> > Could you explain some things that were still not clear for binary
> > >> object
> > >> > format:
> > >> >
> > >> > 1. What a reason to introduce separated type codes for arrays? Why
> > just
> > >> we
> > >> > can't use the following?
> > >> > *<1 byte universal array code>*
> > >> > *<1 byte primitive code>*
> > >> > *<4 bytes length>*
> > >> > *<N bytes data>*
> > >> >
> > >> > We get 1 byte overhead but save 10 bytes in the code table. For
> arrays
> > >> the
> > >> > overhead is really insignificant:10 longs array takes now
> 1+4+4*10=45
> > >> bytes
> > >> > vs 1+1+4+4*10=46 bytes for the approach
> > >> > Moreover for that appoach a new primitive code will be available for
> > >> using
> > >> > for array immediately.
> > >> >
> > >> > 2. Why the arrays force to use a selected type? For python there's
> no
> > >> > limitations to use different types across one array (list). Would be
> > >> good
> > >> > to introduce a new type that will allow that. It could be look like
> > >> > following
> > >> > *<1 byte universal array code>*
> > >> > *<1 byte no common type code*> <-- this says that every item must
> > >> provide
> > >> > its date type code like it does regular primitive data
> > >> > *<4 bytes length>*
> > >> > *<1 byte item 0 type code>* <-- item provides its code
> > >> > *<N byte item 0 data>* <-- item provides its data
> > >> > *<1 byte item 1 type code>*
> > >> > *<N byte item 1 data>*
> > >> > etc
> > >> >
> > >> > Also that allow to put nested arrays without changes in type code
> > table!
> > >> > For instance if we want to store 9 longs and 1 boolean it will take
> > >> > now 1+1+4+(1+9)*4+(1+1)=48
> > >> > bytes (vs 45 bytes to store as 10 longs as usual).
> > >> >
> > >> > 3. Ther's only one way to store a dictionary (key-value) structure
> as
> > >> value
> > >> > in the cache via Complex Object. But it looks like overcomplicated.
> I
> > >> > suppose to introduce a code for that
> > >> > *<1 byte key-value dictionary code>*
> > >> >
> > >> > *<4 bytes length>*
> > >> > *<1 byte key 1 **name **type code>*
> > >> > *<N byte key 1 name data>*
> > >> > *<1 byte value 1 type code>*
> > >> > *<N byte value 1 value>*
> > >> > *<1 byte key 2 **name **type code>*
> > >> > *<N byte key 2 name data>*
> > >> > *<1 byte value 2 type code>*
> > >> > *<N byte value 2 value>*
> > >> > etc
> > >> >
> > >> > Also that allow to put nested dictionaries without changes in type
> > code
> > >> > table!
> > >> > Of course for the appoach above we get significat overhead for key
> > >> > storing. But I think it is acceptable for some cases and definitely
> ok
> > >> for
> > >> > Python
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]>
> > >> wrote:
> > >> >
> > >> > > Thanks Pavel! The document has good information. I'll create one
> on
> > >> > > readme.io; will also add some examples there.
> > >> > >
> > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
> > [hidden email]
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Igniters,
> > >> > > >
> > >> > > > I've put together a detailed description of our Thin Client
> > protocol
> > >> > > > in form of IEP on wiki:
> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > >> > > > 9+Thin+Client+Protocol
> > >> > > >
> > >> > > >
> > >> > > > To clarify:
> > >> > > > - Protocol implementation is in master (see ClientMessageParser
> > >> class)
> > >> > > > - Protocol has not been released yet, so we are free to change
> > >> anything
> > >> > > > - Protocol is only used by .NET Thin Client for now, but is
> > >> supposed to
> > >> > > be
> > >> > > > used from other languages by third party contributors
> > >> > > > - More operations will be added in future, this is a first set
> of
> > >> them,
> > >> > > > cache-related
> > >> > > >
> > >> > > >
> > >> > > > Please review the document and let me know your thoughts.
> > >> > > > Is there anything missing or wrong?
> > >> > > >
> > >> > > > We should make sure that the foundation is solid and extensible.
> > >> > > >
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Pavel
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Sergey Kozlov
> > >> > GridGain Systems
> > >> > www.gridgain.com
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Sergey Kozlov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>

Pavel Tupitsyn-3

Re: Thin Client Protocol documentation

Hi Prachi,

Flags parameter in all cache operations is reserved for things like
withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries,
withPartitionRecover.
See methods in IgniteCache interface.

Thanks,
Pavel

On Wed, Nov 29, 2017 at 7:10 PM, Prachi Garg <[hidden email]> wrote:

> Hi Pavel,
>
> In the operation request, what does the 'flags' parameter mean?
>
>
> Thanks,
> -Prachi
>
> On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
>> Sergey, good point, done.
>>
>> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <[hidden email]>
>> wrote:
>>
>> > Pavel
>> >
>> > Could you update the page by following:
>> >
>> > - String, date, UUID arrays allow to put NULL. Due to that every item
>> in
>> > the array written as type code byte (default item type or null type
>> code) +
>> > type data. It should be detailed explained (looks like that the table
>> > should have an addtional column called for instance "nullable")
>> >
>> > - UUID type takes 16 bytes length
>> >
>> > thanks
>> >
>> >
>> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <[hidden email]>
>> > wrote:
>> >
>> > > Pavel
>> > >
>> > > Thanks for explanations!
>> > >
>> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <[hidden email]
>> >
>> > > wrote:
>> > >
>> > >> Sergey,
>> > >>
>> > >> 1. Code table size does not affect anything, as I understand, so
>> there
>> > is
>> > >> no reason to introduce an extra byte.
>> > >> 2. We have object arrays (code 23), I forgot to mention them, fixed.
>> > >> 3. Also forgot, see code 25 in the updated document.
>> > >>
>> > >> Also note that operation codes have been updated (grouped by
>> purpose) as
>> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>> > >>
>> > >> Thanks,
>> > >> Pavel
>> > >>
>> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <[hidden email]
>> >
>> > >> wrote:
>> > >>
>> > >> > Pavel
>> > >> >
>> > >> > Thanks for the document and your efforts for new protocol. It was
>> > really
>> > >> > helpful for playing around the python thin client design.
>> > >> >
>> > >> > Could you explain some things that were still not clear for binary
>> > >> object
>> > >> > format:
>> > >> >
>> > >> > 1. What a reason to introduce separated type codes for arrays? Why
>> > just
>> > >> we
>> > >> > can't use the following?
>> > >> > *<1 byte universal array code>*
>> > >> > *<1 byte primitive code>*
>> > >> > *<4 bytes length>*
>> > >> > *<N bytes data>*
>> > >> >
>> > >> > We get 1 byte overhead but save 10 bytes in the code table. For
>> arrays
>> > >> the
>> > >> > overhead is really insignificant:10 longs array takes now
>> 1+4+4*10=45
>> > >> bytes
>> > >> > vs 1+1+4+4*10=46 bytes for the approach
>> > >> > Moreover for that appoach a new primitive code will be available
>> for
>> > >> using
>> > >> > for array immediately.
>> > >> >
>> > >> > 2. Why the arrays force to use a selected type? For python there's
>> no
>> > >> > limitations to use different types across one array (list). Would
>> be
>> > >> good
>> > >> > to introduce a new type that will allow that. It could be look like
>> > >> > following
>> > >> > *<1 byte universal array code>*
>> > >> > *<1 byte no common type code*> <-- this says that every item must
>> > >> provide
>> > >> > its date type code like it does regular primitive data
>> > >> > *<4 bytes length>*
>> > >> > *<1 byte item 0 type code>* <-- item provides its code
>> > >> > *<N byte item 0 data>* <-- item provides its data
>> > >> > *<1 byte item 1 type code>*
>> > >> > *<N byte item 1 data>*
>> > >> > etc
>> > >> >
>> > >> > Also that allow to put nested arrays without changes in type code
>> > table!
>> > >> > For instance if we want to store 9 longs and 1 boolean it will take
>> > >> > now 1+1+4+(1+9)*4+(1+1)=48
>> > >> > bytes (vs 45 bytes to store as 10 longs as usual).
>> > >> >
>> > >> > 3. Ther's only one way to store a dictionary (key-value) structure
>> as
>> > >> value
>> > >> > in the cache via Complex Object. But it looks like
>> overcomplicated. I
>> > >> > suppose to introduce a code for that
>> > >> > *<1 byte key-value dictionary code>*
>> > >> >
>> > >> > *<4 bytes length>*
>> > >> > *<1 byte key 1 **name **type code>*
>> > >> > *<N byte key 1 name data>*
>> > >> > *<1 byte value 1 type code>*
>> > >> > *<N byte value 1 value>*
>> > >> > *<1 byte key 2 **name **type code>*
>> > >> > *<N byte key 2 name data>*
>> > >> > *<1 byte value 2 type code>*
>> > >> > *<N byte value 2 value>*
>> > >> > etc
>> > >> >
>> > >> > Also that allow to put nested dictionaries without changes in type
>> > code
>> > >> > table!
>> > >> > Of course for the appoach above we get significat overhead for key
>> > >> > storing. But I think it is acceptable for some cases and
>> definitely ok
>> > >> for
>> > >> > Python
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]>
>> > >> wrote:
>> > >> >
>> > >> > > Thanks Pavel! The document has good information. I'll create one
>> on
>> > >> > > readme.io; will also add some examples there.
>> > >> > >
>> > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
>> > [hidden email]
>> > >> >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Igniters,
>> > >> > > >
>> > >> > > > I've put together a detailed description of our Thin Client
>> > protocol
>> > >> > > > in form of IEP on wiki:
>> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>> > >> > > > 9+Thin+Client+Protocol
>> > >> > > >
>> > >> > > >
>> > >> > > > To clarify:
>> > >> > > > - Protocol implementation is in master (see ClientMessageParser
>> > >> class)
>> > >> > > > - Protocol has not been released yet, so we are free to change
>> > >> anything
>> > >> > > > - Protocol is only used by .NET Thin Client for now, but is
>> > >> supposed to
>> > >> > > be
>> > >> > > > used from other languages by third party contributors
>> > >> > > > - More operations will be added in future, this is a first set
>> of
>> > >> them,
>> > >> > > > cache-related
>> > >> > > >
>> > >> > > >
>> > >> > > > Please review the document and let me know your thoughts.
>> > >> > > > Is there anything missing or wrong?
>> > >> > > >
>> > >> > > > We should make sure that the foundation is solid and
>> extensible.
>> > >> > > >
>> > >> > > >
>> > >> > > > Thanks,
>> > >> > > > Pavel
>> > >> > > >
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Sergey Kozlov
>> > >> > GridGain Systems
>> > >> > www.gridgain.com
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Sergey Kozlov
>> > > GridGain Systems
>> > > www.gridgain.com
>> > >
>> >
>> >
>> >
>> > --
>> > Sergey Kozlov
>> > GridGain Systems
>> > www.gridgain.com
>> >
>>
>
>

Prachi Garg

Re: Thin Client Protocol documentation

Pavel,

If the connection handshake fails, what should be the message length in the
response. When I try to fail the handshake, I get 32 as the message length.
Is this expected?

-Prachi

On Wed, Nov 29, 2017 at 8:20 AM, Pavel Tupitsyn <[hidden email]>
wrote:

> Hi Prachi,
>
> Flags parameter in all cache operations is reserved for things like
> withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries,
> withPartitionRecover.
> See methods in IgniteCache interface.
>
> Thanks,
> Pavel
>
> On Wed, Nov 29, 2017 at 7:10 PM, Prachi Garg <[hidden email]> wrote:
>
>> Hi Pavel,
>>
>> In the operation request, what does the 'flags' parameter mean?
>>
>>
>> Thanks,
>> -Prachi
>>
>> On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn <[hidden email]>
>> wrote:
>>
>>> Sergey, good point, done.
>>>
>>> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <[hidden email]>
>>> wrote:
>>>
>>> > Pavel
>>> >
>>> > Could you update the page by following:
>>> >
>>> > - String, date, UUID arrays allow to put NULL. Due to that every item
>>> in
>>> > the array written as type code byte (default item type or null type
>>> code) +
>>> > type data. It should be detailed explained (looks like that the table
>>> > should have an addtional column called for instance "nullable")
>>> >
>>> > - UUID type takes 16 bytes length
>>> >
>>> > thanks
>>> >
>>> >
>>> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <[hidden email]>
>>> > wrote:
>>> >
>>> > > Pavel
>>> > >
>>> > > Thanks for explanations!
>>> > >
>>> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <
>>> [hidden email]>
>>> > > wrote:
>>> > >
>>> > >> Sergey,
>>> > >>
>>> > >> 1. Code table size does not affect anything, as I understand, so
>>> there
>>> > is
>>> > >> no reason to introduce an extra byte.
>>> > >> 2. We have object arrays (code 23), I forgot to mention them, fixed.
>>> > >> 3. Also forgot, see code 25 in the updated document.
>>> > >>
>>> > >> Also note that operation codes have been updated (grouped by
>>> purpose) as
>>> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>>> > >>
>>> > >> Thanks,
>>> > >> Pavel
>>> > >>
>>> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <
>>> [hidden email]>
>>> > >> wrote:
>>> > >>
>>> > >> > Pavel
>>> > >> >
>>> > >> > Thanks for the document and your efforts for new protocol. It was
>>> > really
>>> > >> > helpful for playing around the python thin client design.
>>> > >> >
>>> > >> > Could you explain some things that were still not clear for binary
>>> > >> object
>>> > >> > format:
>>> > >> >
>>> > >> > 1. What a reason to introduce separated type codes for arrays? Why
>>> > just
>>> > >> we
>>> > >> > can't use the following?
>>> > >> > *<1 byte universal array code>*
>>> > >> > *<1 byte primitive code>*
>>> > >> > *<4 bytes length>*
>>> > >> > *<N bytes data>*
>>> > >> >
>>> > >> > We get 1 byte overhead but save 10 bytes in the code table. For
>>> arrays
>>> > >> the
>>> > >> > overhead is really insignificant:10 longs array takes now
>>> 1+4+4*10=45
>>> > >> bytes
>>> > >> > vs 1+1+4+4*10=46 bytes for the approach
>>> > >> > Moreover for that appoach a new primitive code will be available
>>> for
>>> > >> using
>>> > >> > for array immediately.
>>> > >> >
>>> > >> > 2. Why the arrays force to use a selected type? For python
>>> there's no
>>> > >> > limitations to use different types across one array (list). Would
>>> be
>>> > >> good
>>> > >> > to introduce a new type that will allow that. It could be look
>>> like
>>> > >> > following
>>> > >> > *<1 byte universal array code>*
>>> > >> > *<1 byte no common type code*> <-- this says that every item must
>>> > >> provide
>>> > >> > its date type code like it does regular primitive data
>>> > >> > *<4 bytes length>*
>>> > >> > *<1 byte item 0 type code>* <-- item provides its code
>>> > >> > *<N byte item 0 data>* <-- item provides its data
>>> > >> > *<1 byte item 1 type code>*
>>> > >> > *<N byte item 1 data>*
>>> > >> > etc
>>> > >> >
>>> > >> > Also that allow to put nested arrays without changes in type code
>>> > table!
>>> > >> > For instance if we want to store 9 longs and 1 boolean it will
>>> take
>>> > >> > now 1+1+4+(1+9)*4+(1+1)=48
>>> > >> > bytes (vs 45 bytes to store as 10 longs as usual).
>>> > >> >
>>> > >> > 3. Ther's only one way to store a dictionary (key-value)
>>> structure as
>>> > >> value
>>> > >> > in the cache via Complex Object. But it looks like
>>> overcomplicated. I
>>> > >> > suppose to introduce a code for that
>>> > >> > *<1 byte key-value dictionary code>*
>>> > >> >
>>> > >> > *<4 bytes length>*
>>> > >> > *<1 byte key 1 **name **type code>*
>>> > >> > *<N byte key 1 name data>*
>>> > >> > *<1 byte value 1 type code>*
>>> > >> > *<N byte value 1 value>*
>>> > >> > *<1 byte key 2 **name **type code>*
>>> > >> > *<N byte key 2 name data>*
>>> > >> > *<1 byte value 2 type code>*
>>> > >> > *<N byte value 2 value>*
>>> > >> > etc
>>> > >> >
>>> > >> > Also that allow to put nested dictionaries without changes in type
>>> > code
>>> > >> > table!
>>> > >> > Of course for the appoach above we get significat overhead for
>>> key
>>> > >> > storing. But I think it is acceptable for some cases and
>>> definitely ok
>>> > >> for
>>> > >> > Python
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]>
>>> > >> wrote:
>>> > >> >
>>> > >> > > Thanks Pavel! The document has good information. I'll create
>>> one on
>>> > >> > > readme.io; will also add some examples there.
>>> > >> > >
>>> > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
>>> > [hidden email]
>>> > >> >
>>> > >> > > wrote:
>>> > >> > >
>>> > >> > > > Igniters,
>>> > >> > > >
>>> > >> > > > I've put together a detailed description of our Thin Client
>>> > protocol
>>> > >> > > > in form of IEP on wiki:
>>> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>>> > >> > > > 9+Thin+Client+Protocol
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > To clarify:
>>> > >> > > > - Protocol implementation is in master (see
>>> ClientMessageParser
>>> > >> class)
>>> > >> > > > - Protocol has not been released yet, so we are free to change
>>> > >> anything
>>> > >> > > > - Protocol is only used by .NET Thin Client for now, but is
>>> > >> supposed to
>>> > >> > > be
>>> > >> > > > used from other languages by third party contributors
>>> > >> > > > - More operations will be added in future, this is a first
>>> set of
>>> > >> them,
>>> > >> > > > cache-related
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > Please review the document and let me know your thoughts.
>>> > >> > > > Is there anything missing or wrong?
>>> > >> > > >
>>> > >> > > > We should make sure that the foundation is solid and
>>> extensible.
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > Thanks,
>>> > >> > > > Pavel
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > --
>>> > >> > Sergey Kozlov
>>> > >> > GridGain Systems
>>> > >> > www.gridgain.com
>>> > >> >
>>> > >>
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Sergey Kozlov
>>> > > GridGain Systems
>>> > > www.gridgain.com
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Sergey Kozlov
>>> > GridGain Systems
>>> > www.gridgain.com
>>> >
>>>
>>
>>
>

Pavel Tupitsyn

Re: Thin Client Protocol documentation

Prachi, see "Response (failure)" table.
Failed response includes server protocol version and an error message.

On Thu, Nov 30, 2017 at 4:55 AM, Prachi Garg <[hidden email]> wrote:

> Pavel,
>
> If the connection handshake fails, what should be the message length in
> the response. When I try to fail the handshake, I get 32 as the message
> length. Is this expected?
>
> -Prachi
>
> On Wed, Nov 29, 2017 at 8:20 AM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
>> Hi Prachi,
>>
>> Flags parameter in all cache operations is reserved for things like
>> withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries,
>> withPartitionRecover.
>> See methods in IgniteCache interface.
>>
>> Thanks,
>> Pavel
>>
>> On Wed, Nov 29, 2017 at 7:10 PM, Prachi Garg <[hidden email]> wrote:
>>
>>> Hi Pavel,
>>>
>>> In the operation request, what does the 'flags' parameter mean?
>>>
>>>
>>> Thanks,
>>> -Prachi
>>>
>>> On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn <[hidden email]>
>>> wrote:
>>>
>>>> Sergey, good point, done.
>>>>
>>>> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <[hidden email]>
>>>> wrote:
>>>>
>>>> > Pavel
>>>> >
>>>> > Could you update the page by following:
>>>> >
>>>> > - String, date, UUID arrays allow to put NULL. Due to that every
>>>> item in
>>>> > the array written as type code byte (default item type or null type
>>>> code) +
>>>> > type data. It should be detailed explained (looks like that the table
>>>> > should have an addtional column called for instance "nullable")
>>>> >
>>>> > - UUID type takes 16 bytes length
>>>> >
>>>> > thanks
>>>> >
>>>> >
>>>> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <[hidden email]>
>>>> > wrote:
>>>> >
>>>> > > Pavel
>>>> > >
>>>> > > Thanks for explanations!
>>>> > >
>>>> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <
>>>> [hidden email]>
>>>> > > wrote:
>>>> > >
>>>> > >> Sergey,
>>>> > >>
>>>> > >> 1. Code table size does not affect anything, as I understand, so
>>>> there
>>>> > is
>>>> > >> no reason to introduce an extra byte.
>>>> > >> 2. We have object arrays (code 23), I forgot to mention them,
>>>> fixed.
>>>> > >> 3. Also forgot, see code 25 in the updated document.
>>>> > >>
>>>> > >> Also note that operation codes have been updated (grouped by
>>>> purpose) as
>>>> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>>>> > >>
>>>> > >> Thanks,
>>>> > >> Pavel
>>>> > >>
>>>> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <
>>>> [hidden email]>
>>>> > >> wrote:
>>>> > >>
>>>> > >> > Pavel
>>>> > >> >
>>>> > >> > Thanks for the document and your efforts for new protocol. It was
>>>> > really
>>>> > >> > helpful for playing around the python thin client design.
>>>> > >> >
>>>> > >> > Could you explain some things that were still not clear for
>>>> binary
>>>> > >> object
>>>> > >> > format:
>>>> > >> >
>>>> > >> > 1. What a reason to introduce separated type codes for arrays?
>>>> Why
>>>> > just
>>>> > >> we
>>>> > >> > can't use the following?
>>>> > >> > *<1 byte universal array code>*
>>>> > >> > *<1 byte primitive code>*
>>>> > >> > *<4 bytes length>*
>>>> > >> > *<N bytes data>*
>>>> > >> >
>>>> > >> > We get 1 byte overhead but save 10 bytes in the code table. For
>>>> arrays
>>>> > >> the
>>>> > >> > overhead is really insignificant:10 longs array takes now
>>>> 1+4+4*10=45
>>>> > >> bytes
>>>> > >> > vs 1+1+4+4*10=46 bytes for the approach
>>>> > >> > Moreover for that appoach a new primitive code will be available
>>>> for
>>>> > >> using
>>>> > >> > for array immediately.
>>>> > >> >
>>>> > >> > 2. Why the arrays force to use a selected type? For python
>>>> there's no
>>>> > >> > limitations to use different types across one array (list).
>>>> Would be
>>>> > >> good
>>>> > >> > to introduce a new type that will allow that. It could be look
>>>> like
>>>> > >> > following
>>>> > >> > *<1 byte universal array code>*
>>>> > >> > *<1 byte no common type code*> <-- this says that every item must
>>>> > >> provide
>>>> > >> > its date type code like it does regular primitive data
>>>> > >> > *<4 bytes length>*
>>>> > >> > *<1 byte item 0 type code>* <-- item provides its code
>>>> > >> > *<N byte item 0 data>* <-- item provides its data
>>>> > >> > *<1 byte item 1 type code>*
>>>> > >> > *<N byte item 1 data>*
>>>> > >> > etc
>>>> > >> >
>>>> > >> > Also that allow to put nested arrays without changes in type code
>>>> > table!
>>>> > >> > For instance if we want to store 9 longs and 1 boolean it will
>>>> take
>>>> > >> > now 1+1+4+(1+9)*4+(1+1)=48
>>>> > >> > bytes (vs 45 bytes to store as 10 longs as usual).
>>>> > >> >
>>>> > >> > 3. Ther's only one way to store a dictionary (key-value)
>>>> structure as
>>>> > >> value
>>>> > >> > in the cache via Complex Object. But it looks like
>>>> overcomplicated. I
>>>> > >> > suppose to introduce a code for that
>>>> > >> > *<1 byte key-value dictionary code>*
>>>> > >> >
>>>> > >> > *<4 bytes length>*
>>>> > >> > *<1 byte key 1 **name **type code>*
>>>> > >> > *<N byte key 1 name data>*
>>>> > >> > *<1 byte value 1 type code>*
>>>> > >> > *<N byte value 1 value>*
>>>> > >> > *<1 byte key 2 **name **type code>*
>>>> > >> > *<N byte key 2 name data>*
>>>> > >> > *<1 byte value 2 type code>*
>>>> > >> > *<N byte value 2 value>*
>>>> > >> > etc
>>>> > >> >
>>>> > >> > Also that allow to put nested dictionaries without changes in
>>>> type
>>>> > code
>>>> > >> > table!
>>>> > >> > Of course for the appoach above we get significat overhead for
>>>> key
>>>> > >> > storing. But I think it is acceptable for some cases and
>>>> definitely ok
>>>> > >> for
>>>> > >> > Python
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <[hidden email]
>>>> >
>>>> > >> wrote:
>>>> > >> >
>>>> > >> > > Thanks Pavel! The document has good information. I'll create
>>>> one on
>>>> > >> > > readme.io; will also add some examples there.
>>>> > >> > >
>>>> > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
>>>> > [hidden email]
>>>> > >> >
>>>> > >> > > wrote:
>>>> > >> > >
>>>> > >> > > > Igniters,
>>>> > >> > > >
>>>> > >> > > > I've put together a detailed description of our Thin Client
>>>> > protocol
>>>> > >> > > > in form of IEP on wiki:
>>>> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>>>> > >> > > > 9+Thin+Client+Protocol
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > > To clarify:
>>>> > >> > > > - Protocol implementation is in master (see
>>>> ClientMessageParser
>>>> > >> class)
>>>> > >> > > > - Protocol has not been released yet, so we are free to
>>>> change
>>>> > >> anything
>>>> > >> > > > - Protocol is only used by .NET Thin Client for now, but is
>>>> > >> supposed to
>>>> > >> > > be
>>>> > >> > > > used from other languages by third party contributors
>>>> > >> > > > - More operations will be added in future, this is a first
>>>> set of
>>>> > >> them,
>>>> > >> > > > cache-related
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > > Please review the document and let me know your thoughts.
>>>> > >> > > > Is there anything missing or wrong?
>>>> > >> > > >
>>>> > >> > > > We should make sure that the foundation is solid and
>>>> extensible.
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > > Thanks,
>>>> > >> > > > Pavel
>>>> > >> > > >
>>>> > >> > >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > --
>>>> > >> > Sergey Kozlov
>>>> > >> > GridGain Systems
>>>> > >> > www.gridgain.com
>>>> > >> >
>>>> > >>
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Sergey Kozlov
>>>> > > GridGain Systems
>>>> > > www.gridgain.com
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Sergey Kozlov
>>>> > GridGain Systems
>>>> > www.gridgain.com
>>>> >
>>>>
>>>
>>>
>>
>

Alexey Popov

RE: Thin Client Protocol documentation

In reply to this post by Pavel Tupitsyn

Hi Pavel,

Let me add my 5 cents.

1. It would be great if both Handshake request & response have some "magic" number (2 or 4 bytes) inside their msg body. That will simplify handling situations when non-Ignite client connects to Ignite server and vice versa.

2. It makes sense to add server version to successful Handshake response as well. It will help to understand & debug possible backward compatibility issues in the field by *.pcap logs analysis & etc.

3. Can we have a more strict header for all message types?
As far as I understand,
Handshake request has:
1) length - 4 byte
2) Handshake code - 1 byte
3) body - (length - 1) bytes

while OP_CACHE_GET request has:
1) length - 4 byte
2) OP_CACHE_GET code - 2 bytes
3) request id - 4 bytes
4) body - (length - 2 - 4) bytes

Why some messages have Operation code with 1 byte while others - 2 bytes? Why some requests/responses have request-id while others don't? Let's simplify parser work )

4. The same comments for success flag (1 byte) and status code (4 bytes) in responses. Let's leave only status code.

Thank you,
Alexey

From: Pavel Tupitsyn
Sent: Wednesday, November 22, 2017 4:04 PM
To: [hidden email]
Subject: Thin Client Protocol documentation

Igniters,

I've put together a detailed description of our Thin Client protocol
in form of IEP on wiki:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol

To clarify:
- Protocol implementation is in master (see ClientMessageParser class)
- Protocol has not been released yet, so we are free to change anything
- Protocol is only used by .NET Thin Client for now, but is supposed to be
used from other languages by third party contributors
- More operations will be added in future, this is a first set of them,
cache-related

Please review the document and let me know your thoughts.
Is there anything missing or wrong?

We should make sure that the foundation is solid and extensible.

Thanks,
Pavel

Pavel Tupitsyn

Re: Thin Client Protocol documentation

Hi Alexey,

1,2,3 are related only to handshake. All other operations are consistent.

Handshake request format is dictated by existing client connector that is
shared with ODBC and JDBC clients (see
ClientListenerNioListener.onHandshake).
so we can't add magic numbers or change operation code.

But yes, we can add server version to the handshake response, and I think
this makes sense.

> 4. The same comments for success flag (1 byte) and status code (4 bytes)
in responses. Let's leave only status code.
We don't have a success flag in responses, there is just a 4-byte status
code, 0 indicates success, everything else is an error.

Thanks,
Pavel

On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <[hidden email]> wrote:

> Hi Pavel,
>
> Let me add my 5 cents.
>
> 1. It would be great if both Handshake request & response have some
> "magic" number (2 or 4 bytes) inside their msg body. That will simplify
> handling situations when non-Ignite client connects to Ignite server and
> vice versa.
>
> 2. It makes sense to add server version to successful Handshake response
> as well. It will help to understand & debug possible backward compatibility
> issues in the field by *.pcap logs analysis & etc.
>
> 3. Can we have a more strict header for all message types?
> As far as I understand,
> Handshake request has:
> 1) length - 4 byte
> 2) Handshake code - 1 byte
> 3) body - (length - 1) bytes
>
> while OP_CACHE_GET request has:
> 1) length - 4 byte
> 2) OP_CACHE_GET code - 2 bytes
> 3) request id - 4 bytes
> 4) body - (length - 2 - 4) bytes
>
> Why some messages have Operation code with 1 byte while others - 2 bytes?
> Why some requests/responses have request-id while others don't? Let's
> simplify parser work )
>
> 4. The same comments for success flag (1 byte) and status code (4 bytes)
> in responses. Let's leave only status code.
>
> Thank you,
> Alexey
>
> From: Pavel Tupitsyn
> Sent: Wednesday, November 22, 2017 4:04 PM
> To: [hidden email]
> Subject: Thin Client Protocol documentation
>
> Igniters,
>
> I've put together a detailed description of our Thin Client protocol
> in form of IEP on wiki:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 9+Thin+Client+Protocol
>
>
> To clarify:
> - Protocol implementation is in master (see ClientMessageParser class)
> - Protocol has not been released yet, so we are free to change anything
> - Protocol is only used by .NET Thin Client for now, but is supposed to be
> used from other languages by third party contributors
> - More operations will be added in future, this is a first set of them,
> cache-related
>
>
> Please review the document and let me know your thoughts.
> Is there anything missing or wrong?
>
> We should make sure that the foundation is solid and extensible.
>
>
> Thanks,
> Pavel
>
>

Alexey Popov

RE: Thin Client Protocol documentation

Pavel,

I believe ClientListenerNioListener.onHandshake() could support more than one Handshake request format (be backward compatible), for instance, if we will have a new handshake code = 0xABCD that differs from 0x01 byte.

It is a design vs architecture question.

I can’t understand why the legacy Handshake format should be used for a new protocol. If this protocol is supposed to be public it should have no exceptions for message formats.

Thank you,
Alexey

From: Pavel Tupitsyn
Sent: Thursday, November 30, 2017 12:11 PM
To: [hidden email]
Subject: Re: Thin Client Protocol documentation

Hi Alexey,

1,2,3 are related only to handshake. All other operations are consistent.

Handshake request format is dictated by existing client connector that is
shared with ODBC and JDBC clients (see
ClientListenerNioListener.onHandshake).
so we can't add magic numbers or change operation code.

But yes, we can add server version to the handshake response, and I think
this makes sense.

> 4. The same comments for success flag (1 byte) and status code (4 bytes)
in responses. Let's leave only status code.
We don't have a success flag in responses, there is just a 4-byte status
code, 0 indicates success, everything else is an error.

Thanks,
Pavel

On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <[hidden email]> wrote:

Pavel Tupitsyn

Re: Thin Client Protocol documentation

There are no legacy formats. JDBC and ODBC clients are not "legacy", quite
the opposite.

In future we may even want to combine JDBC Thin and general-purpose Thin
clients since they have a lot in common.
So let's keep the handshake format consistent across clients.

> exceptions for message formats
Handshake is an exception anyway, it does not have (or need) requestId, etc.

On Fri, Dec 1, 2017 at 1:38 PM, Alexey Popov <[hidden email]> wrote:

> Pavel,
>
> I believe ClientListenerNioListener.onHandshake() could support more than
> one Handshake request format (be backward compatible), for instance, if we
> will have a new handshake code = 0xABCD that differs from 0x01 byte.
>
> It is a design vs architecture question.
>
> I can’t understand why the legacy Handshake format should be used for a
> new protocol. If this protocol is supposed to be public it should have no
> exceptions for message formats.
>
> Thank you,
> Alexey
>
> From: Pavel Tupitsyn
> Sent: Thursday, November 30, 2017 12:11 PM
> To: [hidden email]
> Subject: Re: Thin Client Protocol documentation
>
> Hi Alexey,
>
> 1,2,3 are related only to handshake. All other operations are consistent.
>
> Handshake request format is dictated by existing client connector that is
> shared with ODBC and JDBC clients (see
> ClientListenerNioListener.onHandshake).
> so we can't add magic numbers or change operation code.
>
> But yes, we can add server version to the handshake response, and I think
> this makes sense.
>
> > 4. The same comments for success flag (1 byte) and status code (4 bytes)
> in responses. Let's leave only status code.
> We don't have a success flag in responses, there is just a 4-byte status
> code, 0 indicates success, everything else is an error.
>
> Thanks,
> Pavel
>
> On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <[hidden email]>
> wrote:
>
> > Hi Pavel,
> >
> > Let me add my 5 cents.
> >
> > 1. It would be great if both Handshake request & response have some
> > "magic" number (2 or 4 bytes) inside their msg body. That will simplify
> > handling situations when non-Ignite client connects to Ignite server and
> > vice versa.
> >
> > 2. It makes sense to add server version to successful Handshake response
> > as well. It will help to understand & debug possible backward
> compatibility
> > issues in the field by *.pcap logs analysis & etc.
> >
> > 3. Can we have a more strict header for all message types?
> > As far as I understand,
> > Handshake request has:
> > 1) length - 4 byte
> > 2) Handshake code - 1 byte
> > 3) body - (length - 1) bytes
> >
> > while OP_CACHE_GET request has:
> > 1) length - 4 byte
> > 2) OP_CACHE_GET code - 2 bytes
> > 3) request id - 4 bytes
> > 4) body - (length - 2 - 4) bytes
> >
> > Why some messages have Operation code with 1 byte while others - 2 bytes?
> > Why some requests/responses have request-id while others don't? Let's
> > simplify parser work )
> >
> > 4. The same comments for success flag (1 byte) and status code (4 bytes)
> > in responses. Let's leave only status code.
> >
> > Thank you,
> > Alexey
> >
> > From: Pavel Tupitsyn
> > Sent: Wednesday, November 22, 2017 4:04 PM
> > To: [hidden email]
> > Subject: Thin Client Protocol documentation
> >
> > Igniters,
> >
> > I've put together a detailed description of our Thin Client protocol
> > in form of IEP on wiki:
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 9+Thin+Client+Protocol
> >
> >
> > To clarify:
> > - Protocol implementation is in master (see ClientMessageParser class)
> > - Protocol has not been released yet, so we are free to change anything
> > - Protocol is only used by .NET Thin Client for now, but is supposed to
> be
> > used from other languages by third party contributors
> > - More operations will be added in future, this is a first set of them,
> > cache-related
> >
> >
> > Please review the document and let me know your thoughts.
> > Is there anything missing or wrong?
> >
> > We should make sure that the foundation is solid and extensible.
> >
> >
> > Thanks,
> > Pavel
> >
> >
>
>

Andrey Kornev

Re: Thin Client Protocol documentation

Pavel,

Great job on the Thin Client protocol design!

I'm a bit late to the party, but I'm wondering why none of the compute features are supported? Such omission is unfortunate. Is it intentional? If so, what are the reasons?

I think it would be useful for the clients to be able to invoke already deployed tasks/services, or send a closure (if the client happens to be written in Java) for execution.

Thanks
Andrey

________________________________
From: Pavel Tupitsyn <[hidden email]>
Sent: Friday, December 1, 2017 5:48 AM
To: [hidden email]
Subject: Re: Thin Client Protocol documentation

There are no legacy formats. JDBC and ODBC clients are not "legacy", quite
the opposite.

In future we may even want to combine JDBC Thin and general-purpose Thin
clients since they have a lot in common.
So let's keep the handshake format consistent across clients.

> exceptions for message formats
Handshake is an exception anyway, it does not have (or need) requestId, etc.

On Fri, Dec 1, 2017 at 1:38 PM, Alexey Popov <[hidden email]> wrote:

Pavel Tupitsyn

Re: Thin Client Protocol documentation

Hi Andrey,

Compute and other APIs are certainly planned, cache is just a start.
We intentionally limit the scope to actually release something in 2.4 and
not delay it further.

Adding operations to existing protocol is relatively easy.
Current focus is to make sure that the protocol itself is solid and
future-proof.

Thanks,
Pavel

Prachi Garg

Re: Thin Client Protocol documentation

Pavel,

How do you specify the flags (withSkipStore, withExpiryPolicy, etc..) in
bytes? Do you have a code for each of them?

On Sun, Dec 3, 2017 at 9:23 AM, Pavel Tupitsyn <[hidden email]> wrote:

> Hi Andrey,
>
> Compute and other APIs are certainly planned, cache is just a start.
> We intentionally limit the scope to actually release something in 2.4 and
> not delay it further.
>
> Adding operations to existing protocol is relatively easy.
> Current focus is to make sure that the protocol itself is solid and
> future-proof.
>
> Thanks,
> Pavel
>

Pavel Tupitsyn

Re: Thin Client Protocol documentation

Prachi,

Added "Cache flags" note to the doc.

Thanks,
Pavel

On Mon, Dec 4, 2017 at 7:46 PM, Prachi Garg <[hidden email]> wrote:

> Pavel,
>
> How do you specify the flags (withSkipStore, withExpiryPolicy, etc..) in
> bytes? Do you have a code for each of them?
>
> On Sun, Dec 3, 2017 at 9:23 AM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > Hi Andrey,
> >
> > Compute and other APIs are certainly planned, cache is just a start.
> > We intentionally limit the scope to actually release something in 2.4 and
> > not delay it further.
> >
> > Adding operations to existing protocol is relatively easy.
> > Current focus is to make sure that the protocol itself is solid and
> > future-proof.
> >
> > Thanks,
> > Pavel
> >
>

Andrey Kornev

Re: Thin Client Protocol documentation

In reply to this post by Pavel Tupitsyn

Pavel,

Thanks! While we're at it, are there any plans to add cluster-related operations? For example, I think it'd be nice to allow the thin clients to obtain a current topology snapshot. This would make it possible the clients to send requests directly to the affinity host for colocated computation. To make it even more useful, all server responses could optionally include the topology version the operation has been executed against. This would effectively give us a kind out-of-band topology change notification mechanism. This way the clients can detect a topology change and refresh the topology snapshot next time they need to compute affinity.

Regards
Andrey
________________________________
From: Pavel Tupitsyn <[hidden email]>
Sent: Sunday, December 3, 2017 9:23 AM
To: [hidden email]
Subject: Re: Thin Client Protocol documentation

Hi Andrey,

Compute and other APIs are certainly planned, cache is just a start.
We intentionally limit the scope to actually release something in 2.4 and
not delay it further.

Adding operations to existing protocol is relatively easy.
Current focus is to make sure that the protocol itself is solid and
future-proof.

Thanks,
Pavel