Thin clients: WithExpiryPolicy

[Aleksandr Shapkin]

Igniters,

I would like to add WithExpirePolicy support to thin clients. [1]
For a thick client, we can obtain a reference to a cache wrapper instance and use cache API through it. At the same time, the thin client protocol is stateless, we do not hold a reference to a cache but rather a cache name identifier is used for a server to create an appropriate cache instance.

We could extend the protocol as we did with WithKeepBinaryMethod: 
every time we need to call some API on a cache with expiration, a serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This approach works well, but things could get worse if we decided to add a few more WithSomething* methods.

Initially, I was thinking about introducing some state context to a protocol, similar to a QueryCursor API. For instance, we can save an expire policy configuration for the first call and use some hash value based on an ExpiryPolicy for further calls, just as we do for cache names. I.e. newCacheId = [cacheId, new AdditionalValues(expiryPolicyId, binaryModeId, ....)] But this approach complicates logic and leads to additional memory consumption.

I think it's ok for now to use the first approach with ExpiryPolicy serialization.
But any ideas are welcome.

[1] - https://issues.apache.org/jira/browse/IGNITE-9033

[Pavel Tupitsyn]

Stateless approach looks a lot better to me.

We have a choice:
* Keep expiry policy on server and send an ID with every request (like a
query cursor ID - 8 bytes)
* Send full expiry policy with every expiry-enabled request (24 bytes - or
maybe less? We should think about the format)

Stateful approach will bring a lot of complexity if we consider Affinity
Awareness [1] mechanism (and also automatic reconnect).
We would have to keep ExpiryPolicyId for every server and choose the right
one based on the affinity for every operation.
This can easily negate any performance gain from saving 16 bytes.

And there is always CacheConfiguration.ExpiryPolicyFactory, which allows us
to set up default expiry policy.

> things could get worse if we decided to add a few more WithSomething*
methods
I don't think this is a good argument - we should decide on case by case
basis.
Anyway, other With* methods don't have any parameters, so they carry only 1
bit of information.

https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients


On Fri, Oct 18, 2019 at 5:14 PM Alexandr Shapkin <[hidden email]> wrote:

> Igniters,
>
> I would like to add WithExpirePolicy support to thin clients. [1]
> For a thick client, we can obtain a reference to a cache wrapper instance
> and use cache API through it. At the same time, the thin client protocol is
> stateless, we do not hold a reference to a cache but rather a cache name
> identifier is used for a server to create an appropriate cache instance.
>
> We could extend the protocol as we did with WithKeepBinaryMethod:
> every time we need to call some API on a cache with expiration, a
> serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This approach
> works well, but things could get worse if we decided to add a few more
> WithSomething* methods.
>
> Initially, I was thinking about introducing some state context to a
> protocol, similar to a QueryCursor API. For instance, we can save an expire
> policy configuration for the first call and use some hash value based on an
> ExpiryPolicy for further calls, just as we do for cache names. I.e.
> newCacheId = [cacheId, new AdditionalValues(expiryPolicyId, binaryModeId,
> ....)] But this approach complicates logic and leads to additional memory
> consumption.
>
> I think it's ok for now to use the first approach with ExpiryPolicy
> serialization.
> But any ideas are welcome.
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-9033
>
>

[Aleksandr Shapkin]

Pavel,
 
Thanks for sharing your thoughts.
 
Right now I see that 2 reserved bytes come with every cache request: cacheId and flags.
The first one is obvious, but the second one is used only for java thin client with enabled KeepBinary mode.
 
I think we could use the flags byte and write an expiration policy flag when required.
Whenever the server sees that there is a request with expiration flag, we deserialize a policy and apply it to the request.

From: Pavel Tupitsyn
Sent: Friday, October 18, 2019 7:05 PM
To: dev
Subject: Re: Thin clients: WithExpiryPolicy

Stateless approach looks a lot better to me.

We have a choice:
* Keep expiry policy on server and send an ID with every request (like a
query cursor ID - 8 bytes)
* Send full expiry policy with every expiry-enabled request (24 bytes - or
maybe less? We should think about the format)

Stateful approach will bring a lot of complexity if we consider Affinity
Awareness [1] mechanism (and also automatic reconnect).
We would have to keep ExpiryPolicyId for every server and choose the right
one based on the affinity for every operation.
This can easily negate any performance gain from saving 16 bytes.

And there is always CacheConfiguration.ExpiryPolicyFactory, which allows us
to set up default expiry policy.

> things could get worse if we decided to add a few more WithSomething*
methods
I don't think this is a good argument - we should decide on case by case
basis.
Anyway, other With* methods don't have any parameters, so they carry only 1
bit of information.

https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients


On Fri, Oct 18, 2019 at 5:14 PM Alexandr Shapkin <[hidden email]> wrote:

> Igniters,
>
> I would like to add WithExpirePolicy support to thin clients. [1]
> For a thick client, we can obtain a reference to a cache wrapper instance
> and use cache API through it. At the same time, the thin client protocol is
> stateless, we do not hold a reference to a cache but rather a cache name
> identifier is used for a server to create an appropriate cache instance.
>
> We could extend the protocol as we did with WithKeepBinaryMethod:
> every time we need to call some API on a cache with expiration, a
> serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This approach
> works well, but things could get worse if we decided to add a few more
> WithSomething* methods.
>
> Initially, I was thinking about introducing some state context to a
> protocol, similar to a QueryCursor API. For instance, we can save an expire
> policy configuration for the first call and use some hash value based on an
> ExpiryPolicy for further calls, just as we do for cache names. I.e.
> newCacheId = [cacheId, new AdditionalValues(expiryPolicyId, binaryModeId,
> ....)] But this approach complicates logic and leads to additional memory
> consumption.
>
> I think it's ok for now to use the first approach with ExpiryPolicy
> serialization.
> But any ideas are welcome.
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-9033
>
>

[Pavel Tupitsyn]

Alexandr,

Sounds good to me.

Remaining question is - how do we serialize the expiry policy?
Thick client passes this to JNI as 3 int64 values (duration in
milliseconds).
But I don't think we need millisecond precision for expiration, so we could
pass seconds as float values (3*4 bytes).

Thoughts?

On Fri, Oct 18, 2019 at 8:21 PM Alexandr Shapkin <[hidden email]> wrote:

> Pavel,
>
> Thanks for sharing your thoughts.
>
> Right now I see that 2 reserved bytes come with every cache request:
> cacheId and flags.
> The first one is obvious, but the second one is used only for java thin
> client with enabled KeepBinary mode.
>
> I think we could use the flags byte and write an expiration policy flag
> when required.
> Whenever the server sees that there is a request with expiration flag, we
> deserialize a policy and apply it to the request.
>
> From: Pavel Tupitsyn
> Sent: Friday, October 18, 2019 7:05 PM
> To: dev
> Subject: Re: Thin clients: WithExpiryPolicy
>
> Stateless approach looks a lot better to me.
>
> We have a choice:
> * Keep expiry policy on server and send an ID with every request (like a
> query cursor ID - 8 bytes)
> * Send full expiry policy with every expiry-enabled request (24 bytes - or
> maybe less? We should think about the format)
>
> Stateful approach will bring a lot of complexity if we consider Affinity
> Awareness [1] mechanism (and also automatic reconnect).
> We would have to keep ExpiryPolicyId for every server and choose the right
> one based on the affinity for every operation.
> This can easily negate any performance gain from saving 16 bytes.
>
> And there is always CacheConfiguration.ExpiryPolicyFactory, which allows us
> to set up default expiry policy.
>
> > things could get worse if we decided to add a few more WithSomething*
> methods
> I don't think this is a good argument - we should decide on case by case
> basis.
> Anyway, other With* methods don't have any parameters, so they carry only 1
> bit of information.
>
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
>
>
> On Fri, Oct 18, 2019 at 5:14 PM Alexandr Shapkin <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > I would like to add WithExpirePolicy support to thin clients. [1]
> > For a thick client, we can obtain a reference to a cache wrapper instance
> > and use cache API through it. At the same time, the thin client protocol
> is
> > stateless, we do not hold a reference to a cache but rather a cache name
> > identifier is used for a server to create an appropriate cache instance.
> >
> > We could extend the protocol as we did with WithKeepBinaryMethod:
> > every time we need to call some API on a cache with expiration, a
> > serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This
> approach
> > works well, but things could get worse if we decided to add a few more
> > WithSomething* methods.
> >
> > Initially, I was thinking about introducing some state context to a
> > protocol, similar to a QueryCursor API. For instance, we can save an
> expire
> > policy configuration for the first call and use some hash value based on
> an
> > ExpiryPolicy for further calls, just as we do for cache names. I.e.
> > newCacheId = [cacheId, new AdditionalValues(expiryPolicyId, binaryModeId,
> > ....)] But this approach complicates logic and leads to additional memory
> > consumption.
> >
> > I think it's ok for now to use the first approach with ExpiryPolicy
> > serialization.
> > But any ideas are welcome.
> >
> > [1] - https://issues.apache.org/jira/browse/IGNITE-9033
> >
> >
>
>

[Aleksandr Shapkin]

Pavel,

I think that in some cases milliseconds could make sense,
so I would like to keep the current precision.

With this approach we will avoid unnecessary code changes as well as
keeping API in consistence with the thick client method and cacheConfiguration settings.


From: Pavel Tupitsyn
Sent: Saturday, October 19, 2019 9:20 PM
To: dev
Subject: Re: Thin clients: WithExpiryPolicy

Alexandr,

Sounds good to me.

Remaining question is - how do we serialize the expiry policy?
Thick client passes this to JNI as 3 int64 values (duration in
milliseconds).
But I don't think we need millisecond precision for expiration, so we could
pass seconds as float values (3*4 bytes).

Thoughts?

On Fri, Oct 18, 2019 at 8:21 PM Alexandr Shapkin <[hidden email]> wrote:

> Pavel,
>
> Thanks for sharing your thoughts.
>
> Right now I see that 2 reserved bytes come with every cache request:
> cacheId and flags.
> The first one is obvious, but the second one is used only for java thin
> client with enabled KeepBinary mode.
>
> I think we could use the flags byte and write an expiration policy flag
> when required.
> Whenever the server sees that there is a request with expiration flag, we
> deserialize a policy and apply it to the request.
>
> From: Pavel Tupitsyn
> Sent: Friday, October 18, 2019 7:05 PM
> To: dev
> Subject: Re: Thin clients: WithExpiryPolicy
>
> Stateless approach looks a lot better to me.
>
> We have a choice:
> * Keep expiry policy on server and send an ID with every request (like a
> query cursor ID - 8 bytes)
> * Send full expiry policy with every expiry-enabled request (24 bytes - or
> maybe less? We should think about the format)
>
> Stateful approach will bring a lot of complexity if we consider Affinity
> Awareness [1] mechanism (and also automatic reconnect).
> We would have to keep ExpiryPolicyId for every server and choose the right
> one based on the affinity for every operation.
> This can easily negate any performance gain from saving 16 bytes.
>
> And there is always CacheConfiguration.ExpiryPolicyFactory, which allows us
> to set up default expiry policy.
>
> > things could get worse if we decided to add a few more WithSomething*
> methods
> I don't think this is a good argument - we should decide on case by case
> basis.
> Anyway, other With* methods don't have any parameters, so they carry only 1
> bit of information.
>
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
>
>
> On Fri, Oct 18, 2019 at 5:14 PM Alexandr Shapkin <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > I would like to add WithExpirePolicy support to thin clients. [1]
> > For a thick client, we can obtain a reference to a cache wrapper instance
> > and use cache API through it. At the same time, the thin client protocol
> is
> > stateless, we do not hold a reference to a cache but rather a cache name
> > identifier is used for a server to create an appropriate cache instance.
> >
> > We could extend the protocol as we did with WithKeepBinaryMethod:
> > every time we need to call some API on a cache with expiration, a
> > serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This
> approach
> > works well, but things could get worse if we decided to add a few more
> > WithSomething* methods.
> >
> > Initially, I was thinking about introducing some state context to a
> > protocol, similar to a QueryCursor API. For instance, we can save an
> expire
> > policy configuration for the first call and use some hash value based on
> an
> > ExpiryPolicy for further calls, just as we do for cache names. I.e.
> > newCacheId = [cacheId, new AdditionalValues(expiryPolicyId, binaryModeId,
> > ....)] But this approach complicates logic and leads to additional memory
> > consumption.
> >
> > I think it's ok for now to use the first approach with ExpiryPolicy
> > serialization.
> > But any ideas are welcome.
> >
> > [1] - https://issues.apache.org/jira/browse/IGNITE-9033
> >
> >
>
>

[Pavel Tupitsyn]

Ok, no objections

On Wed, Oct 23, 2019 at 4:36 PM Alexandr Shapkin <[hidden email]> wrote:

> Pavel,
>
> I think that in some cases milliseconds could make sense,
> so I would like to keep the current precision.
>
> With this approach we will avoid unnecessary code changes as well as
> keeping API in consistence with the thick client method and
> cacheConfiguration settings.
>
>
> From: Pavel Tupitsyn
> Sent: Saturday, October 19, 2019 9:20 PM
> To: dev
> Subject: Re: Thin clients: WithExpiryPolicy
>
> Alexandr,
>
> Sounds good to me.
>
> Remaining question is - how do we serialize the expiry policy?
> Thick client passes this to JNI as 3 int64 values (duration in
> milliseconds).
> But I don't think we need millisecond precision for expiration, so we could
> pass seconds as float values (3*4 bytes).
>
> Thoughts?
>
> On Fri, Oct 18, 2019 at 8:21 PM Alexandr Shapkin <[hidden email]>
> wrote:
>
> > Pavel,
> >
> > Thanks for sharing your thoughts.
> >
> > Right now I see that 2 reserved bytes come with every cache request:
> > cacheId and flags.
> > The first one is obvious, but the second one is used only for java thin
> > client with enabled KeepBinary mode.
> >
> > I think we could use the flags byte and write an expiration policy flag
> > when required.
> > Whenever the server sees that there is a request with expiration flag, we
> > deserialize a policy and apply it to the request.
> >
> > From: Pavel Tupitsyn
> > Sent: Friday, October 18, 2019 7:05 PM
> > To: dev
> > Subject: Re: Thin clients: WithExpiryPolicy
> >
> > Stateless approach looks a lot better to me.
> >
> > We have a choice:
> > * Keep expiry policy on server and send an ID with every request (like a
> > query cursor ID - 8 bytes)
> > * Send full expiry policy with every expiry-enabled request (24 bytes -
> or
> > maybe less? We should think about the format)
> >
> > Stateful approach will bring a lot of complexity if we consider Affinity
> > Awareness [1] mechanism (and also automatic reconnect).
> > We would have to keep ExpiryPolicyId for every server and choose the
> right
> > one based on the affinity for every operation.
> > This can easily negate any performance gain from saving 16 bytes.
> >
> > And there is always CacheConfiguration.ExpiryPolicyFactory, which allows
> us
> > to set up default expiry policy.
> >
> > > things could get worse if we decided to add a few more WithSomething*
> > methods
> > I don't think this is a good argument - we should decide on case by case
> > basis.
> > Anyway, other With* methods don't have any parameters, so they carry
> only 1
> > bit of information.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> >
> >
> > On Fri, Oct 18, 2019 at 5:14 PM Alexandr Shapkin <[hidden email]>
> > wrote:
> >
> > > Igniters,
> > >
> > > I would like to add WithExpirePolicy support to thin clients. [1]
> > > For a thick client, we can obtain a reference to a cache wrapper
> instance
> > > and use cache API through it. At the same time, the thin client
> protocol
> > is
> > > stateless, we do not hold a reference to a cache but rather a cache
> name
> > > identifier is used for a server to create an appropriate cache
> instance.
> > >
> > > We could extend the protocol as we did with WithKeepBinaryMethod:
> > > every time we need to call some API on a cache with expiration, a
> > > serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This
> > approach
> > > works well, but things could get worse if we decided to add a few more
> > > WithSomething* methods.
> > >
> > > Initially, I was thinking about introducing some state context to a
> > > protocol, similar to a QueryCursor API. For instance, we can save an
> > expire
> > > policy configuration for the first call and use some hash value based
> on
> > an
> > > ExpiryPolicy for further calls, just as we do for cache names. I.e.
> > > newCacheId = [cacheId, new AdditionalValues(expiryPolicyId,
> binaryModeId,
> > > ....)] But this approach complicates logic and leads to additional
> memory
> > > consumption.
> > >
> > > I think it's ok for now to use the first approach with ExpiryPolicy
> > > serialization.
> > > But any ideas are welcome.
> > >
> > > [1] - https://issues.apache.org/jira/browse/IGNITE-9033
> > >
> > >
> >
> >
>
>