IgfsPerBlockLruEvictionPolicy does not work as expected any more, what shall we do with it?

[Ivan V.]

Hi, colleagues,

as Ignite caches moved to paged offheap memory , the
IgfsPerBlockLruEvictionPolicy does not seem to work as expected any more,
because
1) interface org.apache.ignite.cache.eviction.EvictionPolicy now only
defines "eviction from on-heap", not a real eviction, because each on-heap
cache is now accompanied with underlying off-heap cache. It can become
"real eviction" only for "on-heap-only" mode caches, once they get
 implemented.
2) for off-heap eviction an entire page is evicted, not a specific k-v
pair, and LRU policy is not exactly LRU any more (see
org.apache.ignite.configuration.DataPageEvictionMode#RANDOM_LRU). So, it
appears to be impossible to re-implement this policy for the off-heap layer.

Thus, now IgfsPerBlockLruEvictionPolicy is not quite valid, and some of
corresponding tests fail
(org.apache.ignite.internal.processors.igfs.IgfsCachePerBlockLruEvictionPolicySelfTest#testDataSizeEviction,
 org.apache.ignite.internal.processors.igfs.IgfsCachePerBlockLruEvictionPolicySelfTest#testBlockCountEviction)

So, the options I see are:
1) deprecate/remove IgfsPerBlockLruEvictionPolicy ;
2) leave it as is, but explain in javadocs that it only works for on-heap
layer, that does not in fact evict  blocks from the underlying offheap
layer.

Please share your opinions.

[Ivan Rakov]

As for me, there's no reason to store IGFS blocks in cache with on-heap
layer enabled.
I vote for the first option: deprecate/remove
IgfsPerBlockLruEvictionPolicy (along with tests), recommend users to use
page-based eviction if they need to evict blocks from primary IGFS.

--
Best Regards,
Ivan Rakov

On 24.05.2017 19:57, Ivan V. wrote:

> Hi, colleagues,
>
> as Ignite caches moved to paged offheap memory , the
> IgfsPerBlockLruEvictionPolicy does not seem to work as expected any more,
> because
> 1) interface org.apache.ignite.cache.eviction.EvictionPolicy now only
> defines "eviction from on-heap", not a real eviction, because each on-heap
> cache is now accompanied with underlying off-heap cache. It can become
> "real eviction" only for "on-heap-only" mode caches, once they get
>   implemented.
> 2) for off-heap eviction an entire page is evicted, not a specific k-v
> pair, and LRU policy is not exactly LRU any more (see
> org.apache.ignite.configuration.DataPageEvictionMode#RANDOM_LRU). So, it
> appears to be impossible to re-implement this policy for the off-heap layer.
>
> Thus, now IgfsPerBlockLruEvictionPolicy is not quite valid, and some of
> corresponding tests fail
> (org.apache.ignite.internal.processors.igfs.IgfsCachePerBlockLruEvictionPolicySelfTest#testDataSizeEviction,
>   org.apache.ignite.internal.processors.igfs.IgfsCachePerBlockLruEvictionPolicySelfTest#testBlockCountEviction)
>
> So, the options I see are:
> 1) deprecate/remove IgfsPerBlockLruEvictionPolicy ;
> 2) leave it as is, but explain in javadocs that it only works for on-heap
> layer, that does not in fact evict  blocks from the underlying offheap
> layer.
>
> Please share your opinions.
>

[dmagda]

Hi Ivan,

I’m for this approach

> 2) leave it as is, but explain in javadocs that it only works for on-heap
> layer, that does not in fact evict  blocks from the underlying offheap
> layer.

because it should be feasible to enable on-heap caching for IGFS, right? Using the memory policies. So, I would reimplement the tests with the on-heap caching enabling and checking up that data is pushed out of the heap.

—
Denis

> On May 24, 2017, at 9:57 AM, Ivan V. <[hidden email]> wrote:
>
> Hi, colleagues,
>
> as Ignite caches moved to paged offheap memory , the
> IgfsPerBlockLruEvictionPolicy does not seem to work as expected any more,
> because
> 1) interface org.apache.ignite.cache.eviction.EvictionPolicy now only
> defines "eviction from on-heap", not a real eviction, because each on-heap
> cache is now accompanied with underlying off-heap cache. It can become
> "real eviction" only for "on-heap-only" mode caches, once they get
> implemented.
> 2) for off-heap eviction an entire page is evicted, not a specific k-v
> pair, and LRU policy is not exactly LRU any more (see
> org.apache.ignite.configuration.DataPageEvictionMode#RANDOM_LRU). So, it
> appears to be impossible to re-implement this policy for the off-heap layer.
>
> Thus, now IgfsPerBlockLruEvictionPolicy is not quite valid, and some of
> corresponding tests fail
> (org.apache.ignite.internal.processors.igfs.IgfsCachePerBlockLruEvictionPolicySelfTest#testDataSizeEviction,
> org.apache.ignite.internal.processors.igfs.IgfsCachePerBlockLruEvictionPolicySelfTest#testBlockCountEviction)
>
> So, the options I see are:
> 1) deprecate/remove IgfsPerBlockLruEvictionPolicy ;
> 2) leave it as is, but explain in javadocs that it only works for on-heap
> layer, that does not in fact evict  blocks from the underlying offheap
> layer.
>
> Please share your opinions.

[Alexey Goncharuk]

Guys, I think it makes little or no sense to keep blocks on-heap. If I
understand correctly, the eviction policy was used in combined modes where
partial data eviction is allowed. To make this work in the new PageMemory
architecture, we only need to make sure that IGFS block size equals to the
data page free space size. In this case, our standard offheap eviction
policy will be semantically equal to the old per-block eviction policy. No
need to go on-heap at all.

Thoughts?

2017-05-25 4:21 GMT+03:00 Denis Magda <[hidden email]>:

> Hi Ivan,
>
> I’m for this approach
>
> > 2) leave it as is, but explain in javadocs that it only works for on-heap
> > layer, that does not in fact evict  blocks from the underlying offheap
> > layer.
>
> because it should be feasible to enable on-heap caching for IGFS, right?
> Using the memory policies. So, I would reimplement the tests with the
> on-heap caching enabling and checking up that data is pushed out of the
> heap.
>
> —
> Denis
>
> > On May 24, 2017, at 9:57 AM, Ivan V. <[hidden email]> wrote:
> >
> > Hi, colleagues,
> >
> > as Ignite caches moved to paged offheap memory , the
> > IgfsPerBlockLruEvictionPolicy does not seem to work as expected any more,
> > because
> > 1) interface org.apache.ignite.cache.eviction.EvictionPolicy now only
> > defines "eviction from on-heap", not a real eviction, because each
> on-heap
> > cache is now accompanied with underlying off-heap cache. It can become
> > "real eviction" only for "on-heap-only" mode caches, once they get
> > implemented.
> > 2) for off-heap eviction an entire page is evicted, not a specific k-v
> > pair, and LRU policy is not exactly LRU any more (see
> > org.apache.ignite.configuration.DataPageEvictionMode#RANDOM_LRU). So, it
> > appears to be impossible to re-implement this policy for the off-heap
> layer.
> >
> > Thus, now IgfsPerBlockLruEvictionPolicy is not quite valid, and some of
> > corresponding tests fail
> > (org.apache.ignite.internal.processors.igfs.
> IgfsCachePerBlockLruEvictionPolicySelfTest#testDataSizeEviction,
> > org.apache.ignite.internal.processors.igfs.
> IgfsCachePerBlockLruEvictionPolicySelfTest#testBlockCountEviction)
> >
> > So, the options I see are:
> > 1) deprecate/remove IgfsPerBlockLruEvictionPolicy ;
> > 2) leave it as is, but explain in javadocs that it only works for on-heap
> > layer, that does not in fact evict  blocks from the underlying offheap
> > layer.
> >
> > Please share your opinions.
>
>

[Ivan V.]

I think, we should answer the following questions.
1) does the interface org.apache.ignite.cache.eviction.EvictionPolicy and
*all* its implementations now de-facto get deprecated? (I mean, the
question appears to be wider than just the IGFS eviction policy).
2) is on-heap data access faster than off-heap? If yes, how large the
on-heap vs. off-heap difference is?
3) does an ability to evict from on-heap layer make any sense at all if the
same data are anyway backed up by off-heap cache layer, that has different
eviction policies?


On Thu, May 25, 2017 at 2:42 PM, Alexey Goncharuk <
[hidden email]> wrote:

> Guys, I think it makes little or no sense to keep blocks on-heap. If I
> understand correctly, the eviction policy was used in combined modes where
> partial data eviction is allowed. To make this work in the new PageMemory
> architecture, we only need to make sure that IGFS block size equals to the
> data page free space size. In this case, our standard offheap eviction
> policy will be semantically equal to the old per-block eviction policy. No
> need to go on-heap at all.
>
> Thoughts?
>
> 2017-05-25 4:21 GMT+03:00 Denis Magda <[hidden email]>:
>
> > Hi Ivan,
> >
> > I’m for this approach
> >
> > > 2) leave it as is, but explain in javadocs that it only works for
> on-heap
> > > layer, that does not in fact evict  blocks from the underlying offheap
> > > layer.
> >
> > because it should be feasible to enable on-heap caching for IGFS, right?
> > Using the memory policies. So, I would reimplement the tests with the
> > on-heap caching enabling and checking up that data is pushed out of the
> > heap.
> >
> > —
> > Denis
> >
> > > On May 24, 2017, at 9:57 AM, Ivan V. <[hidden email]>
> wrote:
> > >
> > > Hi, colleagues,
> > >
> > > as Ignite caches moved to paged offheap memory , the
> > > IgfsPerBlockLruEvictionPolicy does not seem to work as expected any
> more,
> > > because
> > > 1) interface org.apache.ignite.cache.eviction.EvictionPolicy now only
> > > defines "eviction from on-heap", not a real eviction, because each
> > on-heap
> > > cache is now accompanied with underlying off-heap cache. It can become
> > > "real eviction" only for "on-heap-only" mode caches, once they get
> > > implemented.
> > > 2) for off-heap eviction an entire page is evicted, not a specific k-v
> > > pair, and LRU policy is not exactly LRU any more (see
> > > org.apache.ignite.configuration.DataPageEvictionMode#RANDOM_LRU). So,
> it
> > > appears to be impossible to re-implement this policy for the off-heap
> > layer.
> > >
> > > Thus, now IgfsPerBlockLruEvictionPolicy is not quite valid, and some of
> > > corresponding tests fail
> > > (org.apache.ignite.internal.processors.igfs.
> > IgfsCachePerBlockLruEvictionPolicySelfTest#testDataSizeEviction,
> > > org.apache.ignite.internal.processors.igfs.
> > IgfsCachePerBlockLruEvictionPolicySelfTest#testBlockCountEviction)
> > >
> > > So, the options I see are:
> > > 1) deprecate/remove IgfsPerBlockLruEvictionPolicy ;
> > > 2) leave it as is, but explain in javadocs that it only works for
> on-heap
> > > layer, that does not in fact evict  blocks from the underlying offheap
> > > layer.
> > >
> > > Please share your opinions.
> >
> >
>

[dmagda]

> On May 25, 2017, at 5:17 AM, Ivan V. <[hidden email]> wrote:
>
> I think, we should answer the following questions.
> 1) does the interface org.apache.ignite.cache.eviction.EvictionPolicy and
> *all* its implementations now de-facto get deprecated? (I mean, the
> question appears to be wider than just the IGFS eviction policy).

Now, this interface is used for “on-heap caching” scenario:
https://apacheignite.readme.io/docs/page-memory#on-heap-caching

Here is we clarify what this interfaced is used for:
https://apacheignite.readme.io/docs/evictions#on-heap-cache-entries-based-eviction

> 2) is on-heap data access faster than off-heap? If yes, how large the
> on-heap vs. off-heap difference is?

It’s comparable. Hope the guys can share precise numbers.

In general, the on-heap caching is for scenarios when you do a lot of cache reads on server nodes that work with cache entries in the binary form or invoke cache entries' deserialization. For instance, this might happen when a distributed computation or deployed service gets some data from caches for further processing.

> 3) does an ability to evict from on-heap layer make any sense at all if the
> same data are anyway backed up by off-heap cache layer, that has different
> eviction policies?

As soon as you enable the on-heap caching you should set up a eviction policy. Otherwise the Java heap can grow endlessly.

—
Denis

>
>
> On Thu, May 25, 2017 at 2:42 PM, Alexey Goncharuk <
> [hidden email]> wrote:
>
>> Guys, I think it makes little or no sense to keep blocks on-heap. If I
>> understand correctly, the eviction policy was used in combined modes where
>> partial data eviction is allowed. To make this work in the new PageMemory
>> architecture, we only need to make sure that IGFS block size equals to the
>> data page free space size. In this case, our standard offheap eviction
>> policy will be semantically equal to the old per-block eviction policy. No
>> need to go on-heap at all.
>>
>> Thoughts?
>>
>> 2017-05-25 4:21 GMT+03:00 Denis Magda <[hidden email]>:
>>
>>> Hi Ivan,
>>>
>>> I’m for this approach
>>>
>>>> 2) leave it as is, but explain in javadocs that it only works for
>> on-heap
>>>> layer, that does not in fact evict  blocks from the underlying offheap
>>>> layer.
>>>
>>> because it should be feasible to enable on-heap caching for IGFS, right?
>>> Using the memory policies. So, I would reimplement the tests with the
>>> on-heap caching enabling and checking up that data is pushed out of the
>>> heap.
>>>
>>> —
>>> Denis
>>>
>>>> On May 24, 2017, at 9:57 AM, Ivan V. <[hidden email]>
>> wrote:
>>>>
>>>> Hi, colleagues,
>>>>
>>>> as Ignite caches moved to paged offheap memory , the
>>>> IgfsPerBlockLruEvictionPolicy does not seem to work as expected any
>> more,
>>>> because
>>>> 1) interface org.apache.ignite.cache.eviction.EvictionPolicy now only
>>>> defines "eviction from on-heap", not a real eviction, because each
>>> on-heap
>>>> cache is now accompanied with underlying off-heap cache. It can become
>>>> "real eviction" only for "on-heap-only" mode caches, once they get
>>>> implemented.
>>>> 2) for off-heap eviction an entire page is evicted, not a specific k-v
>>>> pair, and LRU policy is not exactly LRU any more (see
>>>> org.apache.ignite.configuration.DataPageEvictionMode#RANDOM_LRU). So,
>> it
>>>> appears to be impossible to re-implement this policy for the off-heap
>>> layer.
>>>>
>>>> Thus, now IgfsPerBlockLruEvictionPolicy is not quite valid, and some of
>>>> corresponding tests fail
>>>> (org.apache.ignite.internal.processors.igfs.
>>> IgfsCachePerBlockLruEvictionPolicySelfTest#testDataSizeEviction,
>>>> org.apache.ignite.internal.processors.igfs.
>>> IgfsCachePerBlockLruEvictionPolicySelfTest#testBlockCountEviction)
>>>>
>>>> So, the options I see are:
>>>> 1) deprecate/remove IgfsPerBlockLruEvictionPolicy ;
>>>> 2) leave it as is, but explain in javadocs that it only works for
>> on-heap
>>>> layer, that does not in fact evict  blocks from the underlying offheap
>>>> layer.
>>>>
>>>> Please share your opinions.
>>>
>>>
>>

[Ivan V.]

Denis, many thanks for the info.

Phrase in docs "on-heap caching is for scenarios when you do a lot of cache
reads on server nodes that *work with cache entries in the binary form* or
invoke cache entries' deserialization."
may be contains a mistake: should it say "*work with cache entries in
non-binary form*" instead?

As applied to IGFS, its data (file blocks) are binary by nature (byte[]
arrays), and do not assume any deserialization.
So, I would agree with @Ivan Rakov and @Alexey Goncharuk -- if the access
speed is nearly the same, there is no sense to store IGFS file blocks
on-heap.

It looks like option "1) deprecate/remove IgfsPerBlockLruEvictionPolicy "
 is the way to go.  Please let me know if there are any objections.

On Thu, May 25, 2017 at 9:25 PM, Denis Magda <[hidden email]> wrote:

>
> > On May 25, 2017, at 5:17 AM, Ivan V. <[hidden email]> wrote:
> >
> > I think, we should answer the following questions.
> > 1) does the interface org.apache.ignite.cache.eviction.EvictionPolicy
> and
> > *all* its implementations now de-facto get deprecated? (I mean, the
> > question appears to be wider than just the IGFS eviction policy).
>
> Now, this interface is used for “on-heap caching” scenario:
> https://apacheignite.readme.io/docs/page-memory#on-heap-caching
>
> Here is we clarify what this interfaced is used for:
> https://apacheignite.readme.io/docs/evictions#on-heap-
> cache-entries-based-eviction
>
> > 2) is on-heap data access faster than off-heap? If yes, how large the
> > on-heap vs. off-heap difference is?
>
> It’s comparable. Hope the guys can share precise numbers.
>
> In general, the on-heap caching is for scenarios when you do a lot of
> cache reads on server nodes that work with cache entries in the binary form
> or invoke cache entries' deserialization. For instance, this might happen
> when a distributed computation or deployed service gets some data from
> caches for further processing.
>
> > 3) does an ability to evict from on-heap layer make any sense at all if
> the
> > same data are anyway backed up by off-heap cache layer, that has
> different
> > eviction policies?
>
> As soon as you enable the on-heap caching you should set up a eviction
> policy. Otherwise the Java heap can grow endlessly.
>
> —
> Denis
>
> >
> >
> > On Thu, May 25, 2017 at 2:42 PM, Alexey Goncharuk <
> > [hidden email]> wrote:
> >
> >> Guys, I think it makes little or no sense to keep blocks on-heap. If I
> >> understand correctly, the eviction policy was used in combined modes
> where
> >> partial data eviction is allowed. To make this work in the new
> PageMemory
> >> architecture, we only need to make sure that IGFS block size equals to
> the
> >> data page free space size. In this case, our standard offheap eviction
> >> policy will be semantically equal to the old per-block eviction policy.
> No
> >> need to go on-heap at all.
> >>
> >> Thoughts?
> >>
> >> 2017-05-25 4:21 GMT+03:00 Denis Magda <[hidden email]>:
> >>
> >>> Hi Ivan,
> >>>
> >>> I’m for this approach
> >>>
> >>>> 2) leave it as is, but explain in javadocs that it only works for
> >> on-heap
> >>>> layer, that does not in fact evict  blocks from the underlying offheap
> >>>> layer.
> >>>
> >>> because it should be feasible to enable on-heap caching for IGFS,
> right?
> >>> Using the memory policies. So, I would reimplement the tests with the
> >>> on-heap caching enabling and checking up that data is pushed out of the
> >>> heap.
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On May 24, 2017, at 9:57 AM, Ivan V. <[hidden email]>
> >> wrote:
> >>>>
> >>>> Hi, colleagues,
> >>>>
> >>>> as Ignite caches moved to paged offheap memory , the
> >>>> IgfsPerBlockLruEvictionPolicy does not seem to work as expected any
> >> more,
> >>>> because
> >>>> 1) interface org.apache.ignite.cache.eviction.EvictionPolicy now only
> >>>> defines "eviction from on-heap", not a real eviction, because each
> >>> on-heap
> >>>> cache is now accompanied with underlying off-heap cache. It can become
> >>>> "real eviction" only for "on-heap-only" mode caches, once they get
> >>>> implemented.
> >>>> 2) for off-heap eviction an entire page is evicted, not a specific k-v
> >>>> pair, and LRU policy is not exactly LRU any more (see
> >>>> org.apache.ignite.configuration.DataPageEvictionMode#RANDOM_LRU). So,
> >> it
> >>>> appears to be impossible to re-implement this policy for the off-heap
> >>> layer.
> >>>>
> >>>> Thus, now IgfsPerBlockLruEvictionPolicy is not quite valid, and some
> of
> >>>> corresponding tests fail
> >>>> (org.apache.ignite.internal.processors.igfs.
> >>> IgfsCachePerBlockLruEvictionPolicySelfTest#testDataSizeEviction,
> >>>> org.apache.ignite.internal.processors.igfs.
> >>> IgfsCachePerBlockLruEvictionPolicySelfTest#testBlockCountEviction)
> >>>>
> >>>> So, the options I see are:
> >>>> 1) deprecate/remove IgfsPerBlockLruEvictionPolicy ;
> >>>> 2) leave it as is, but explain in javadocs that it only works for
> >> on-heap
> >>>> layer, that does not in fact evict  blocks from the underlying offheap
> >>>> layer.
> >>>>
> >>>> Please share your opinions.
> >>>
> >>>
> >>
>
>