Data eviction/expiration from Ignite persistence

[Dmitriy Pavlov]

Hi Denis,

I've created summary of terms here
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Eviction,rotationandexpiration

Hope this should solve a lot of miscommunications.

In this thread I meant only Expiration (aka TTL)

Sincerely,
Dmitriy Pavlov

вт, 13 мар. 2018 г. в 10:50, Alexey Goncharuk <[hidden email]>:

> Denis,
>
> What do you mean by 'current behavior when data is evicted from the memory
> only'? TTL expiration effectively means that the corresponding key-value
> pairs are destroyed. If you are talking about page replacement, then there
> is no way to do this on per-key basis because a page must be replaced as a
> whole and it makes no sense to track keys.
>
> --AG
>
> 2018-03-13 0:03 GMT+03:00 Denis Magda <[hidden email]>:
>
> > Dmitriy,
> >
> > It will break the current default behavior when data is evicted from the
> > memory only, and I would disagree that it's a right decision overall.
> >
> > There are many scenarios when users need to have the eviction in the
> memory
> > layer and preserve data on disk for later usage. So, can we keep what we
> > have now and merely expand the eviction to disk if the user requires it?
> >
> > --
> > Denis
> >
> >
> > On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > Denis,
> > >
> > > I suppose there is no configuration will be required. If TTL configured
> > > entry will be removed from disk & memory both.
> > >
> > > SIncerely,
> > > Dmitriy Pavlov
> > >
> > > пн, 12 мар. 2018 г. в 23:32, Denis Magda <[hidden email]>:
> > >
> > > > Alexey, Dmitriy,
> > > >
> > > > What would be the configuration parameter that defines if the
> eviction
> > > > should happen in the in-memory layer only or in both memory and disk?
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > > [hidden email]> wrote:
> > > >
> > > > > Val,
> > > > >
> > > > > Yes, the entries will be removed from both memory and persistence.
> > > > >
> > > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > > [hidden email]>:
> > > > >
> > > > > > Alex,
> > > > > >
> > > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> > expired
> > > > > entry
> > > > > > be removed from both memory and persistence?
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > > [hidden email]> wrote:
> > > > > >
> > > > > > > The ticket [1] is in patch available state looks good, the only
> > > thing
> > > > > > left
> > > > > > > there is to transfer old entries to new storage. I hope Andrey
> > will
> > > > > have
> > > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > > >
> > > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> > > > > > >
> > > > > > > > Val,
> > > > > > > >
> > > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please
> chime
> > > in.
> > > > > > > >
> > > > > > > > In general, the more deployments the persistence will get the
> > > more
> > > > > > demand
> > > > > > > > we will see for that capability. Personally, I'd create a
> > ticket
> > > > for
> > > > > > now
> > > > > > > > and put it off to our backlog.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Denis
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > As far as I know there is no plans. Denis please correct
> me
> > > if
> > > > > I'm
> > > > > > > > wrong.
> > > > > > > > > >
> > > > > > > > > > But users found these or similar bugs, it seems we need
> to
> > > > > support
> > > > > > > PDS
> > > > > > > > > and
> > > > > > > > > > TTL.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > We should either support it or throw a clear  exception on
> > > > startup
> > > > > > > > clearly
> > > > > > > > > stating that it is not. I suppose, it should be the latter
> > for
> > > > now.
> > > > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

On Tue, Mar 13, 2018 at 3:50 AM, Alexey Goncharuk <
[hidden email]> wrote:

> Denis,
>
> What do you mean by 'current behavior when data is evicted from the memory
> only'? TTL expiration effectively means that the corresponding key-value
> pairs are destroyed.


AG, can you please explain how this is implemented? Are we really keeping
the whole TTL map in memory?

[Alexey Goncharuk]

Dmitriy,

The TTL map is a regular B+Tree-based map with the key being a pair (expire
time + key). It is obviously stored in memory when there is no persistence.
When persistence is enabled, it is handled as any other index tree - the
page replacement algorithm is applied. No heap is consumed for this tree.

--AG

2018-03-13 15:30 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

>
>
> On Tue, Mar 13, 2018 at 3:50 AM, Alexey Goncharuk <
> [hidden email]> wrote:
>
>> Denis,
>>
>> What do you mean by 'current behavior when data is evicted from the memory
>> only'? TTL expiration effectively means that the corresponding key-value
>> pairs are destroyed.
>
>
> AG, can you please explain how this is implemented? Are we really keeping
> the whole TTL map in memory?
>
>

[dmagda]

Alexey,

My understanding was that the expiration policies worked for data in RAM
only. Ok, if an expired entry is removed from both RAM and Ignite
persistence then what happens if a cache store is used instead of Ignite
storage? Do we remove expired entries from RDBMs, Cassandra, etc? My guess
that we don't which doesn't look consistent product wide.

--
Denis

On Tue, Mar 13, 2018 at 12:50 AM, Alexey Goncharuk <
[hidden email]> wrote:

> Denis,
>
> What do you mean by 'current behavior when data is evicted from the memory
> only'? TTL expiration effectively means that the corresponding key-value
> pairs are destroyed. If you are talking about page replacement, then there
> is no way to do this on per-key basis because a page must be replaced as a
> whole and it makes no sense to track keys.
>
> --AG
>
> 2018-03-13 0:03 GMT+03:00 Denis Magda <[hidden email]>:
>
> > Dmitriy,
> >
> > It will break the current default behavior when data is evicted from the
> > memory only, and I would disagree that it's a right decision overall.
> >
> > There are many scenarios when users need to have the eviction in the
> memory
> > layer and preserve data on disk for later usage. So, can we keep what we
> > have now and merely expand the eviction to disk if the user requires it?
> >
> > --
> > Denis
> >
> >
> > On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > Denis,
> > >
> > > I suppose there is no configuration will be required. If TTL configured
> > > entry will be removed from disk & memory both.
> > >
> > > SIncerely,
> > > Dmitriy Pavlov
> > >
> > > пн, 12 мар. 2018 г. в 23:32, Denis Magda <[hidden email]>:
> > >
> > > > Alexey, Dmitriy,
> > > >
> > > > What would be the configuration parameter that defines if the
> eviction
> > > > should happen in the in-memory layer only or in both memory and disk?
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > > [hidden email]> wrote:
> > > >
> > > > > Val,
> > > > >
> > > > > Yes, the entries will be removed from both memory and persistence.
> > > > >
> > > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > > [hidden email]>:
> > > > >
> > > > > > Alex,
> > > > > >
> > > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> > expired
> > > > > entry
> > > > > > be removed from both memory and persistence?
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > > [hidden email]> wrote:
> > > > > >
> > > > > > > The ticket [1] is in patch available state looks good, the only
> > > thing
> > > > > > left
> > > > > > > there is to transfer old entries to new storage. I hope Andrey
> > will
> > > > > have
> > > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > > >
> > > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> > > > > > >
> > > > > > > > Val,
> > > > > > > >
> > > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please
> chime
> > > in.
> > > > > > > >
> > > > > > > > In general, the more deployments the persistence will get the
> > > more
> > > > > > demand
> > > > > > > > we will see for that capability. Personally, I'd create a
> > ticket
> > > > for
> > > > > > now
> > > > > > > > and put it off to our backlog.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Denis
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > As far as I know there is no plans. Denis please correct
> me
> > > if
> > > > > I'm
> > > > > > > > wrong.
> > > > > > > > > >
> > > > > > > > > > But users found these or similar bugs, it seems we need
> to
> > > > > support
> > > > > > > PDS
> > > > > > > > > and
> > > > > > > > > > TTL.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > We should either support it or throw a clear  exception on
> > > > startup
> > > > > > > > clearly
> > > > > > > > > stating that it is not. I suppose, it should be the latter
> > for
> > > > now.
> > > > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Alexey Goncharuk]

Denis,

With the approach of Ignite Durable Memory there is no difference between
'memory' and 'disk'. The data is expired from the Ignite data storage which
can be persisted or not. Before persistence was introduced, TTL was mostly
used when write-through was enabled, otherwise data was cleared from Ignite
data storage. Currently, the situation stays the same - if an entry is
expired, it is removed from the Ignite storage, which looks absolutely
consistent to me.

2018-03-13 21:30 GMT+03:00 Denis Magda <[hidden email]>:

> Alexey,
>
> My understanding was that the expiration policies worked for data in RAM
> only. Ok, if an expired entry is removed from both RAM and Ignite
> persistence then what happens if a cache store is used instead of Ignite
> storage? Do we remove expired entries from RDBMs, Cassandra, etc? My guess
> that we don't which doesn't look consistent product wide.
>
> --
> Denis
>
> On Tue, Mar 13, 2018 at 12:50 AM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > Denis,
> >
> > What do you mean by 'current behavior when data is evicted from the
> memory
> > only'? TTL expiration effectively means that the corresponding key-value
> > pairs are destroyed. If you are talking about page replacement, then
> there
> > is no way to do this on per-key basis because a page must be replaced as
> a
> > whole and it makes no sense to track keys.
> >
> > --AG
> >
> > 2018-03-13 0:03 GMT+03:00 Denis Magda <[hidden email]>:
> >
> > > Dmitriy,
> > >
> > > It will break the current default behavior when data is evicted from
> the
> > > memory only, and I would disagree that it's a right decision overall.
> > >
> > > There are many scenarios when users need to have the eviction in the
> > memory
> > > layer and preserve data on disk for later usage. So, can we keep what
> we
> > > have now and merely expand the eviction to disk if the user requires
> it?
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov <[hidden email]>
> > > wrote:
> > >
> > > > Denis,
> > > >
> > > > I suppose there is no configuration will be required. If TTL
> configured
> > > > entry will be removed from disk & memory both.
> > > >
> > > > SIncerely,
> > > > Dmitriy Pavlov
> > > >
> > > > пн, 12 мар. 2018 г. в 23:32, Denis Magda <[hidden email]>:
> > > >
> > > > > Alexey, Dmitriy,
> > > > >
> > > > > What would be the configuration parameter that defines if the
> > eviction
> > > > > should happen in the in-memory layer only or in both memory and
> disk?
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > > > [hidden email]> wrote:
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > Yes, the entries will be removed from both memory and
> persistence.
> > > > > >
> > > > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > > > [hidden email]>:
> > > > > >
> > > > > > > Alex,
> > > > > > >
> > > > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> > > expired
> > > > > > entry
> > > > > > > be removed from both memory and persistence?
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > > > [hidden email]> wrote:
> > > > > > >
> > > > > > > > The ticket [1] is in patch available state looks good, the
> only
> > > > thing
> > > > > > > left
> > > > > > > > there is to transfer old entries to new storage. I hope
> Andrey
> > > will
> > > > > > have
> > > > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > > > >
> > > > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> > > > > > > >
> > > > > > > > > Val,
> > > > > > > > >
> > > > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please
> > chime
> > > > in.
> > > > > > > > >
> > > > > > > > > In general, the more deployments the persistence will get
> the
> > > > more
> > > > > > > demand
> > > > > > > > > we will see for that capability. Personally, I'd create a
> > > ticket
> > > > > for
> > > > > > > now
> > > > > > > > > and put it off to our backlog.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > As far as I know there is no plans. Denis please
> correct
> > me
> > > > if
> > > > > > I'm
> > > > > > > > > wrong.
> > > > > > > > > > >
> > > > > > > > > > > But users found these or similar bugs, it seems we need
> > to
> > > > > > support
> > > > > > > > PDS
> > > > > > > > > > and
> > > > > > > > > > > TTL.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > We should either support it or throw a clear  exception
> on
> > > > > startup
> > > > > > > > > clearly
> > > > > > > > > > stating that it is not. I suppose, it should be the
> latter
> > > for
> > > > > now.
> > > > > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

On Wed, Mar 14, 2018 at 1:41 PM, Alexey Goncharuk <
[hidden email]> wrote:

> Denis,
>
> With the approach of Ignite Durable Memory there is no difference between
> 'memory' and 'disk'. The data is expired from the Ignite data storage which
> can be persisted or not. Before persistence was introduced, TTL was mostly
> used when write-through was enabled, otherwise data was cleared from Ignite
> data storage. Currently, the situation stays the same - if an entry is
> expired, it is removed from the Ignite storage, which looks absolutely
> consistent to me.
>

Agree with AG. There is a difference between expiration and eviction. If an
entry is expired, then it should be removed from the store, regardless if
it is in memory or on disk.

However, evicting from memory because there is not enough space does not
remove an entry from the store.

D.

[dmagda]

>
> Agree with AG. There is a difference between expiration and eviction. If an
> entry is expired, then it should be removed from the store, regardless if
> it is in memory or on disk.


Well, then it works this way now depending on a memory configuration:

   - memory only mode: expired entry removed from memory storage
   - memory + Ignite persistence: expired entry removed from both memory
   and disk tiers
   - memory + 3rd party: expired entry is removed from the memory storage
   only.

Let me know if I'm wrong somewhere. Otherwise, I'll improve the docs to
bring more clarity.

However, evicting from memory because there is not enough space does not
> remove an entry from the store.


That was another topic of the discussion. How can we support the *eviction*
from disk (by using a particular configuration parameter)?

--
Denis

On Wed, Mar 14, 2018 at 7:01 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Wed, Mar 14, 2018 at 1:41 PM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > Denis,
> >
> > With the approach of Ignite Durable Memory there is no difference between
> > 'memory' and 'disk'. The data is expired from the Ignite data storage
> which
> > can be persisted or not. Before persistence was introduced, TTL was
> mostly
> > used when write-through was enabled, otherwise data was cleared from
> Ignite
> > data storage. Currently, the situation stays the same - if an entry is
> > expired, it is removed from the Ignite storage, which looks absolutely
> > consistent to me.
> >
>
> Agree with AG. There is a difference between expiration and eviction. If an
> entry is expired, then it should be removed from the store, regardless if
> it is in memory or on disk.
>
> However, evicting from memory because there is not enough space does not
> remove an entry from the store.
>
> D.
>

[dsetrakyan]

On Thu, Mar 15, 2018 at 12:31 PM, Denis Magda <[hidden email]> wrote:

> >
> > Agree with AG. There is a difference between expiration and eviction. If
> an
> > entry is expired, then it should be removed from the store, regardless if
> > it is in memory or on disk.
>
>
> Well, then it works this way now depending on a memory configuration:
>
>    - memory only mode: expired entry removed from memory storage
>    - memory + Ignite persistence: expired entry removed from both memory
>    and disk tiers
>    - memory + 3rd party: expired entry is removed from the memory storage
>    only.
>

I think it works correctly. Whenever Ignite is configured with a 3rd party
database, entries are usually expired from memory exactly because the
database may have a more recent value. This way, the more recent value will
be reloaded form database on next access.


> Let me know if I'm wrong somewhere. Otherwise, I'll improve the docs to
> bring more clarity.
>

I believe you are correct.


> > However, evicting from memory because there is not enough space does not
> remove an entry from the store.
>
>
> That was another topic of the discussion. How can we support the *eviction*
> from disk (by using a particular configuration parameter)?
>

I do not think we need to evict from disk. None of the other databases
support it, and I do not think it is a required feature. Also, it will be
hard to implement in my view.