Apache Ignite Developers - Legacy Mail Archive

Data eviction/expiration from Ignite persistence

Classic

List

Threaded

28 messages Options

dmagda

Data eviction/expiration from Ignite persistence

Igniters,

I’ve been talking to many Ignite users here and there who are already on Ignite persistence or consider to turn it on. The majority of them are more than satisfied with its current state and provided capabilities. That’s is really good news for us.

However, I tend to come across the people who ask about eviction/expiration policies for the persistence itself. Had around 6 conversation about the topic this month only.

Usually the requirement is connected with a streaming use case. When an application streams a lot of data (IoT, metrics, etc.) to the cluster but the data becomes stale in some period of time (day, couple of days, etc.). The user doesn’t want to waste the disk space and needs to simple purge the data from there.

My suggestion here is to create a timer task that will remove the stale data from the cluster. However, since the demand is growing probably it’s a good time to discuss a feasibility of this feature.

Alex G, as the main architect of the persistence, could you share your thoughts on this? What will it cost to us to support eviction/expiration for the persistence?

—
Denis

Dmitriy Pavlov

Re: Data eviction/expiration from Ignite persistence

Hi Denis,

What is the difference of required by users functionality with TTL cache
expiration?

By some posts I can suppose TTL cache is compatible with native
persistence.

Sincerely,
Dmitriy Pavlov

сб, 18 нояб. 2017 г. в 0:41, Denis Magda <[hidden email]>:

> Igniters,
>
> I’ve been talking to many Ignite users here and there who are already on
> Ignite persistence or consider to turn it on. The majority of them are more
> than satisfied with its current state and provided capabilities. That’s is
> really good news for us.
>
> However, I tend to come across the people who ask about
> eviction/expiration policies for the persistence itself. Had around 6
> conversation about the topic this month only.
>
> Usually the requirement is connected with a streaming use case. When an
> application streams a lot of data (IoT, metrics, etc.) to the cluster but
> the data becomes stale in some period of time (day, couple of days, etc.).
> The user doesn’t want to waste the disk space and needs to simple purge the
> data from there.
>
> My suggestion here is to create a timer task that will remove the stale
> data from the cluster. However, since the demand is growing probably it’s a
> good time to discuss a feasibility of this feature.
>
> Alex G, as the main architect of the persistence, could you share your
> thoughts on this? What will it cost to us to support eviction/expiration
> for the persistence?
>
> —
> Denis

Dmitriy Pavlov

Re: Data eviction/expiration from Ignite persistence

Hi Denis,

Is this need covered by PDS + TTL?

For the very first TTL test, I found some delay after applying TTL with the
repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964

And I'm wondering if the user's needs are covered by
https://apacheignite.readme.io/docs/expiry-policies plus
https://apacheignite.readme.io/docs/distributed-persistent-store

Sincerely,
Dmitriy Pavlov

сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov <[hidden email]>:

> Hi Denis,
>
> What is the difference of required by users functionality with TTL cache
> expiration?
>
> By some posts I can suppose TTL cache is compatible with native
> persistence.
>
> Sincerely,
> Dmitriy Pavlov
>
> сб, 18 нояб. 2017 г. в 0:41, Denis Magda <[hidden email]>:
>
>> Igniters,
>>
>> I’ve been talking to many Ignite users here and there who are already on
>> Ignite persistence or consider to turn it on. The majority of them are more
>> than satisfied with its current state and provided capabilities. That’s is
>> really good news for us.
>>
>> However, I tend to come across the people who ask about
>> eviction/expiration policies for the persistence itself. Had around 6
>> conversation about the topic this month only.
>>
>> Usually the requirement is connected with a streaming use case. When an
>> application streams a lot of data (IoT, metrics, etc.) to the cluster but
>> the data becomes stale in some period of time (day, couple of days, etc.).
>> The user doesn’t want to waste the disk space and needs to simple purge the
>> data from there.
>>
>> My suggestion here is to create a timer task that will remove the stale
>> data from the cluster. However, since the demand is growing probably it’s a
>> good time to discuss a feasibility of this feature.
>>
>> Alex G, as the main architect of the persistence, could you share your
>> thoughts on this? What will it cost to us to support eviction/expiration
>> for the persistence?
>>
>> —
>> Denis
>
>

dmagda

Re: Data eviction/expiration from Ignite persistence

Dmitriy,

That’s about TTL and eviction support for Ignite persistence. Presently if you set an expiration or eviction policy for a cache it will be applied for data stored in memory. The policy never affects the persistence layer.

—
Denis

> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov <[hidden email]> wrote:
>
> Hi Denis,
>
> Is this need covered by PDS + TTL?
>
> For the very first TTL test, I found some delay after applying TTL with the
> repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
>
> And I'm wondering if the user's needs are covered by
> https://apacheignite.readme.io/docs/expiry-policies plus
> https://apacheignite.readme.io/docs/distributed-persistent-store
>
> Sincerely,
> Dmitriy Pavlov
>
> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov <[hidden email]>:
>
>> Hi Denis,
>>
>> What is the difference of required by users functionality with TTL cache
>> expiration?
>>
>> By some posts I can suppose TTL cache is compatible with native
>> persistence.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> сб, 18 нояб. 2017 г. в 0:41, Denis Magda <[hidden email]>:
>>
>>> Igniters,
>>>
>>> I’ve been talking to many Ignite users here and there who are already on
>>> Ignite persistence or consider to turn it on. The majority of them are more
>>> than satisfied with its current state and provided capabilities. That’s is
>>> really good news for us.
>>>
>>> However, I tend to come across the people who ask about
>>> eviction/expiration policies for the persistence itself. Had around 6
>>> conversation about the topic this month only.
>>>
>>> Usually the requirement is connected with a streaming use case. When an
>>> application streams a lot of data (IoT, metrics, etc.) to the cluster but
>>> the data becomes stale in some period of time (day, couple of days, etc.).
>>> The user doesn’t want to waste the disk space and needs to simple purge the
>>> data from there.
>>>
>>> My suggestion here is to create a timer task that will remove the stale
>>> data from the cluster. However, since the demand is growing probably it’s a
>>> good time to discuss a feasibility of this feature.
>>>
>>> Alex G, as the main architect of the persistence, could you share your
>>> thoughts on this? What will it cost to us to support eviction/expiration
>>> for the persistence?
>>>
>>> —
>>> Denis
>>
>>

Dmitriy Pavlov

Re: Data eviction/expiration from Ignite persistence

Hi Denis,

Second fix we need to do is B+ tree separation in per-partition basis:
https://issues.apache.org/jira/browse/IGNITE-5874

Should we take into account compatibilty issues with previous Ignite
persistent store versions, because current TTL tree is persisted, and will
change its format?

Sincerely,
Dmitriy Pavlov

вт, 21 нояб. 2017 г. в 2:13, Denis Magda <[hidden email]>:

> Dmitriy,
>
> That’s about TTL and eviction support for Ignite persistence. Presently if
> you set an expiration or eviction policy for a cache it will be applied for
> data stored in memory. The policy never affects the persistence layer.
>
> —
> Denis
>
> > On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov <[hidden email]>
> wrote:
> >
> > Hi Denis,
> >
> > Is this need covered by PDS + TTL?
> >
> > For the very first TTL test, I found some delay after applying TTL with
> the
> > repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
> >
> > And I'm wondering if the user's needs are covered by
> > https://apacheignite.readme.io/docs/expiry-policies plus
> > https://apacheignite.readme.io/docs/distributed-persistent-store
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov <[hidden email]>:
> >
> >> Hi Denis,
> >>
> >> What is the difference of required by users functionality with TTL cache
> >> expiration?
> >>
> >> By some posts I can suppose TTL cache is compatible with native
> >> persistence.
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> сб, 18 нояб. 2017 г. в 0:41, Denis Magda <[hidden email]>:
> >>
> >>> Igniters,
> >>>
> >>> I’ve been talking to many Ignite users here and there who are already
> on
> >>> Ignite persistence or consider to turn it on. The majority of them are
> more
> >>> than satisfied with its current state and provided capabilities.
> That’s is
> >>> really good news for us.
> >>>
> >>> However, I tend to come across the people who ask about
> >>> eviction/expiration policies for the persistence itself. Had around 6
> >>> conversation about the topic this month only.
> >>>
> >>> Usually the requirement is connected with a streaming use case. When an
> >>> application streams a lot of data (IoT, metrics, etc.) to the cluster
> but
> >>> the data becomes stale in some period of time (day, couple of days,
> etc.).
> >>> The user doesn’t want to waste the disk space and needs to simple
> purge the
> >>> data from there.
> >>>
> >>> My suggestion here is to create a timer task that will remove the stale
> >>> data from the cluster. However, since the demand is growing probably
> it’s a
> >>> good time to discuss a feasibility of this feature.
> >>>
> >>> Alex G, as the main architect of the persistence, could you share your
> >>> thoughts on this? What will it cost to us to support
> eviction/expiration
> >>> for the persistence?
> >>>
> >>> —
> >>> Denis
> >>
> >>
>
>

dmagda

Re: Data eviction/expiration from Ignite persistence

We might break the compatibility for the next major release or even create a tool that will migrate persistence files from an old to new formats.

—
Denis

> On Nov 21, 2017, at 8:34 AM, Dmitry Pavlov <[hidden email]> wrote:
>
> Hi Denis,
>
> Second fix we need to do is B+ tree separation in per-partition basis:
> https://issues.apache.org/jira/browse/IGNITE-5874
>
> Should we take into account compatibilty issues with previous Ignite
> persistent store versions, because current TTL tree is persisted, and will
> change its format?
>
> Sincerely,
> Dmitriy Pavlov
>
>
> вт, 21 нояб. 2017 г. в 2:13, Denis Magda <[hidden email]>:
>
>> Dmitriy,
>>
>> That’s about TTL and eviction support for Ignite persistence. Presently if
>> you set an expiration or eviction policy for a cache it will be applied for
>> data stored in memory. The policy never affects the persistence layer.
>>
>> —
>> Denis
>>
>>> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov <[hidden email]>
>> wrote:
>>>
>>> Hi Denis,
>>>
>>> Is this need covered by PDS + TTL?
>>>
>>> For the very first TTL test, I found some delay after applying TTL with
>> the
>>> repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
>>>
>>> And I'm wondering if the user's needs are covered by
>>> https://apacheignite.readme.io/docs/expiry-policies plus
>>> https://apacheignite.readme.io/docs/distributed-persistent-store
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov <[hidden email]>:
>>>
>>>> Hi Denis,
>>>>
>>>> What is the difference of required by users functionality with TTL cache
>>>> expiration?
>>>>
>>>> By some posts I can suppose TTL cache is compatible with native
>>>> persistence.
>>>>
>>>> Sincerely,
>>>> Dmitriy Pavlov
>>>>
>>>> сб, 18 нояб. 2017 г. в 0:41, Denis Magda <[hidden email]>:
>>>>
>>>>> Igniters,
>>>>>
>>>>> I’ve been talking to many Ignite users here and there who are already
>> on
>>>>> Ignite persistence or consider to turn it on. The majority of them are
>> more
>>>>> than satisfied with its current state and provided capabilities.
>> That’s is
>>>>> really good news for us.
>>>>>
>>>>> However, I tend to come across the people who ask about
>>>>> eviction/expiration policies for the persistence itself. Had around 6
>>>>> conversation about the topic this month only.
>>>>>
>>>>> Usually the requirement is connected with a streaming use case. When an
>>>>> application streams a lot of data (IoT, metrics, etc.) to the cluster
>> but
>>>>> the data becomes stale in some period of time (day, couple of days,
>> etc.).
>>>>> The user doesn’t want to waste the disk space and needs to simple
>> purge the
>>>>> data from there.
>>>>>
>>>>> My suggestion here is to create a timer task that will remove the stale
>>>>> data from the cluster. However, since the demand is growing probably
>> it’s a
>>>>> good time to discuss a feasibility of this feature.
>>>>>
>>>>> Alex G, as the main architect of the persistence, could you share your
>>>>> thoughts on this? What will it cost to us to support
>> eviction/expiration
>>>>> for the persistence?
>>>>>
>>>>> —
>>>>> Denis
>>>>
>>>>
>>
>>

Valentin Kulichenko

Re: Data eviction/expiration from Ignite persistence

Guys,

What is the result of this discussion? Do we still not support eviction and
expiration on persistence level? If so, any plans to change this?

-Val

On Tue, Nov 21, 2017 at 9:45 AM, Denis Magda <[hidden email]> wrote:

> We might break the compatibility for the next major release or even create
> a tool that will migrate persistence files from an old to new formats.
>
> —
> Denis
>
> > On Nov 21, 2017, at 8:34 AM, Dmitry Pavlov <[hidden email]>
> wrote:
> >
> > Hi Denis,
> >
> > Second fix we need to do is B+ tree separation in per-partition basis:
> > https://issues.apache.org/jira/browse/IGNITE-5874
> >
> > Should we take into account compatibilty issues with previous Ignite
> > persistent store versions, because current TTL tree is persisted, and
> will
> > change its format?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> >
> > вт, 21 нояб. 2017 г. в 2:13, Denis Magda <[hidden email]>:
> >
> >> Dmitriy,
> >>
> >> That’s about TTL and eviction support for Ignite persistence. Presently
> if
> >> you set an expiration or eviction policy for a cache it will be applied
> for
> >> data stored in memory. The policy never affects the persistence layer.
> >>
> >> —
> >> Denis
> >>
> >>> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov <[hidden email]>
> >> wrote:
> >>>
> >>> Hi Denis,
> >>>
> >>> Is this need covered by PDS + TTL?
> >>>
> >>> For the very first TTL test, I found some delay after applying TTL with
> >> the
> >>> repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
> >>>
> >>> And I'm wondering if the user's needs are covered by
> >>> https://apacheignite.readme.io/docs/expiry-policies plus
> >>> https://apacheignite.readme.io/docs/distributed-persistent-store
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov <[hidden email]>:
> >>>
> >>>> Hi Denis,
> >>>>
> >>>> What is the difference of required by users functionality with TTL
> cache
> >>>> expiration?
> >>>>
> >>>> By some posts I can suppose TTL cache is compatible with native
> >>>> persistence.
> >>>>
> >>>> Sincerely,
> >>>> Dmitriy Pavlov
> >>>>
> >>>> сб, 18 нояб. 2017 г. в 0:41, Denis Magda <[hidden email]>:
> >>>>
> >>>>> Igniters,
> >>>>>
> >>>>> I’ve been talking to many Ignite users here and there who are already
> >> on
> >>>>> Ignite persistence or consider to turn it on. The majority of them
> are
> >> more
> >>>>> than satisfied with its current state and provided capabilities.
> >> That’s is
> >>>>> really good news for us.
> >>>>>
> >>>>> However, I tend to come across the people who ask about
> >>>>> eviction/expiration policies for the persistence itself. Had around 6
> >>>>> conversation about the topic this month only.
> >>>>>
> >>>>> Usually the requirement is connected with a streaming use case. When
> an
> >>>>> application streams a lot of data (IoT, metrics, etc.) to the cluster
> >> but
> >>>>> the data becomes stale in some period of time (day, couple of days,
> >> etc.).
> >>>>> The user doesn’t want to waste the disk space and needs to simple
> >> purge the
> >>>>> data from there.
> >>>>>
> >>>>> My suggestion here is to create a timer task that will remove the
> stale
> >>>>> data from the cluster. However, since the demand is growing probably
> >> it’s a
> >>>>> good time to discuss a feasibility of this feature.
> >>>>>
> >>>>> Alex G, as the main architect of the persistence, could you share
> your
> >>>>> thoughts on this? What will it cost to us to support
> >> eviction/expiration
> >>>>> for the persistence?
> >>>>>
> >>>>> —
> >>>>> Denis
> >>>>
> >>>>
> >>
> >>
>
>

Dmitriy Pavlov

Re: Data eviction/expiration from Ignite persistence

As far as I know there is no plans. Denis please correct me if I'm wrong.

But users found these or similar bugs, it seems we need to support PDS and
TTL.

пт, 9 мар. 2018 г., 4:36 Valentin Kulichenko <[hidden email]
>:

> Guys,
>
> What is the result of this discussion? Do we still not support eviction and
> expiration on persistence level? If so, any plans to change this?
>
> -Val
>
> On Tue, Nov 21, 2017 at 9:45 AM, Denis Magda <[hidden email]> wrote:
>
> > We might break the compatibility for the next major release or even
> create
> > a tool that will migrate persistence files from an old to new formats.
> >
> > —
> > Denis
> >
> > > On Nov 21, 2017, at 8:34 AM, Dmitry Pavlov <[hidden email]>
> > wrote:
> > >
> > > Hi Denis,
> > >
> > > Second fix we need to do is B+ tree separation in per-partition basis:
> > > https://issues.apache.org/jira/browse/IGNITE-5874
> > >
> > > Should we take into account compatibilty issues with previous Ignite
> > > persistent store versions, because current TTL tree is persisted, and
> > will
> > > change its format?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > >
> > > вт, 21 нояб. 2017 г. в 2:13, Denis Magda <[hidden email]>:
> > >
> > >> Dmitriy,
> > >>
> > >> That’s about TTL and eviction support for Ignite persistence.
> Presently
> > if
> > >> you set an expiration or eviction policy for a cache it will be
> applied
> > for
> > >> data stored in memory. The policy never affects the persistence layer.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov <[hidden email]>
> > >> wrote:
> > >>>
> > >>> Hi Denis,
> > >>>
> > >>> Is this need covered by PDS + TTL?
> > >>>
> > >>> For the very first TTL test, I found some delay after applying TTL
> with
> > >> the
> > >>> repository enabled:
> https://issues.apache.org/jira/browse/IGNITE-6964
> > >>>
> > >>> And I'm wondering if the user's needs are covered by
> > >>> https://apacheignite.readme.io/docs/expiry-policies plus
> > >>> https://apacheignite.readme.io/docs/distributed-persistent-store
> > >>>
> > >>> Sincerely,
> > >>> Dmitriy Pavlov
> > >>>
> > >>> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov <[hidden email]>:
> > >>>
> > >>>> Hi Denis,
> > >>>>
> > >>>> What is the difference of required by users functionality with TTL
> > cache
> > >>>> expiration?
> > >>>>
> > >>>> By some posts I can suppose TTL cache is compatible with native
> > >>>> persistence.
> > >>>>
> > >>>> Sincerely,
> > >>>> Dmitriy Pavlov
> > >>>>
> > >>>> сб, 18 нояб. 2017 г. в 0:41, Denis Magda <[hidden email]>:
> > >>>>
> > >>>>> Igniters,
> > >>>>>
> > >>>>> I’ve been talking to many Ignite users here and there who are
> already
> > >> on
> > >>>>> Ignite persistence or consider to turn it on. The majority of them
> > are
> > >> more
> > >>>>> than satisfied with its current state and provided capabilities.
> > >> That’s is
> > >>>>> really good news for us.
> > >>>>>
> > >>>>> However, I tend to come across the people who ask about
> > >>>>> eviction/expiration policies for the persistence itself. Had
> around 6
> > >>>>> conversation about the topic this month only.
> > >>>>>
> > >>>>> Usually the requirement is connected with a streaming use case.
> When
> > an
> > >>>>> application streams a lot of data (IoT, metrics, etc.) to the
> cluster
> > >> but
> > >>>>> the data becomes stale in some period of time (day, couple of days,
> > >> etc.).
> > >>>>> The user doesn’t want to waste the disk space and needs to simple
> > >> purge the
> > >>>>> data from there.
> > >>>>>
> > >>>>> My suggestion here is to create a timer task that will remove the
> > stale
> > >>>>> data from the cluster. However, since the demand is growing
> probably
> > >> it’s a
> > >>>>> good time to discuss a feasibility of this feature.
> > >>>>>
> > >>>>> Alex G, as the main architect of the persistence, could you share
> > your
> > >>>>> thoughts on this? What will it cost to us to support
> > >> eviction/expiration
> > >>>>> for the persistence?
> > >>>>>
> > >>>>> —
> > >>>>> Denis
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>

dsetrakyan

Re: Data eviction/expiration from Ignite persistence

On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <[hidden email]> wrote:

> As far as I know there is no plans. Denis please correct me if I'm wrong.
>
> But users found these or similar bugs, it seems we need to support PDS and
> TTL.
>

We should either support it or throw a clear exception on startup clearly
stating that it is not. I suppose, it should be the latter for now.
Dmitriy, any chance you can file a ticket for it?

dmagda

Re: Data eviction/expiration from Ignite persistence

Val,

I'd like to hear Alexey G. opinion on this? Alex, please chime in.

In general, the more deployments the persistence will get the more demand
we will see for that capability. Personally, I'd create a ticket for now
and put it off to our backlog.

--
Denis

On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > As far as I know there is no plans. Denis please correct me if I'm wrong.
> >
> > But users found these or similar bugs, it seems we need to support PDS
> and
> > TTL.
> >
>
> We should either support it or throw a clear exception on startup clearly
> stating that it is not. I suppose, it should be the latter for now.
> Dmitriy, any chance you can file a ticket for it?
>

Alexey Goncharuk

Re: Data eviction/expiration from Ignite persistence

The ticket [1] is in patch available state looks good, the only thing left
there is to transfer old entries to new storage. I hope Andrey will have
time to finish this soon, so we can target the fix for 2.5.

[1] https://issues.apache.org/jira/browse/IGNITE-5874

2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:

> Val,
>
> I'd like to hear Alexey G. opinion on this? Alex, please chime in.
>
> In general, the more deployments the persistence will get the more demand
> we will see for that capability. Personally, I'd create a ticket for now
> and put it off to our backlog.
>
> --
> Denis
>
>
> On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <[hidden email]>
> > wrote:
> >
> > > As far as I know there is no plans. Denis please correct me if I'm
> wrong.
> > >
> > > But users found these or similar bugs, it seems we need to support PDS
> > and
> > > TTL.
> > >
> >
> > We should either support it or throw a clear exception on startup
> clearly
> > stating that it is not. I suppose, it should be the latter for now.
> > Dmitriy, any chance you can file a ticket for it?
> >
>

dmagda

Re: Data eviction/expiration from Ignite persistence

Alex,

Is that ticket about the eviction from the persistence layer? It's not
obvious from the description.

Also, what do you mean by the "transfer of old entries to new storage" (how
is it intended to work and what's a rationale if you're talking about pages
movement from Ignite persistence to another)?

--
Denis

On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
[hidden email]> wrote:

> The ticket [1] is in patch available state looks good, the only thing left
> there is to transfer old entries to new storage. I hope Andrey will have
> time to finish this soon, so we can target the fix for 2.5.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5874
>
> 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
>
> > Val,
> >
> > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> >
> > In general, the more deployments the persistence will get the more demand
> > we will see for that capability. Personally, I'd create a ticket for now
> > and put it off to our backlog.
> >
> > --
> > Denis
> >
> >
> > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <[hidden email]>
> > > wrote:
> > >
> > > > As far as I know there is no plans. Denis please correct me if I'm
> > wrong.
> > > >
> > > > But users found these or similar bugs, it seems we need to support
> PDS
> > > and
> > > > TTL.
> > > >
> > >
> > > We should either support it or throw a clear exception on startup
> > clearly
> > > stating that it is not. I suppose, it should be the latter for now.
> > > Dmitriy, any chance you can file a ticket for it?
> > >
> >
>

Dmitriy Pavlov

Re: Data eviction/expiration from Ignite persistence

Hi Denis,

yes, this is about PDS + entry expiration (TTL).

Please note there is second issue, than is visible by user
https://issues.apache.org/jira/browse/IGNITE-6964

'transfer of old entries to new storage' means DB entries migration from
old format to new. We defenetely should have PDS compatiblity test for this
case.

Sincerely,
Dmitriy Pavlov

вс, 11 мар. 2018 г. в 11:06, Denis Magda <[hidden email]>:

> Alex,
>
> Is that ticket about the eviction from the persistence layer? It's not
> obvious from the description.
>
> Also, what do you mean by the "transfer of old entries to new storage" (how
> is it intended to work and what's a rationale if you're talking about pages
> movement from Ignite persistence to another)?
>
> --
> Denis
>
> On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > The ticket [1] is in patch available state looks good, the only thing
> left
> > there is to transfer old entries to new storage. I hope Andrey will have
> > time to finish this soon, so we can target the fix for 2.5.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> >
> > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> >
> > > Val,
> > >
> > > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> > >
> > > In general, the more deployments the persistence will get the more
> demand
> > > we will see for that capability. Personally, I'd create a ticket for
> now
> > > and put it off to our backlog.
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > As far as I know there is no plans. Denis please correct me if I'm
> > > wrong.
> > > > >
> > > > > But users found these or similar bugs, it seems we need to support
> > PDS
> > > > and
> > > > > TTL.
> > > > >
> > > >
> > > > We should either support it or throw a clear exception on startup
> > > clearly
> > > > stating that it is not. I suppose, it should be the latter for now.
> > > > Dmitriy, any chance you can file a ticket for it?
> > > >
> > >
> >
>

Valentin Kulichenko

Re: Data eviction/expiration from Ignite persistence

In reply to this post by Alexey Goncharuk

Alex,

What is behavior going to be after IGNITE-5874 is fixed? Will expired entry
be removed from both memory and persistence?

-Val

On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
[hidden email]> wrote:

Alexey Goncharuk

Re: Data eviction/expiration from Ignite persistence

Val,

Yes, the entries will be removed from both memory and persistence.

2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
[hidden email]>:

> Alex,
>
> What is behavior going to be after IGNITE-5874 is fixed? Will expired entry
> be removed from both memory and persistence?
>
> -Val
>
> On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > The ticket [1] is in patch available state looks good, the only thing
> left
> > there is to transfer old entries to new storage. I hope Andrey will have
> > time to finish this soon, so we can target the fix for 2.5.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> >
> > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> >
> > > Val,
> > >
> > > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> > >
> > > In general, the more deployments the persistence will get the more
> demand
> > > we will see for that capability. Personally, I'd create a ticket for
> now
> > > and put it off to our backlog.
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > As far as I know there is no plans. Denis please correct me if I'm
> > > wrong.
> > > > >
> > > > > But users found these or similar bugs, it seems we need to support
> > PDS
> > > > and
> > > > > TTL.
> > > > >
> > > >
> > > > We should either support it or throw a clear exception on startup
> > > clearly
> > > > stating that it is not. I suppose, it should be the latter for now.
> > > > Dmitriy, any chance you can file a ticket for it?
> > > >
> > >
> >
>

Denis Magda

Re: Data eviction/expiration from Ignite persistence

Alexey, Dmitriy,

What would be the configuration parameter that defines if the eviction
should happen in the in-memory layer only or in both memory and disk?

--
Denis

On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
[hidden email]> wrote:

> Val,
>
> Yes, the entries will be removed from both memory and persistence.
>
> 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> [hidden email]>:
>
> > Alex,
> >
> > What is behavior going to be after IGNITE-5874 is fixed? Will expired
> entry
> > be removed from both memory and persistence?
> >
> > -Val
> >
> > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > [hidden email]> wrote:
> >
> > > The ticket [1] is in patch available state looks good, the only thing
> > left
> > > there is to transfer old entries to new storage. I hope Andrey will
> have
> > > time to finish this soon, so we can target the fix for 2.5.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > >
> > > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> > >
> > > > Val,
> > > >
> > > > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> > > >
> > > > In general, the more deployments the persistence will get the more
> > demand
> > > > we will see for that capability. Personally, I'd create a ticket for
> > now
> > > > and put it off to our backlog.
> > > >
> > > > --
> > > > Denis
> > > >
> > > >
> > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > As far as I know there is no plans. Denis please correct me if
> I'm
> > > > wrong.
> > > > > >
> > > > > > But users found these or similar bugs, it seems we need to
> support
> > > PDS
> > > > > and
> > > > > > TTL.
> > > > > >
> > > > >
> > > > > We should either support it or throw a clear exception on startup
> > > > clearly
> > > > > stating that it is not. I suppose, it should be the latter for now.
> > > > > Dmitriy, any chance you can file a ticket for it?
> > > > >
> > > >
> > >
> >
>

Dmitriy Pavlov

Re: Data eviction/expiration from Ignite persistence

Denis,

I suppose there is no configuration will be required. If TTL configured
entry will be removed from disk & memory both.

SIncerely,
Dmitriy Pavlov

пн, 12 мар. 2018 г. в 23:32, Denis Magda <[hidden email]>:

> Alexey, Dmitriy,
>
> What would be the configuration parameter that defines if the eviction
> should happen in the in-memory layer only or in both memory and disk?
>
> --
> Denis
>
> On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > Val,
> >
> > Yes, the entries will be removed from both memory and persistence.
> >
> > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > [hidden email]>:
> >
> > > Alex,
> > >
> > > What is behavior going to be after IGNITE-5874 is fixed? Will expired
> > entry
> > > be removed from both memory and persistence?
> > >
> > > -Val
> > >
> > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > [hidden email]> wrote:
> > >
> > > > The ticket [1] is in patch available state looks good, the only thing
> > > left
> > > > there is to transfer old entries to new storage. I hope Andrey will
> > have
> > > > time to finish this soon, so we can target the fix for 2.5.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > >
> > > > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> > > >
> > > > > Val,
> > > > >
> > > > > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> > > > >
> > > > > In general, the more deployments the persistence will get the more
> > > demand
> > > > > we will see for that capability. Personally, I'd create a ticket
> for
> > > now
> > > > > and put it off to our backlog.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > >
> > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > As far as I know there is no plans. Denis please correct me if
> > I'm
> > > > > wrong.
> > > > > > >
> > > > > > > But users found these or similar bugs, it seems we need to
> > support
> > > > PDS
> > > > > > and
> > > > > > > TTL.
> > > > > > >
> > > > > >
> > > > > > We should either support it or throw a clear exception on
> startup
> > > > > clearly
> > > > > > stating that it is not. I suppose, it should be the latter for
> now.
> > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > >
> > > > >
> > > >
> > >
> >
>

dmagda

Re: Data eviction/expiration from Ignite persistence

Dmitriy,

It will break the current default behavior when data is evicted from the
memory only, and I would disagree that it's a right decision overall.

There are many scenarios when users need to have the eviction in the memory
layer and preserve data on disk for later usage. So, can we keep what we
have now and merely expand the eviction to disk if the user requires it?

--
Denis

On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov <[hidden email]>
wrote:

> Denis,
>
> I suppose there is no configuration will be required. If TTL configured
> entry will be removed from disk & memory both.
>
> SIncerely,
> Dmitriy Pavlov
>
> пн, 12 мар. 2018 г. в 23:32, Denis Magda <[hidden email]>:
>
> > Alexey, Dmitriy,
> >
> > What would be the configuration parameter that defines if the eviction
> > should happen in the in-memory layer only or in both memory and disk?
> >
> > --
> > Denis
> >
> > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > [hidden email]> wrote:
> >
> > > Val,
> > >
> > > Yes, the entries will be removed from both memory and persistence.
> > >
> > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > [hidden email]>:
> > >
> > > > Alex,
> > > >
> > > > What is behavior going to be after IGNITE-5874 is fixed? Will expired
> > > entry
> > > > be removed from both memory and persistence?
> > > >
> > > > -Val
> > > >
> > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > [hidden email]> wrote:
> > > >
> > > > > The ticket [1] is in patch available state looks good, the only
> thing
> > > > left
> > > > > there is to transfer old entries to new storage. I hope Andrey will
> > > have
> > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > >
> > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > I'd like to hear Alexey G. opinion on this? Alex, please chime
> in.
> > > > > >
> > > > > > In general, the more deployments the persistence will get the
> more
> > > > demand
> > > > > > we will see for that capability. Personally, I'd create a ticket
> > for
> > > > now
> > > > > > and put it off to our backlog.
> > > > > >
> > > > > > --
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > As far as I know there is no plans. Denis please correct me
> if
> > > I'm
> > > > > > wrong.
> > > > > > > >
> > > > > > > > But users found these or similar bugs, it seems we need to
> > > support
> > > > > PDS
> > > > > > > and
> > > > > > > > TTL.
> > > > > > > >
> > > > > > >
> > > > > > > We should either support it or throw a clear exception on
> > startup
> > > > > > clearly
> > > > > > > stating that it is not. I suppose, it should be the latter for
> > now.
> > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

dsetrakyan

Re: Data eviction/expiration from Ignite persistence

To be honest, I am not clear how this would be implemented. Currently, we
keep TTL map in memory and it fits in memory. However, if we start tracking
entries on disk, the TTL map will grow too large and may not fit in memory
any longer.

Can someone explain how this will be handled?

D.

On Mon, Mar 12, 2018 at 2:03 PM, Denis Magda <[hidden email]> wrote:

> Dmitriy,
>
> It will break the current default behavior when data is evicted from the
> memory only, and I would disagree that it's a right decision overall.
>
> There are many scenarios when users need to have the eviction in the memory
> layer and preserve data on disk for later usage. So, can we keep what we
> have now and merely expand the eviction to disk if the user requires it?
>
> --
> Denis
>
>
> On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > Denis,
> >
> > I suppose there is no configuration will be required. If TTL configured
> > entry will be removed from disk & memory both.
> >
> > SIncerely,
> > Dmitriy Pavlov
> >
> > пн, 12 мар. 2018 г. в 23:32, Denis Magda <[hidden email]>:
> >
> > > Alexey, Dmitriy,
> > >
> > > What would be the configuration parameter that defines if the eviction
> > > should happen in the in-memory layer only or in both memory and disk?
> > >
> > > --
> > > Denis
> > >
> > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > [hidden email]> wrote:
> > >
> > > > Val,
> > > >
> > > > Yes, the entries will be removed from both memory and persistence.
> > > >
> > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > [hidden email]>:
> > > >
> > > > > Alex,
> > > > >
> > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> expired
> > > > entry
> > > > > be removed from both memory and persistence?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > [hidden email]> wrote:
> > > > >
> > > > > > The ticket [1] is in patch available state looks good, the only
> > thing
> > > > > left
> > > > > > there is to transfer old entries to new storage. I hope Andrey
> will
> > > > have
> > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > >
> > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda <[hidden email]>:
> > > > > >
> > > > > > > Val,
> > > > > > >
> > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please chime
> > in.
> > > > > > >
> > > > > > > In general, the more deployments the persistence will get the
> > more
> > > > > demand
> > > > > > > we will see for that capability. Personally, I'd create a
> ticket
> > > for
> > > > > now
> > > > > > > and put it off to our backlog.
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > [hidden email]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > As far as I know there is no plans. Denis please correct me
> > if
> > > > I'm
> > > > > > > wrong.
> > > > > > > > >
> > > > > > > > > But users found these or similar bugs, it seems we need to
> > > > support
> > > > > > PDS
> > > > > > > > and
> > > > > > > > > TTL.
> > > > > > > > >
> > > > > > > >
> > > > > > > > We should either support it or throw a clear exception on
> > > startup
> > > > > > > clearly
> > > > > > > > stating that it is not. I suppose, it should be the latter
> for
> > > now.
> > > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Alexey Goncharuk

Re: Data eviction/expiration from Ignite persistence

In reply to this post by dmagda

Denis,

What do you mean by 'current behavior when data is evicted from the memory
only'? TTL expiration effectively means that the corresponding key-value
pairs are destroyed. If you are talking about page replacement, then there
is no way to do this on per-key basis because a page must be replaced as a
whole and it makes no sense to track keys.

--AG

2018-03-13 0:03 GMT+03:00 Denis Magda <[hidden email]>: