Enabling swap space and Ignite Persistence

[Prachi Garg]

Engineers,

How does persistence and swap work when both are enabled? I was under the
impression that for a data region you can either have swap or persistence
configured at a time, but not both. Please clarify.

Thanks,
-Prachi

[Ivan Rakov]

Prachi,

Swap is legacy lightweight version of Ignite Native Persistence. In swap
mode, we fully rely on OS in storing offheap memory into memory-mapped
file. We don't provide durability guarantees in this mode. From my point
of view, after 2.1 release there's no reason to prefer swap mode over
Ignite Native Persistence.
Igniters, please correct me if there are still any actual cases.

Right now we indeed can configure both Ignite Native Persistence and
swapping, but this makes even less sense. Node will just perform extra
job by persisting data twice.

Best Regards,
Ivan Rakov

On 02.03.2018 7:20, Prachi Garg wrote:
> Engineers,
>
> How does persistence and swap work when both are enabled? I was under the
> impression that for a data region you can either have swap or persistence
> configured at a time, but not both. Please clarify.
>
> Thanks,
> -Prachi
>

[dmagda]

Hi Ivan,


> Swap is legacy lightweight version of Ignite Native Persistence. In swap
> mode, we fully rely on OS in storing offheap memory into memory-mapped
> file. We don't provide durability guarantees in this mode. From my point of
> view, after 2.1 release there's no reason to prefer swap mode over Ignite
> Native Persistence.
> Igniters, please correct me if there are still any actual cases.


There is a business use case for the swap space I've come across with
recently. Some applications want to store data entirely in RAM avoiding any
persistence in general (Ignite persistence or 3rd party DB). It's ok for
them to lose a data set in case of a cluster shutdown. However, they want
to avoid OOM exception that might happen if they don't scale out the
cluster in time. And here is the swap space comes to rescue. If a node is
running out of RAM, the OS begins the swap-out/in the process putting off
OOM and DevOps will have much more time to scale the cluster and rebalance
the data.

Right now we indeed can configure both Ignite Native Persistence and
> swapping, but this makes even less sense. Node will just perform extra job
> by persisting data twice.


Guess, that's the point Prachi tried to point out. Could we throw an
exception if a user tries to configure both? As we agreed, it's error-prone
and doesn't make sense in general.

--
Denis

On Fri, Mar 2, 2018 at 1:28 AM, Ivan Rakov <[hidden email]> wrote:

> Prachi,
>
> Swap is legacy lightweight version of Ignite Native Persistence. In swap
> mode, we fully rely on OS in storing offheap memory into memory-mapped
> file. We don't provide durability guarantees in this mode. From my point of
> view, after 2.1 release there's no reason to prefer swap mode over Ignite
> Native Persistence.
> Igniters, please correct me if there are still any actual cases.
>
> Right now we indeed can configure both Ignite Native Persistence and
> swapping, but this makes even less sense. Node will just perform extra job
> by persisting data twice.
>
> Best Regards,
> Ivan Rakov
>
>
> On 02.03.2018 7:20, Prachi Garg wrote:
>
>> Engineers,
>>
>> How does persistence and swap work when both are enabled? I was under the
>> impression that for a data region you can either have swap or persistence
>> configured at a time, but not both. Please clarify.
>>
>> Thanks,
>> -Prachi
>>
>>
>

[Valentin Kulichenko]

I stumbled across couple of use cases where swap space was more suitable
than persistence. However, enabling both for a same region definitely
doesn't make sense to me, I would throw an exception in this case.

-Val

On Fri, Mar 2, 2018 at 9:46 AM, Denis Magda <[hidden email]> wrote:

> Hi Ivan,
>
>
> > Swap is legacy lightweight version of Ignite Native Persistence. In swap
> > mode, we fully rely on OS in storing offheap memory into memory-mapped
> > file. We don't provide durability guarantees in this mode. From my point
> of
> > view, after 2.1 release there's no reason to prefer swap mode over Ignite
> > Native Persistence.
> > Igniters, please correct me if there are still any actual cases.
>
>
> There is a business use case for the swap space I've come across with
> recently. Some applications want to store data entirely in RAM avoiding any
> persistence in general (Ignite persistence or 3rd party DB). It's ok for
> them to lose a data set in case of a cluster shutdown. However, they want
> to avoid OOM exception that might happen if they don't scale out the
> cluster in time. And here is the swap space comes to rescue. If a node is
> running out of RAM, the OS begins the swap-out/in the process putting off
> OOM and DevOps will have much more time to scale the cluster and rebalance
> the data.
>
> Right now we indeed can configure both Ignite Native Persistence and
> > swapping, but this makes even less sense. Node will just perform extra
> job
> > by persisting data twice.
>
>
> Guess, that's the point Prachi tried to point out. Could we throw an
> exception if a user tries to configure both? As we agreed, it's error-prone
> and doesn't make sense in general.
>
> --
> Denis
>
> On Fri, Mar 2, 2018 at 1:28 AM, Ivan Rakov <[hidden email]> wrote:
>
> > Prachi,
> >
> > Swap is legacy lightweight version of Ignite Native Persistence. In swap
> > mode, we fully rely on OS in storing offheap memory into memory-mapped
> > file. We don't provide durability guarantees in this mode. From my point
> of
> > view, after 2.1 release there's no reason to prefer swap mode over Ignite
> > Native Persistence.
> > Igniters, please correct me if there are still any actual cases.
> >
> > Right now we indeed can configure both Ignite Native Persistence and
> > swapping, but this makes even less sense. Node will just perform extra
> job
> > by persisting data twice.
> >
> > Best Regards,
> > Ivan Rakov
> >
> >
> > On 02.03.2018 7:20, Prachi Garg wrote:
> >
> >> Engineers,
> >>
> >> How does persistence and swap work when both are enabled? I was under
> the
> >> impression that for a data region you can either have swap or
> persistence
> >> configured at a time, but not both. Please clarify.
> >>
> >> Thanks,
> >> -Prachi
> >>
> >>
> >
>

[Prachi Garg]

Created JIRA ticket for this issue -
https://issues.apache.org/jira/browse/IGNITE-7902

On Fri, Mar 2, 2018 at 10:34 AM, Valentin Kulichenko <
[hidden email]> wrote:

> I stumbled across couple of use cases where swap space was more suitable
> than persistence. However, enabling both for a same region definitely
> doesn't make sense to me, I would throw an exception in this case.
>
> -Val
>
> On Fri, Mar 2, 2018 at 9:46 AM, Denis Magda <[hidden email]> wrote:
>
> > Hi Ivan,
> >
> >
> > > Swap is legacy lightweight version of Ignite Native Persistence. In
> swap
> > > mode, we fully rely on OS in storing offheap memory into memory-mapped
> > > file. We don't provide durability guarantees in this mode. From my
> point
> > of
> > > view, after 2.1 release there's no reason to prefer swap mode over
> Ignite
> > > Native Persistence.
> > > Igniters, please correct me if there are still any actual cases.
> >
> >
> > There is a business use case for the swap space I've come across with
> > recently. Some applications want to store data entirely in RAM avoiding
> any
> > persistence in general (Ignite persistence or 3rd party DB). It's ok for
> > them to lose a data set in case of a cluster shutdown. However, they want
> > to avoid OOM exception that might happen if they don't scale out the
> > cluster in time. And here is the swap space comes to rescue. If a node is
> > running out of RAM, the OS begins the swap-out/in the process putting off
> > OOM and DevOps will have much more time to scale the cluster and
> rebalance
> > the data.
> >
> > Right now we indeed can configure both Ignite Native Persistence and
> > > swapping, but this makes even less sense. Node will just perform extra
> > job
> > > by persisting data twice.
> >
> >
> > Guess, that's the point Prachi tried to point out. Could we throw an
> > exception if a user tries to configure both? As we agreed, it's
> error-prone
> > and doesn't make sense in general.
> >
> > --
> > Denis
> >
> > On Fri, Mar 2, 2018 at 1:28 AM, Ivan Rakov <[hidden email]>
> wrote:
> >
> > > Prachi,
> > >
> > > Swap is legacy lightweight version of Ignite Native Persistence. In
> swap
> > > mode, we fully rely on OS in storing offheap memory into memory-mapped
> > > file. We don't provide durability guarantees in this mode. From my
> point
> > of
> > > view, after 2.1 release there's no reason to prefer swap mode over
> Ignite
> > > Native Persistence.
> > > Igniters, please correct me if there are still any actual cases.
> > >
> > > Right now we indeed can configure both Ignite Native Persistence and
> > > swapping, but this makes even less sense. Node will just perform extra
> > job
> > > by persisting data twice.
> > >
> > > Best Regards,
> > > Ivan Rakov
> > >
> > >
> > > On 02.03.2018 7:20, Prachi Garg wrote:
> > >
> > >> Engineers,
> > >>
> > >> How does persistence and swap work when both are enabled? I was under
> > the
> > >> impression that for a data region you can either have swap or
> > persistence
> > >> configured at a time, but not both. Please clarify.
> > >>
> > >> Thanks,
> > >> -Prachi
> > >>
> > >>
> > >
> >
>