Memory policies examples fails

[dmagda]

Igniters,

I’ve created a simple example to demonstrate how to set up various policies:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/MemoryPoliciesExample.java

However, this tiny example fails with

Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException: null
        at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:255)
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.createCacheDataStore0(IgniteCacheOffheapManagerImpl.java:727)
        at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.createCacheDataStore(IgniteCacheOffheapManagerImpl.java:711)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:162)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitionTopologyImpl.java:718)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitionTopologyImpl.java:405)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.java:569)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:844)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:573)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1800)
        ... 2 more



What I’m doing wrong? Alex G., could you take a look? Probably I’m missing something due to a lack of documentation.

—
Denis

[Sergey Chugunov]

Denis,

It looks like partition exchange mechanism allocates a lot of pages right
after the cache is started.
I'm trying to figure out what's going on.

Thanks,
Sergey.

On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda <[hidden email]> wrote:

> Igniters,
>
> I’ve created a simple example to demonstrate how to set up various
> policies:
> https://github.com/apache/ignite/blob/master/examples/
> src/main/java/org/apache/ignite/examples/datagrid/
> MemoryPoliciesExample.java
>
> However, this tiny example fails with
>
> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException: null
>         at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.
> allocatePage(PageMemoryNoStoreImpl.java:255)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.allocateForTree(
> IgniteCacheOffheapManagerImpl.java:635)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.createCacheDataStore0(
> IgniteCacheOffheapManagerImpl.java:727)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.createCacheDataStore(
> IgniteCacheOffheapManagerImpl.java:711)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:162)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitionTopologyImpl.
> java:718)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitionTopologyImpl.
> java:405)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.
> java:569)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(
> GridDhtPartitionsExchangeFuture.java:844)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.preloader.GridDhtPartitionsExchangeFuture.init(
> GridDhtPartitionsExchangeFuture.java:573)
>         at org.apache.ignite.internal.processors.cache.
> GridCachePartitionExchangeManager$ExchangeWorker.body(
> GridCachePartitionExchangeManager.java:1800)
>         ... 2 more
>
>
>
> What I’m doing wrong? Alex G., could you take a look? Probably I’m missing
> something due to a lack of documentation.
>
> —
> Denis
>
>

[Sergey Chugunov]

Denis,

I found an issue with the example: each cache introduces some memory
overhead (mostly for cache partitions data structures) and it must be
counted in MemoryPolicy size.

To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
and *5MB_Region_Swapping *to 10 MB.

I didn't find any documentation on this, but I think it is part of new
PageMemory-based storage so it needs to be documented as part of ignite-2.0
release.

Thanks,
Sergey.



On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov <[hidden email]>
wrote:

> Denis,
>
> It looks like partition exchange mechanism allocates a lot of pages right
> after the cache is started.
> I'm trying to figure out what's going on.
>
> Thanks,
> Sergey.
>
> On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda <[hidden email]> wrote:
>
>> Igniters,
>>
>> I’ve created a simple example to demonstrate how to set up various
>> policies:
>> https://github.com/apache/ignite/blob/master/examples/src/
>> main/java/org/apache/ignite/examples/datagrid/MemoryPoliciesExample.java
>>
>> However, this tiny example fails with
>>
>> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
>> null
>>         at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImp
>> l.allocatePage(PageMemoryNoStoreImpl.java:255)
>>         at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
>>         at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
>> rImpl.java:727)
>>         at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
>> Impl.java:711)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:162)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
>> nTopologyImpl.java:718)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
>> nTopologyImpl.java:405)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
>> TopologyImpl.java:569)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPar
>> titionsExchangeFuture.java:844)
>>         at org.apache.ignite.internal.processors.cache.distributed.dht.
>> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
>> ionsExchangeFuture.java:573)
>>         at org.apache.ignite.internal.processors.cache.GridCachePartiti
>> onExchangeManager$ExchangeWorker.body(GridCacheP
>> artitionExchangeManager.java:1800)
>>         ... 2 more
>>
>>
>>
>> What I’m doing wrong? Alex G., could you take a look? Probably I’m
>> missing something due to a lack of documentation.
>>
>> —
>> Denis
>>
>>
>

[dsetrakyan]

Wow.... If it happened to Denis, it will happen to many other users as
well. With the exception error message saying "Null", I doubt any user
would figure out what to do.

Can we fix the error message?

D.


On Mon, Apr 17, 2017 at 6:03 AM, Sergey Chugunov <[hidden email]>
wrote:

> Denis,
>
> I found an issue with the example: each cache introduces some memory
> overhead (mostly for cache partitions data structures) and it must be
> counted in MemoryPolicy size.
>
> To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
> and *5MB_Region_Swapping *to 10 MB.
>
> I didn't find any documentation on this, but I think it is part of new
> PageMemory-based storage so it needs to be documented as part of ignite-2.0
> release.
>
> Thanks,
> Sergey.
>
>
>
> On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov <
> [hidden email]>
> wrote:
>
> > Denis,
> >
> > It looks like partition exchange mechanism allocates a lot of pages right
> > after the cache is started.
> > I'm trying to figure out what's going on.
> >
> > Thanks,
> > Sergey.
> >
> > On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda <[hidden email]> wrote:
> >
> >> Igniters,
> >>
> >> I’ve created a simple example to demonstrate how to set up various
> >> policies:
> >> https://github.com/apache/ignite/blob/master/examples/src/
> >> main/java/org/apache/ignite/examples/datagrid/MemoryPolicies
> Example.java
> >>
> >> However, this tiny example fails with
> >>
> >> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
> >> null
> >>         at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImp
> >> l.allocatePage(PageMemoryNoStoreImpl.java:255)
> >>         at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
> >>         at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
> >> rImpl.java:727)
> >>         at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
> >> Impl.java:711)
> >>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:162)
> >>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
> >> nTopologyImpl.java:718)
> >>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
> >> nTopologyImpl.java:405)
> >>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
> >> TopologyImpl.java:569)
> >>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> preloader.GridDhtPartitionsExchangeFuture.
> distributedExchange(GridDhtPar
> >> titionsExchangeFuture.java:844)
> >>         at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
> >> ionsExchangeFuture.java:573)
> >>         at org.apache.ignite.internal.processors.cache.GridCachePartiti
> >> onExchangeManager$ExchangeWorker.body(GridCacheP
> >> artitionExchangeManager.java:1800)
> >>         ... 2 more
> >>
> >>
> >>
> >> What I’m doing wrong? Alex G., could you take a look? Probably I’m
> >> missing something due to a lack of documentation.
> >>
> >> —
> >> Denis
> >>
> >>
> >
>

[Sergey Chugunov]

Dmitriy,

I think this issue highlights another lack in our documentation that
emerged with introduction of PageMemory.

In current documentation we have some recommendations about memory capacity
planning.
As PageMemory storage data structures introduce their own overhead; this
section of documentation must be updated.

As per "null" in OutOfMemoryException message, I've already addressed it so
now when PageMemory fails to allocate new page, it throws an
IgniteOutOfMemoryException with message like "Failed to allocate new page
within <MemoryPolicyName> MemoryPolicy".


Thanks,
Sergey.

On Mon, Apr 17, 2017 at 6:52 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Wow.... If it happened to Denis, it will happen to many other users as
> well. With the exception error message saying "Null", I doubt any user
> would figure out what to do.
>
> Can we fix the error message?
>
> D.
>
>
> On Mon, Apr 17, 2017 at 6:03 AM, Sergey Chugunov <
> [hidden email]>
> wrote:
>
> > Denis,
> >
> > I found an issue with the example: each cache introduces some memory
> > overhead (mostly for cache partitions data structures) and it must be
> > counted in MemoryPolicy size.
> >
> > To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
> > and *5MB_Region_Swapping *to 10 MB.
> >
> > I didn't find any documentation on this, but I think it is part of new
> > PageMemory-based storage so it needs to be documented as part of
> ignite-2.0
> > release.
> >
> > Thanks,
> > Sergey.
> >
> >
> >
> > On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov <
> > [hidden email]>
> > wrote:
> >
> > > Denis,
> > >
> > > It looks like partition exchange mechanism allocates a lot of pages
> right
> > > after the cache is started.
> > > I'm trying to figure out what's going on.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > > On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda <[hidden email]>
> wrote:
> > >
> > >> Igniters,
> > >>
> > >> I’ve created a simple example to demonstrate how to set up various
> > >> policies:
> > >> https://github.com/apache/ignite/blob/master/examples/src/
> > >> main/java/org/apache/ignite/examples/datagrid/MemoryPolicies
> > Example.java
> > >>
> > >> However, this tiny example fails with
> > >>
> > >> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
> > >> null
> > >>         at org.apache.ignite.internal.pagemem.impl.
> PageMemoryNoStoreImp
> > >> l.allocatePage(PageMemoryNoStoreImpl.java:255)
> > >>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
> > >>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
> > >> rImpl.java:727)
> > >>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
> > >> Impl.java:711)
> > >>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:162)
> > >>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
> > >> nTopologyImpl.java:718)
> > >>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
> > >> nTopologyImpl.java:405)
> > >>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
> > >> TopologyImpl.java:569)
> > >>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> preloader.GridDhtPartitionsExchangeFuture.
> > distributedExchange(GridDhtPar
> > >> titionsExchangeFuture.java:844)
> > >>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
> > >> ionsExchangeFuture.java:573)
> > >>         at org.apache.ignite.internal.processors.cache.
> GridCachePartiti
> > >> onExchangeManager$ExchangeWorker.body(GridCacheP
> > >> artitionExchangeManager.java:1800)
> > >>         ... 2 more
> > >>
> > >>
> > >>
> > >> What I’m doing wrong? Alex G., could you take a look? Probably I’m
> > >> missing something due to a lack of documentation.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>
> > >
> >
>

[dsetrakyan]

On Mon, Apr 17, 2017 at 9:06 AM, Sergey Chugunov <[hidden email]>
wrote:

> Dmitriy,
>
> I think this issue highlights another lack in our documentation that
> emerged with introduction of PageMemory.
>
> In current documentation we have some recommendations about memory capacity
> planning.
> As PageMemory storage data structures introduce their own overhead; this
> section of documentation must be updated.
>
> As per "null" in OutOfMemoryException message, I've already addressed it so
> now when PageMemory fails to allocate new page, it throws an
> IgniteOutOfMemoryException with message like "Failed to allocate new page
> within <MemoryPolicyName> MemoryPolicy".
>

Sergey, the error message should also mention which configuration property
needs to be changed to fix the issue.

[dmagda]

Sergey,

I tweaked the example following your suggestion. Now it works, thanks.

In addition, a javadoc of MemoryConfiguraiton, MemoryPolicyConfiguration and DataPageEvictionMode has been significantly improved. There are some of the parameters like page memory’s concurrency mode and policy’s "empty page pool size" that are not obvious. Marked all the blur places in those configurations with TODOs and created a ticket [1]. * Alex G *, as an architect of the page memory, please look at [1] and try to close it tomorrow so that I can complete the readme part.

Finally, as for the out of memory exception I fully support Dmitriy’s opinion - let’s be as more concrete as possible. "Failed to allocate new page within <MemoryPolicyName> MemoryPolicy” sounds to generic and doesn’t help me anyhow. Let’s add to the message what was a root cause (policy size, name, etc.) so that the user can fix it immediately. Created a ticket [2].

[1] https://issues.apache.org/jira/browse/IGNITE-5007 <https://issues.apache.org/jira/browse/IGNITE-5007>
[2] https://issues.apache.org/jira/browse/IGNITE-5008

—
Denis

> On Apr 17, 2017, at 2:18 PM, Dmitriy Setrakyan <[hidden email]> wrote:
>
> On Mon, Apr 17, 2017 at 9:06 AM, Sergey Chugunov <[hidden email]>
> wrote:
>
>> Dmitriy,
>>
>> I think this issue highlights another lack in our documentation that
>> emerged with introduction of PageMemory.
>>
>> In current documentation we have some recommendations about memory capacity
>> planning.
>> As PageMemory storage data structures introduce their own overhead; this
>> section of documentation must be updated.
>>
>> As per "null" in OutOfMemoryException message, I've already addressed it so
>> now when PageMemory fails to allocate new page, it throws an
>> IgniteOutOfMemoryException with message like "Failed to allocate new page
>> within <MemoryPolicyName> MemoryPolicy".
>>
>
> Sergey, the error message should also mention which configuration property
> needs to be changed to fix the issue.