Page Memory behavior with default memory policy

[Sergi]

Guys,

If we have a default of 80% of available memory then just starting few
nodes on my laptop will make it hang. This idea does not work until we have
a dynamically expandable memory pools.

Sergi

2017-04-19 22:20 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Sergey,
>
> I have responded in the ticket. Can you please provide the current and the
> proposed configuration examples?
>
> D.
>
> On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> [hidden email]>
> wrote:
>
> > Guys,
> >
> > I created a ticket to implement these improvements, please take a look:
> > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> >
> > Besides employing the idea of allocation 80% of physical memory I'm also
> > suggesting to introduce one more configuration property to specify
> default
> > MemoryPolicy's size in bytes without having to use verbose syntax of
> > *memoryPolicyConfiguration
> > *section.
> >
> > Any thoughts on this?
> >
> > Thanks,
> > Sergey.
> >
> >
> >
> > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > wrote:
> >
> > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > [hidden email]> wrote:
> > >
> > > > I don't see why not, this is the way our tests are currently running.
> > > > Anyways, we can think about efficient dynamic memory expansion in
> 2.1,
> > > this
> > > > may be feasible if we free up some space in PageId to encode segment
> > > > number. There is a ticket for this:
> > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > >
> > >
> > > Alexey, if the operating system is already handling this for us, I
> don't
> > > see a reason to do it manually.
> > >
> > > I also like what Denis and Semyon are proposing. However, I would not
> > grab
> > > the full free memory. How about 80% of the free memory?
> > >
> > > D.
> > >
> >
>

[Alexey Goncharuk]

This actually will affect only nodes started in a single jvm, but I agree
that expandable memory provides better user experience. Let's try to
squeeze this change to 2.0.

2017-04-20 12:32 GMT+03:00 Sergi Vladykin <[hidden email]>:

> Guys,
>
> If we have a default of 80% of available memory then just starting few
> nodes on my laptop will make it hang. This idea does not work until we have
> a dynamically expandable memory pools.
>
> Sergi
>
> 2017-04-19 22:20 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Sergey,
> >
> > I have responded in the ticket. Can you please provide the current and
> the
> > proposed configuration examples?
> >
> > D.
> >
> > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > [hidden email]>
> > wrote:
> >
> > > Guys,
> > >
> > > I created a ticket to implement these improvements, please take a look:
> > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > >
> > > Besides employing the idea of allocation 80% of physical memory I'm
> also
> > > suggesting to introduce one more configuration property to specify
> > default
> > > MemoryPolicy's size in bytes without having to use verbose syntax of
> > > *memoryPolicyConfiguration
> > > *section.
> > >
> > > Any thoughts on this?
> > >
> > > Thanks,
> > > Sergey.
> > >
> > >
> > >
> > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > [hidden email]> wrote:
> > > >
> > > > > I don't see why not, this is the way our tests are currently
> running.
> > > > > Anyways, we can think about efficient dynamic memory expansion in
> > 2.1,
> > > > this
> > > > > may be feasible if we free up some space in PageId to encode
> segment
> > > > > number. There is a ticket for this:
> > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > >
> > > >
> > > > Alexey, if the operating system is already handling this for us, I
> > don't
> > > > see a reason to do it manually.
> > > >
> > > > I also like what Denis and Semyon are proposing. However, I would not
> > > grab
> > > > the full free memory. How about 80% of the free memory?
> > > >
> > > > D.
> > > >
> > >
> >
>

[Vladimir Ozerov]

Guys,

Do we have a ticket for this change?

On Thu, Apr 20, 2017 at 1:13 PM, Alexey Goncharuk <
[hidden email]> wrote:

> This actually will affect only nodes started in a single jvm, but I agree
> that expandable memory provides better user experience. Let's try to
> squeeze this change to 2.0.
>
> 2017-04-20 12:32 GMT+03:00 Sergi Vladykin <[hidden email]>:
>
> > Guys,
> >
> > If we have a default of 80% of available memory then just starting few
> > nodes on my laptop will make it hang. This idea does not work until we
> have
> > a dynamically expandable memory pools.
> >
> > Sergi
> >
> > 2017-04-19 22:20 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > Sergey,
> > >
> > > I have responded in the ticket. Can you please provide the current and
> > the
> > > proposed configuration examples?
> > >
> > > D.
> > >
> > > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Guys,
> > > >
> > > > I created a ticket to implement these improvements, please take a
> look:
> > > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > > >
> > > > Besides employing the idea of allocation 80% of physical memory I'm
> > also
> > > > suggesting to introduce one more configuration property to specify
> > > default
> > > > MemoryPolicy's size in bytes without having to use verbose syntax of
> > > > *memoryPolicyConfiguration
> > > > *section.
> > > >
> > > > Any thoughts on this?
> > > >
> > > > Thanks,
> > > > Sergey.
> > > >
> > > >
> > > >
> > > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > > [hidden email]> wrote:
> > > > >
> > > > > > I don't see why not, this is the way our tests are currently
> > running.
> > > > > > Anyways, we can think about efficient dynamic memory expansion in
> > > 2.1,
> > > > > this
> > > > > > may be feasible if we free up some space in PageId to encode
> > segment
> > > > > > number. There is a ticket for this:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > > >
> > > > >
> > > > > Alexey, if the operating system is already handling this for us, I
> > > don't
> > > > > see a reason to do it manually.
> > > > >
> > > > > I also like what Denis and Semyon are proposing. However, I would
> not
> > > > grab
> > > > > the full free memory. How about 80% of the free memory?
> > > > >
> > > > > D.
> > > > >
> > > >
> > >
> >
>

[Sergey Chugunov]

yes, we do: IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>

On Thu, Apr 20, 2017 at 1:23 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Guys,
>
> Do we have a ticket for this change?
>
> On Thu, Apr 20, 2017 at 1:13 PM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > This actually will affect only nodes started in a single jvm, but I agree
> > that expandable memory provides better user experience. Let's try to
> > squeeze this change to 2.0.
> >
> > 2017-04-20 12:32 GMT+03:00 Sergi Vladykin <[hidden email]>:
> >
> > > Guys,
> > >
> > > If we have a default of 80% of available memory then just starting few
> > > nodes on my laptop will make it hang. This idea does not work until we
> > have
> > > a dynamically expandable memory pools.
> > >
> > > Sergi
> > >
> > > 2017-04-19 22:20 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> > >
> > > > Sergey,
> > > >
> > > > I have responded in the ticket. Can you please provide the current
> and
> > > the
> > > > proposed configuration examples?
> > > >
> > > > D.
> > > >
> > > > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > I created a ticket to implement these improvements, please take a
> > look:
> > > > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > > > >
> > > > > Besides employing the idea of allocation 80% of physical memory I'm
> > > also
> > > > > suggesting to introduce one more configuration property to specify
> > > > default
> > > > > MemoryPolicy's size in bytes without having to use verbose syntax
> of
> > > > > *memoryPolicyConfiguration
> > > > > *section.
> > > > >
> > > > > Any thoughts on this?
> > > > >
> > > > > Thanks,
> > > > > Sergey.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > > > [hidden email]> wrote:
> > > > > >
> > > > > > > I don't see why not, this is the way our tests are currently
> > > running.
> > > > > > > Anyways, we can think about efficient dynamic memory expansion
> in
> > > > 2.1,
> > > > > > this
> > > > > > > may be feasible if we free up some space in PageId to encode
> > > segment
> > > > > > > number. There is a ticket for this:
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > > > >
> > > > > >
> > > > > > Alexey, if the operating system is already handling this for us,
> I
> > > > don't
> > > > > > see a reason to do it manually.
> > > > > >
> > > > > > I also like what Denis and Semyon are proposing. However, I would
> > not
> > > > > grab
> > > > > > the full free memory. How about 80% of the free memory?
> > > > > >
> > > > > > D.
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Sergey Chugunov]

Dmitriy,

Replied in the ticket.

Thanks,
Sergey.

On Wed, Apr 19, 2017 at 10:20 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Sergey,
>
> I have responded in the ticket. Can you please provide the current and the
> proposed configuration examples?
>
> D.
>
> On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> [hidden email]>
> wrote:
>
> > Guys,
> >
> > I created a ticket to implement these improvements, please take a look:
> > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> >
> > Besides employing the idea of allocation 80% of physical memory I'm also
> > suggesting to introduce one more configuration property to specify
> default
> > MemoryPolicy's size in bytes without having to use verbose syntax of
> > *memoryPolicyConfiguration
> > *section.
> >
> > Any thoughts on this?
> >
> > Thanks,
> > Sergey.
> >
> >
> >
> > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > wrote:
> >
> > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > [hidden email]> wrote:
> > >
> > > > I don't see why not, this is the way our tests are currently running.
> > > > Anyways, we can think about efficient dynamic memory expansion in
> 2.1,
> > > this
> > > > may be feasible if we free up some space in PageId to encode segment
> > > > number. There is a ticket for this:
> > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > >
> > >
> > > Alexey, if the operating system is already handling this for us, I
> don't
> > > see a reason to do it manually.
> > >
> > > I also like what Denis and Semyon are proposing. However, I would not
> > grab
> > > the full free memory. How about 80% of the free memory?
> > >
> > > D.
> > >
> >
>

[dsetrakyan]

How long will it take to make the memory expandable? Can it be done for 2.0?

On Thu, Apr 20, 2017 at 5:25 AM, Sergey Chugunov <[hidden email]>
wrote:

> Dmitriy,
>
> Replied in the ticket.
>
> Thanks,
> Sergey.
>
> On Wed, Apr 19, 2017 at 10:20 PM, Dmitriy Setrakyan <[hidden email]
> >
> wrote:
>
> > Sergey,
> >
> > I have responded in the ticket. Can you please provide the current and
> the
> > proposed configuration examples?
> >
> > D.
> >
> > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > [hidden email]>
> > wrote:
> >
> > > Guys,
> > >
> > > I created a ticket to implement these improvements, please take a look:
> > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > >
> > > Besides employing the idea of allocation 80% of physical memory I'm
> also
> > > suggesting to introduce one more configuration property to specify
> > default
> > > MemoryPolicy's size in bytes without having to use verbose syntax of
> > > *memoryPolicyConfiguration
> > > *section.
> > >
> > > Any thoughts on this?
> > >
> > > Thanks,
> > > Sergey.
> > >
> > >
> > >
> > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > [hidden email]> wrote:
> > > >
> > > > > I don't see why not, this is the way our tests are currently
> running.
> > > > > Anyways, we can think about efficient dynamic memory expansion in
> > 2.1,
> > > > this
> > > > > may be feasible if we free up some space in PageId to encode
> segment
> > > > > number. There is a ticket for this:
> > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > >
> > > >
> > > > Alexey, if the operating system is already handling this for us, I
> > don't
> > > > see a reason to do it manually.
> > > >
> > > > I also like what Denis and Semyon are proposing. However, I would not
> > > grab
> > > > the full free memory. How about 80% of the free memory?
> > > >
> > > > D.
> > > >
> > >
> >
>

[Alexey Goncharuk]

Dmitriy,

I have implemented expandable page memory and now I need to merge with the
changes that Sergey made to make max be 80% of physical memory. Once TC
passes, I will merge the changes, should not take too long, so hopefully,
the change will make it to the release.

2017-04-20 20:40 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> How long will it take to make the memory expandable? Can it be done for
> 2.0?
>
> On Thu, Apr 20, 2017 at 5:25 AM, Sergey Chugunov <
> [hidden email]>
> wrote:
>
> > Dmitriy,
> >
> > Replied in the ticket.
> >
> > Thanks,
> > Sergey.
> >
> > On Wed, Apr 19, 2017 at 10:20 PM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > wrote:
> >
> > > Sergey,
> > >
> > > I have responded in the ticket. Can you please provide the current and
> > the
> > > proposed configuration examples?
> > >
> > > D.
> > >
> > > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Guys,
> > > >
> > > > I created a ticket to implement these improvements, please take a
> look:
> > > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > > >
> > > > Besides employing the idea of allocation 80% of physical memory I'm
> > also
> > > > suggesting to introduce one more configuration property to specify
> > > default
> > > > MemoryPolicy's size in bytes without having to use verbose syntax of
> > > > *memoryPolicyConfiguration
> > > > *section.
> > > >
> > > > Any thoughts on this?
> > > >
> > > > Thanks,
> > > > Sergey.
> > > >
> > > >
> > > >
> > > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > > [hidden email]> wrote:
> > > > >
> > > > > > I don't see why not, this is the way our tests are currently
> > running.
> > > > > > Anyways, we can think about efficient dynamic memory expansion in
> > > 2.1,
> > > > > this
> > > > > > may be feasible if we free up some space in PageId to encode
> > segment
> > > > > > number. There is a ticket for this:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > > >
> > > > >
> > > > > Alexey, if the operating system is already handling this for us, I
> > > don't
> > > > > see a reason to do it manually.
> > > > >
> > > > > I also like what Denis and Semyon are proposing. However, I would
> not
> > > > grab
> > > > > the full free memory. How about 80% of the free memory?
> > > > >
> > > > > D.
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

Thanks, Alexey! Sounds good.

On Fri, Apr 21, 2017 at 8:34 AM, Alexey Goncharuk <
[hidden email]> wrote:

> Dmitriy,
>
> I have implemented expandable page memory and now I need to merge with the
> changes that Sergey made to make max be 80% of physical memory. Once TC
> passes, I will merge the changes, should not take too long, so hopefully,
> the change will make it to the release.
>
> 2017-04-20 20:40 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > How long will it take to make the memory expandable? Can it be done for
> > 2.0?
> >
> > On Thu, Apr 20, 2017 at 5:25 AM, Sergey Chugunov <
> > [hidden email]>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > Replied in the ticket.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > > On Wed, Apr 19, 2017 at 10:20 PM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > Sergey,
> > > >
> > > > I have responded in the ticket. Can you please provide the current
> and
> > > the
> > > > proposed configuration examples?
> > > >
> > > > D.
> > > >
> > > > On Wed, Apr 19, 2017 at 2:34 AM, Sergey Chugunov <
> > > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > I created a ticket to implement these improvements, please take a
> > look:
> > > > > IGNITE-5024 <https://issues.apache.org/jira/browse/IGNITE-5024>
> > > > >
> > > > > Besides employing the idea of allocation 80% of physical memory I'm
> > > also
> > > > > suggesting to introduce one more configuration property to specify
> > > > default
> > > > > MemoryPolicy's size in bytes without having to use verbose syntax
> of
> > > > > *memoryPolicyConfiguration
> > > > > *section.
> > > > >
> > > > > Any thoughts on this?
> > > > >
> > > > > Thanks,
> > > > > Sergey.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 18, 2017 at 12:12 PM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > On Tue, Apr 18, 2017 at 2:09 AM, Alexey Goncharuk <
> > > > > > [hidden email]> wrote:
> > > > > >
> > > > > > > I don't see why not, this is the way our tests are currently
> > > running.
> > > > > > > Anyways, we can think about efficient dynamic memory expansion
> in
> > > > 2.1,
> > > > > > this
> > > > > > > may be feasible if we free up some space in PageId to encode
> > > segment
> > > > > > > number. There is a ticket for this:
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-4921
> > > > > >
> > > > > >
> > > > > > Alexey, if the operating system is already handling this for us,
> I
> > > > don't
> > > > > > see a reason to do it manually.
> > > > > >
> > > > > > I also like what Denis and Semyon are proposing. However, I would
> > not
> > > > > grab
> > > > > > the full free memory. How about 80% of the free memory?
> > > > > >
> > > > > > D.
> > > > > >
> > > > >
> > > >
> > >
> >
>