Weird FillFactor metric fluctuation.

[Andrew Mashenkov]

Hi Ignites,

I've tried to estimate in runtime how much free memory there are on a node.
Looking at memory metrics API, I need a pageSize, number of allocated pages
and a fillFactor of these pages.

But it looks like FillFactor metric is broken or useless.

A simple test where new entries are put to cache shows next:
- number of allocated page is constantly grows which looks ok.
- FillFactor metric has weird fluctuation. It seems ok if its value will
slightly decreased when new pages allocated,
but I observe 2-10+ times difference between checks.
Also calculated total used memory can also significanlty decreased.


Here is a ticket for this issue [1].
Can someone clarify, is it expected behavior? what is FillFactor metric
means?
how this can be fixed or how used memory should be calculated if metrics
itself is ok?

Thoughts?


[1] https://issues.apache.org/jira/browse/IGNITE-7489

--
Best regards,
Andrey V. Mashenkov

[Andrey Kuznetsov]

FillFactor metric is in fact FreeList utilization factor, so the name is
definitely misleading. Currently we have no metric to estimate free space.
And we should define what is 'free' in terms of Ignite page memory
implementation.

2018-01-24 17:57 GMT+03:00 Andrey Mashenkov <[hidden email]>:

> Hi Ignites,
>
> I've tried to estimate in runtime how much free memory there are on a node.
> Looking at memory metrics API, I need a pageSize, number of allocated pages
> and a fillFactor of these pages.
>
> But it looks like FillFactor metric is broken or useless.
>
> A simple test where new entries are put to cache shows next:
> - number of allocated page is constantly grows which looks ok.
> - FillFactor metric has weird fluctuation. It seems ok if its value will
> slightly decreased when new pages allocated,
> but I observe 2-10+ times difference between checks.
> Also calculated total used memory can also significanlty decreased.
>
>
> Here is a ticket for this issue [1].
> Can someone clarify, is it expected behavior? what is FillFactor metric
> means?
> how this can be fixed or how used memory should be calculated if metrics
> itself is ok?
>
> Thoughts?
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7489
>
> --
> Best regards,
> Andrey V. Mashenkov
>

--
Best regards,
  Andrey Kuznetsov.

[dmagda]

There will be a way to get used and free size in bytes in Ignite 2.4:
https://issues.apache.org/jira/browse/IGNITE-7491 <https://issues.apache.org/jira/browse/IGNITE-7491>

*Anton*, did we use fill factor parameter to calculate the used space for the new metrics? If it’s so then the issue reported by Andrey needs to be investigated and solved before 2.4 release.

—
Denis

> On Jan 24, 2018, at 7:50 AM, Andrey Kuznetsov <[hidden email]> wrote:
>
> FillFactor metric is in fact FreeList utilization factor, so the name is
> definitely misleading. Currently we have no metric to estimate free space.
> And we should define what is 'free' in terms of Ignite page memory
> implementation.
>
> 2018-01-24 17:57 GMT+03:00 Andrey Mashenkov <[hidden email]>:
>
>> Hi Ignites,
>>
>> I've tried to estimate in runtime how much free memory there are on a node.
>> Looking at memory metrics API, I need a pageSize, number of allocated pages
>> and a fillFactor of these pages.
>>
>> But it looks like FillFactor metric is broken or useless.
>>
>> A simple test where new entries are put to cache shows next:
>> - number of allocated page is constantly grows which looks ok.
>> - FillFactor metric has weird fluctuation. It seems ok if its value will
>> slightly decreased when new pages allocated,
>> but I observe 2-10+ times difference between checks.
>> Also calculated total used memory can also significanlty decreased.
>>
>>
>> Here is a ticket for this issue [1].
>> Can someone clarify, is it expected behavior? what is FillFactor metric
>> means?
>> how this can be fixed or how used memory should be calculated if metrics
>> itself is ok?
>>
>> Thoughts?
>>
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-7489
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>
>
> --
> Best regards,
>  Andrey Kuznetsov.

[Anton Vinogradov-2]

Denis,

Metrics listed at https://issues.apache.org/jira/browse/IGNITE-7491 are not
relevant to FillFactor.

On Thu, Jan 25, 2018 at 3:17 AM, Denis Magda <[hidden email]> wrote:

> There will be a way to get used and free size in bytes in Ignite 2.4:
> https://issues.apache.org/jira/browse/IGNITE-7491
>
> *Anton*, did we use fill factor parameter to calculate the used space for
> the new metrics? If it’s so then the issue reported by Andrey needs to be
> investigated and solved before 2.4 release.
>
> —
> Denis
>
>
> On Jan 24, 2018, at 7:50 AM, Andrey Kuznetsov <[hidden email]> wrote:
>
> FillFactor metric is in fact FreeList utilization factor, so the name is
> definitely misleading. Currently we have no metric to estimate free space.
> And we should define what is 'free' in terms of Ignite page memory
> implementation.
>
> 2018-01-24 17:57 GMT+03:00 Andrey Mashenkov <[hidden email]>:
>
> Hi Ignites,
>
> I've tried to estimate in runtime how much free memory there are on a node.
> Looking at memory metrics API, I need a pageSize, number of allocated pages
> and a fillFactor of these pages.
>
> But it looks like FillFactor metric is broken or useless.
>
> A simple test where new entries are put to cache shows next:
> - number of allocated page is constantly grows which looks ok.
> - FillFactor metric has weird fluctuation. It seems ok if its value will
> slightly decreased when new pages allocated,
> but I observe 2-10+ times difference between checks.
> Also calculated total used memory can also significanlty decreased.
>
>
> Here is a ticket for this issue [1].
> Can someone clarify, is it expected behavior? what is FillFactor metric
> means?
> how this can be fixed or how used memory should be calculated if metrics
> itself is ok?
>
> Thoughts?
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7489
>
> --
> Best regards,
> Andrey V. Mashenkov
>
>
>
>
> --
> Best regards,
>  Andrey Kuznetsov.
>
>
>