New method for MemoryMetrics: segment fill variance

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

New method for MemoryMetrics: segment fill variance

Ivan Rakov
Hello Igniters,

I want to propose one more method for memory metrics -
segmentFillVariance, float value from 0.0 to 1.0.

Motivation: when persistence is enabled, we split memory policy into K
equal segments (K = concurrency level). We calculate hash of pageId to
determine segment where page will be stored.
Pages eviction to disk starts when segment is full. If hash function is
bad enough, one segment can overflow when there are lots of free space,
and evictions can start too early. We want to be able to distinguish
such situations.

I propose to calculate segmentFillVariance as difference between maximum
and minimum percentage of segment fill. Greater variance signals about
worse hash function.

Thoughts?

--
Best Regards,
Ivan Rakov

Reply | Threaded
Open this post in threaded view
|

Re: New method for MemoryMetrics: segment fill variance

dsetrakyan
Ivan,

How about renaming it to hashFunctionScore? This will be more digestable to
users. "segmentFillVariance" sounds too scientific even to me.

D.

On Tue, Jul 18, 2017 at 10:39 AM, Ivan Rakov <[hidden email]> wrote:

> Hello Igniters,
>
> I want to propose one more method for memory metrics -
> segmentFillVariance, float value from 0.0 to 1.0.
>
> Motivation: when persistence is enabled, we split memory policy into K
> equal segments (K = concurrency level). We calculate hash of pageId to
> determine segment where page will be stored.
> Pages eviction to disk starts when segment is full. If hash function is
> bad enough, one segment can overflow when there are lots of free space, and
> evictions can start too early. We want to be able to distinguish such
> situations.
>
> I propose to calculate segmentFillVariance as difference between maximum
> and minimum percentage of segment fill. Greater variance signals about
> worse hash function.
>
> Thoughts?
>
> --
> Best Regards,
> Ivan Rakov
>
>
Reply | Threaded
Open this post in threaded view
|

Re: New method for MemoryMetrics: segment fill variance

Ivan Rakov
Dmitriy,

I agree, hashFunctionScore is more comprehensible.
I'd correct it to "pageHashFunctionScore" to make it clear that it's
about function that distributes pages, not about hashCode() of
user-provided keys.

--
Best Regards,
Ivan Rakov

On 18.07.2017 19:34, Dmitriy Setrakyan wrote:

> Ivan,
>
> How about renaming it to hashFunctionScore? This will be more digestable to
> users. "segmentFillVariance" sounds too scientific even to me.
>
> D.
>
> On Tue, Jul 18, 2017 at 10:39 AM, Ivan Rakov <[hidden email]> wrote:
>
>> Hello Igniters,
>>
>> I want to propose one more method for memory metrics -
>> segmentFillVariance, float value from 0.0 to 1.0.
>>
>> Motivation: when persistence is enabled, we split memory policy into K
>> equal segments (K = concurrency level). We calculate hash of pageId to
>> determine segment where page will be stored.
>> Pages eviction to disk starts when segment is full. If hash function is
>> bad enough, one segment can overflow when there are lots of free space, and
>> evictions can start too early. We want to be able to distinguish such
>> situations.
>>
>> I propose to calculate segmentFillVariance as difference between maximum
>> and minimum percentage of segment fill. Greater variance signals about
>> worse hash function.
>>
>> Thoughts?
>>
>> --
>> Best Regards,
>> Ivan Rakov
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: New method for MemoryMetrics: segment fill variance

dsetrakyan
Sounds good!

⁣D.​

On Jul 19, 2017, 1:54 PM, at 1:54 PM, Ivan Rakov <[hidden email]> wrote:

>Dmitriy,
>
>I agree, hashFunctionScore is more comprehensible.
>I'd correct it to "pageHashFunctionScore" to make it clear that it's
>about function that distributes pages, not about hashCode() of
>user-provided keys.
>
>--
>Best Regards,
>Ivan Rakov
>
>On 18.07.2017 19:34, Dmitriy Setrakyan wrote:
>> Ivan,
>>
>> How about renaming it to hashFunctionScore? This will be more
>digestable to
>> users. "segmentFillVariance" sounds too scientific even to me.
>>
>> D.
>>
>> On Tue, Jul 18, 2017 at 10:39 AM, Ivan Rakov <[hidden email]>
>wrote:
>>
>>> Hello Igniters,
>>>
>>> I want to propose one more method for memory metrics -
>>> segmentFillVariance, float value from 0.0 to 1.0.
>>>
>>> Motivation: when persistence is enabled, we split memory policy into
>K
>>> equal segments (K = concurrency level). We calculate hash of pageId
>to
>>> determine segment where page will be stored.
>>> Pages eviction to disk starts when segment is full. If hash function
>is
>>> bad enough, one segment can overflow when there are lots of free
>space, and
>>> evictions can start too early. We want to be able to distinguish
>such
>>> situations.
>>>
>>> I propose to calculate segmentFillVariance as difference between
>maximum
>>> and minimum percentage of segment fill. Greater variance signals
>about
>>> worse hash function.
>>>
>>> Thoughts?
>>>
>>> --
>>> Best Regards,
>>> Ivan Rakov
>>>
>>>