[jira] [Created] (IGNITE-7083) Reduce memory usage of CachePartitionFullCountersMap

[Anton Vinogradov (Jira)]

Sunny Chan created IGNITE-7083:
----------------------------------

             Summary: Reduce memory usage of CachePartitionFullCountersMap
                 Key: IGNITE-7083
                 URL: https://issues.apache.org/jira/browse/IGNITE-7083
             Project: Ignite
          Issue Type: Improvement
          Components: cache
    Affects Versions: 2.3
         Environment: Any
            Reporter: Sunny Chan


The Cache Partition Exchange Manager kept a copy of the already completed exchange. However, we have found that it uses a significant amount of memory. Upon further investigation using heap dump we have found that a large amount of memory is used by the CachePartitionFullCountersMap. We have also observed in most cases, these maps contains only 0s.

Therefore I propose an optimization for this: Initially the long arrays to store initial update counter and update counter in the CPFCM will be null, and when you get the value and see these tables are null then we will return 0 for the counter. We only allocate the long arrays when there is any non-zero updates to the the map.

In our tests, the amount of heap used by GridCachePartitionExchangeManager was around 70MB (67 copies of these CPFCM), after we apply the optimization it drops to around 9MB.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[dmagda]

Ignite, especially Sam and Alex G., what’d you think about this?

—
Denis

> Begin forwarded message:
>
> From: "Sunny Chan (JIRA)" <[hidden email]>
> Subject: [jira] [Created] (IGNITE-7083) Reduce memory usage of CachePartitionFullCountersMap
> Date: November 30, 2017 at 8:31:01 PM PST
> To: [hidden email]
> Reply-To: [hidden email]
>
> Sunny Chan created IGNITE-7083:
> ----------------------------------
>
>             Summary: Reduce memory usage of CachePartitionFullCountersMap
>                 Key: IGNITE-7083
>                 URL: https://issues.apache.org/jira/browse/IGNITE-7083
>             Project: Ignite
>          Issue Type: Improvement
>          Components: cache
>    Affects Versions: 2.3
>         Environment: Any
>            Reporter: Sunny Chan
>
>
> The Cache Partition Exchange Manager kept a copy of the already completed exchange. However, we have found that it uses a significant amount of memory. Upon further investigation using heap dump we have found that a large amount of memory is used by the CachePartitionFullCountersMap. We have also observed in most cases, these maps contains only 0s.
>
> Therefore I propose an optimization for this: Initially the long arrays to store initial update counter and update counter in the CPFCM will be null, and when you get the value and see these tables are null then we will return 0 for the counter. We only allocate the long arrays when there is any non-zero updates to the the map.
>
> In our tests, the amount of heap used by GridCachePartitionExchangeManager was around 70MB (67 copies of these CPFCM), after we apply the optimization it drops to around 9MB.
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)

[Sunny Chan, CLSA]

I have created a pull request for my proposed change.

https://github.com/apache/ignite/pull/3152

-----Original Message-----
From: Denis Magda [mailto:[hidden email]]
Sent: Saturday, December 02, 2017 3:10 AM
To: [hidden email]; Semen Boikov; Alexey Goncharuk
Subject: Fwd: [jira] [Created] (IGNITE-7083) Reduce memory usage of CachePartitionFullCountersMap

Ignite, especially Sam and Alex G., what’d you think about this?

—
Denis

> Begin forwarded message:
>
> From: "Sunny Chan (JIRA)" <[hidden email]>
> Subject: [jira] [Created] (IGNITE-7083) Reduce memory usage of CachePartitionFullCountersMap
> Date: November 30, 2017 at 8:31:01 PM PST
> To: [hidden email]
> Reply-To: [hidden email]
>
> Sunny Chan created IGNITE-7083:
> ----------------------------------
>
>             Summary: Reduce memory usage of CachePartitionFullCountersMap
>                 Key: IGNITE-7083
>                 URL: https://issues.apache.org/jira/browse/IGNITE-7083
>             Project: Ignite
>          Issue Type: Improvement
>          Components: cache
>    Affects Versions: 2.3
>         Environment: Any
>            Reporter: Sunny Chan
>
>
> The Cache Partition Exchange Manager kept a copy of the already completed exchange. However, we have found that it uses a significant amount of memory. Upon further investigation using heap dump we have found that a large amount of memory is used by the CachePartitionFullCountersMap. We have also observed in most cases, these maps contains only 0s.
>
> Therefore I propose an optimization for this: Initially the long arrays to store initial update counter and update counter in the CPFCM will be null, and when you get the value and see these tables are null then we will return 0 for the counter. We only allocate the long arrays when there is any non-zero updates to the the map.
>
> In our tests, the amount of heap used by GridCachePartitionExchangeManager was around 70MB (67 copies of these CPFCM), after we apply the optimization it drops to around 9MB.
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)

The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.
CLSA is ISO14001 certified and committed to reducing environmental impact.