Out of memory error in ignite 1.6

[C Ravikiran]

[C Ravikiran]

Hi Team,

Our application is running past 2 year onwards, there was no problem with
ignite.
Suddenly last 2 week onwards, we are getting out of memory error in ignite.

There was no code changes in last 2 years.

If we restart the ignite, our memory gradually increasing and we are
getting out of  memory error.

Every day we will remove all the previous cache ids, and create new cach
ids.
We have only one se of cache ids, but still our heap size is gradually
increasing.

1) why we got this problem suddenly.
2) any remediation for this problem
3) is this problem will be solved, if we upgraded to new version.

Regards,
Ravikiran c

[Ilya Kasnacheev]

Hello!

I don't think you will find much expertise on Ignite 1.x here. I recommend
upgrading it to 2.8 eventually.

Regards,
--
Ilya Kasnacheev


пт, 8 мая 2020 г. в 09:57, C Ravikiran <[hidden email]>:

> Hi Team,
>
> Our application is running past 2 year onwards, there was no problem with
> ignite.
> Suddenly last 2 week onwards, we are getting out of memory error in ignite.
>
> There was no code changes in last 2 years.
>
> If we restart the ignite, our memory gradually increasing and we are
> getting out of  memory error.
>
> Every day we will remove all the previous cache ids, and create new cach
> ids.
> We have only one se of cache ids, but still our heap size is gradually
> increasing.
>
> 1) why we got this problem suddenly.
> 2) any remediation for this problem
> 3) is this problem will be solved, if we upgraded to new version.
>
> Regards,
> Ravikiran c
>

[C Ravikiran]

Is this problem will be solved, if I upgraded to 2.8 version.


On Fri 8 May, 2020, 10:02 PM Ilya Kasnacheev, <[hidden email]>
wrote:

> Hello!
>
> I don't think you will find much expertise on Ignite 1.x here. I recommend
> upgrading it to 2.8 eventually.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 8 мая 2020 г. в 09:57, C Ravikiran <[hidden email]>:
>
> > Hi Team,
> >
> > Our application is running past 2 year onwards, there was no problem with
> > ignite.
> > Suddenly last 2 week onwards, we are getting out of memory error in
> ignite.
> >
> > There was no code changes in last 2 years.
> >
> > If we restart the ignite, our memory gradually increasing and we are
> > getting out of  memory error.
> >
> > Every day we will remove all the previous cache ids, and create new cach
> > ids.
> > We have only one se of cache ids, but still our heap size is gradually
> > increasing.
> >
> > 1) why we got this problem suddenly.
> > 2) any remediation for this problem
> > 3) is this problem will be solved, if we upgraded to new version.
> >
> > Regards,
> > Ravikiran c
> >
>

[dmagda]

You just stated that experienced OOM issues with no details for suggestions
or help with troubleshooting. Take heap dumps, analyze GC logs, etc:
https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses

-
Denis


On Fri, May 8, 2020 at 11:31 AM C Ravikiran <[hidden email]>
wrote:

> Is this problem will be solved, if I upgraded to 2.8 version.
>
>
> On Fri 8 May, 2020, 10:02 PM Ilya Kasnacheev, <[hidden email]>
> wrote:
>
> > Hello!
> >
> > I don't think you will find much expertise on Ignite 1.x here. I
> recommend
> > upgrading it to 2.8 eventually.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 8 мая 2020 г. в 09:57, C Ravikiran <[hidden email]>:
> >
> > > Hi Team,
> > >
> > > Our application is running past 2 year onwards, there was no problem
> with
> > > ignite.
> > > Suddenly last 2 week onwards, we are getting out of memory error in
> > ignite.
> > >
> > > There was no code changes in last 2 years.
> > >
> > > If we restart the ignite, our memory gradually increasing and we are
> > > getting out of  memory error.
> > >
> > > Every day we will remove all the previous cache ids, and create new
> cach
> > > ids.
> > > We have only one se of cache ids, but still our heap size is gradually
> > > increasing.
> > >
> > > 1) why we got this problem suddenly.
> > > 2) any remediation for this problem
> > > 3) is this problem will be solved, if we upgraded to new version.
> > >
> > > Regards,
> > > Ravikiran c
> > >
> >
>

[C Ravikiran]

I am new to ignite, if you need any details please let me know, I will
check and provide those details.

I need proper justification, if I need to upgrade to new version.

Suddenly we are getting OOM error, after 2 years.

Every day we are removing all old cache ids and adding new cache ids

Our heap size is 32 GB.





A


On Sat 9 May, 2020, 12:06 AM Denis Magda, <[hidden email]> wrote:

> You just stated that experienced OOM issues with no details for suggestions
> or help with troubleshooting. Take heap dumps, analyze GC logs, etc:
>
> https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses
>
> -
> Denis
>
>
> On Fri, May 8, 2020 at 11:31 AM C Ravikiran <[hidden email]>
> wrote:
>
> > Is this problem will be solved, if I upgraded to 2.8 version.
> >
> >
> > On Fri 8 May, 2020, 10:02 PM Ilya Kasnacheev, <[hidden email]
> >
> > wrote:
> >
> > > Hello!
> > >
> > > I don't think you will find much expertise on Ignite 1.x here. I
> > recommend
> > > upgrading it to 2.8 eventually.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пт, 8 мая 2020 г. в 09:57, C Ravikiran <[hidden email]>:
> > >
> > > > Hi Team,
> > > >
> > > > Our application is running past 2 year onwards, there was no problem
> > with
> > > > ignite.
> > > > Suddenly last 2 week onwards, we are getting out of memory error in
> > > ignite.
> > > >
> > > > There was no code changes in last 2 years.
> > > >
> > > > If we restart the ignite, our memory gradually increasing and we are
> > > > getting out of  memory error.
> > > >
> > > > Every day we will remove all the previous cache ids, and create new
> > cach
> > > > ids.
> > > > We have only one se of cache ids, but still our heap size is
> gradually
> > > > increasing.
> > > >
> > > > 1) why we got this problem suddenly.
> > > > 2) any remediation for this problem
> > > > 3) is this problem will be solved, if we upgraded to new version.
> > > >
> > > > Regards,
> > > > Ravikiran c
> > > >
> > >
> >
>

[dmagda]

You need to get to the bottom of why JVM fails with the OOM on your cluster
nodes. An ignite node is no more than a Java application and you can start
investigating the issue using standard methods. For now, I can only share
some useful pointers:

   -
   https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses
   - https://www.toptal.com/java/hunting-memory-leaks-in-java

Once you pinpoint the memory leak and confirm the leak is caused by Ignite
internal objects, then send us a note.

Also, ensure that you simply didn't exhaust all available memory. It might
happen that you store more data in Ignite than before and there is no
capacity available. In this case, just scale out, add more nodes to the
cluster.

-
Denis


On Fri, May 8, 2020 at 11:46 AM C Ravikiran <[hidden email]>
wrote:

> I am new to ignite, if you need any details please let me know, I will
> check and provide those details.
>
> I need proper justification, if I need to upgrade to new version.
>
> Suddenly we are getting OOM error, after 2 years.
>
> Every day we are removing all old cache ids and adding new cache ids
>
> Our heap size is 32 GB.
>
>
>
>
>
> A
>
>
> On Sat 9 May, 2020, 12:06 AM Denis Magda, <[hidden email]> wrote:
>
> > You just stated that experienced OOM issues with no details for
> suggestions
> > or help with troubleshooting. Take heap dumps, analyze GC logs, etc:
> >
> >
> https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses
> >
> > -
> > Denis
> >
> >
> > On Fri, May 8, 2020 at 11:31 AM C Ravikiran <[hidden email]>
> > wrote:
> >
> > > Is this problem will be solved, if I upgraded to 2.8 version.
> > >
> > >
> > > On Fri 8 May, 2020, 10:02 PM Ilya Kasnacheev, <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > I don't think you will find much expertise on Ignite 1.x here. I
> > > recommend
> > > > upgrading it to 2.8 eventually.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > пт, 8 мая 2020 г. в 09:57, C Ravikiran <[hidden email]>:
> > > >
> > > > > Hi Team,
> > > > >
> > > > > Our application is running past 2 year onwards, there was no
> problem
> > > with
> > > > > ignite.
> > > > > Suddenly last 2 week onwards, we are getting out of memory error in
> > > > ignite.
> > > > >
> > > > > There was no code changes in last 2 years.
> > > > >
> > > > > If we restart the ignite, our memory gradually increasing and we
> are
> > > > > getting out of  memory error.
> > > > >
> > > > > Every day we will remove all the previous cache ids, and create new
> > > cach
> > > > > ids.
> > > > > We have only one se of cache ids, but still our heap size is
> > gradually
> > > > > increasing.
> > > > >
> > > > > 1) why we got this problem suddenly.
> > > > > 2) any remediation for this problem
> > > > > 3) is this problem will be solved, if we upgraded to new version.
> > > > >
> > > > > Regards,
> > > > > Ravikiran c
> > > > >
> > > >
> > >
> >
>

[C Ravikiran]

Last 2 years there is no code changes.
Our application is running in only one node.

Our daily data is around 10GB, as our scheduler is removing old data.
Our application users also less.

Why we got suddenly, these memory leakage problem, when there is no
application code change.

Problem we are getting only in production environment.

We are not getting the same Error in UAT environment, we are using same
configuration.





On Sat 9 May, 2020, 12:23 AM Denis Magda, <[hidden email]> wrote:

> You need to get to the bottom of why JVM fails with the OOM on your cluster
> nodes. An ignite node is no more than a Java application and you can start
> investigating the issue using standard methods. For now, I can only share
> some useful pointers:
>
>    -
>
> https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses
>    - https://www.toptal.com/java/hunting-memory-leaks-in-java
>
> Once you pinpoint the memory leak and confirm the leak is caused by Ignite
> internal objects, then send us a note.
>
> Also, ensure that you simply didn't exhaust all available memory. It might
> happen that you store more data in Ignite than before and there is no
> capacity available. In this case, just scale out, add more nodes to the
> cluster.
>
> -
> Denis
>
>
> On Fri, May 8, 2020 at 11:46 AM C Ravikiran <[hidden email]>
> wrote:
>
> > I am new to ignite, if you need any details please let me know, I will
> > check and provide those details.
> >
> > I need proper justification, if I need to upgrade to new version.
> >
> > Suddenly we are getting OOM error, after 2 years.
> >
> > Every day we are removing all old cache ids and adding new cache ids
> >
> > Our heap size is 32 GB.
> >
> >
> >
> >
> >
> > A
> >
> >
> > On Sat 9 May, 2020, 12:06 AM Denis Magda, <[hidden email]> wrote:
> >
> > > You just stated that experienced OOM issues with no details for
> > suggestions
> > > or help with troubleshooting. Take heap dumps, analyze GC logs, etc:
> > >
> > >
> >
> https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Fri, May 8, 2020 at 11:31 AM C Ravikiran <
> [hidden email]>
> > > wrote:
> > >
> > > > Is this problem will be solved, if I upgraded to 2.8 version.
> > > >
> > > >
> > > > On Fri 8 May, 2020, 10:02 PM Ilya Kasnacheev, <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > I don't think you will find much expertise on Ignite 1.x here. I
> > > > recommend
> > > > > upgrading it to 2.8 eventually.
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > пт, 8 мая 2020 г. в 09:57, C Ravikiran <[hidden email]
> >:
> > > > >
> > > > > > Hi Team,
> > > > > >
> > > > > > Our application is running past 2 year onwards, there was no
> > problem
> > > > with
> > > > > > ignite.
> > > > > > Suddenly last 2 week onwards, we are getting out of memory error
> in
> > > > > ignite.
> > > > > >
> > > > > > There was no code changes in last 2 years.
> > > > > >
> > > > > > If we restart the ignite, our memory gradually increasing and we
> > are
> > > > > > getting out of  memory error.
> > > > > >
> > > > > > Every day we will remove all the previous cache ids, and create
> new
> > > > cach
> > > > > > ids.
> > > > > > We have only one se of cache ids, but still our heap size is
> > > gradually
> > > > > > increasing.
> > > > > >
> > > > > > 1) why we got this problem suddenly.
> > > > > > 2) any remediation for this problem
> > > > > > 3) is this problem will be solved, if we upgraded to new version.
> > > > > >
> > > > > > Regards,
> > > > > > Ravikiran c
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Nikolay Izhikov-2]

Can you, please, send us

1. Detailed error message(stack trace, etc)
2. Ignite configuration.

Any other details that can be usefull.

> 8 мая 2020 г., в 22:08, C Ravikiran <[hidden email]> написал(а):
>
> Last 2 years there is no code changes.
> Our application is running in only one node.
>
> Our daily data is around 10GB, as our scheduler is removing old data.
> Our application users also less.
>
> Why we got suddenly, these memory leakage problem, when there is no
> application code change.
>
> Problem we are getting only in production environment.
>
> We are not getting the same Error in UAT environment, we are using same
> configuration.
>
>
>
>
>
> On Sat 9 May, 2020, 12:23 AM Denis Magda, <[hidden email]> wrote:
>
>> You need to get to the bottom of why JVM fails with the OOM on your cluster
>> nodes. An ignite node is no more than a Java application and you can start
>> investigating the issue using standard methods. For now, I can only share
>> some useful pointers:
>>
>>   -
>>
>> https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses
>>   - https://www.toptal.com/java/hunting-memory-leaks-in-java
>>
>> Once you pinpoint the memory leak and confirm the leak is caused by Ignite
>> internal objects, then send us a note.
>>
>> Also, ensure that you simply didn't exhaust all available memory. It might
>> happen that you store more data in Ignite than before and there is no
>> capacity available. In this case, just scale out, add more nodes to the
>> cluster.
>>
>> -
>> Denis
>>
>>
>> On Fri, May 8, 2020 at 11:46 AM C Ravikiran <[hidden email]>
>> wrote:
>>
>>> I am new to ignite, if you need any details please let me know, I will
>>> check and provide those details.
>>>
>>> I need proper justification, if I need to upgrade to new version.
>>>
>>> Suddenly we are getting OOM error, after 2 years.
>>>
>>> Every day we are removing all old cache ids and adding new cache ids
>>>
>>> Our heap size is 32 GB.
>>>
>>>
>>>
>>>
>>>
>>> A
>>>
>>>
>>> On Sat 9 May, 2020, 12:06 AM Denis Magda, <[hidden email]> wrote:
>>>
>>>> You just stated that experienced OOM issues with no details for
>>> suggestions
>>>> or help with troubleshooting. Take heap dumps, analyze GC logs, etc:
>>>>
>>>>
>>>
>> https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Fri, May 8, 2020 at 11:31 AM C Ravikiran <
>> [hidden email]>
>>>> wrote:
>>>>
>>>>> Is this problem will be solved, if I upgraded to 2.8 version.
>>>>>
>>>>>
>>>>> On Fri 8 May, 2020, 10:02 PM Ilya Kasnacheev, <
>>> [hidden email]
>>>>>
>>>>> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> I don't think you will find much expertise on Ignite 1.x here. I
>>>>> recommend
>>>>>> upgrading it to 2.8 eventually.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> пт, 8 мая 2020 г. в 09:57, C Ravikiran <[hidden email]
>>> :
>>>>>>
>>>>>>> Hi Team,
>>>>>>>
>>>>>>> Our application is running past 2 year onwards, there was no
>>> problem
>>>>> with
>>>>>>> ignite.
>>>>>>> Suddenly last 2 week onwards, we are getting out of memory error
>> in
>>>>>> ignite.
>>>>>>>
>>>>>>> There was no code changes in last 2 years.
>>>>>>>
>>>>>>> If we restart the ignite, our memory gradually increasing and we
>>> are
>>>>>>> getting out of  memory error.
>>>>>>>
>>>>>>> Every day we will remove all the previous cache ids, and create
>> new
>>>>> cach
>>>>>>> ids.
>>>>>>> We have only one se of cache ids, but still our heap size is
>>>> gradually
>>>>>>> increasing.
>>>>>>>
>>>>>>> 1) why we got this problem suddenly.
>>>>>>> 2) any remediation for this problem
>>>>>>> 3) is this problem will be solved, if we upgraded to new version.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Ravikiran c
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

[C Ravikiran]

Hi Team,

Currently my project is ignite 2.6 version.

2.6 configuration in my project is below.

JVM_OPTS= -Xms2g  Xmx32g

But in 2.8 is little different,

JVM_OPTS = -Xms1g Xmx32g

In 2.8 also
Shall I update the same configuration like 2.6.

If change is, is there any problem I will get



On Fri 8 May, 2020, 12:20 PM C Ravikiran, <[hidden email]>
wrote:

>