Fwd: .NET java thread count keeps growing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: .NET java thread count keeps growing

dmagda
Pavel,

Do you think we can get it fixed in 2.7 time frame?

--
Denis

---------- Forwarded message ---------
From: Ilya Kasnacheev <[hidden email]>
Date: Tue, Sep 18, 2018 at 9:30 AM
Subject: Re: .NET java thread count keeps growing
To: <[hidden email]>


Hello!

I can observe the problem that you are describing. I have created a ticket https://issues.apache.org/jira/browse/IGNITE-9638

Regards,
--
Ilya Kasnacheev


пн, 17 сент. 2018 г. в 20:21, Alew <[hidden email]>:

Hi, I found a way to reproduce the issue.

I think it is related to .net thread identity.

New .net thread leads to new java thread.



On 17/09/2018 15:04, Ilya Kasnacheev wrote:
Hello!

You seem to have 3000 threads of form "Thread-NNNN", which is not what Ignite usually uses, and they're all empty.
This is mysterious so I urge you to share the sample reproducer (or, barring that, code snippet) that leads to such behavior.

By the way, do you observe any errors in the log?

Regards,
--
Ilya Kasnacheev


вс, 16 сент. 2018 г. в 4:09, Alew <[hidden email]>:
Hi!

I have growing java thread count. The application crashes with OOME. The
application is very simple, only reading values in a cache.

Any suggestion to debug this issue?

Ignite 2.4



Reply | Threaded
Open this post in threaded view
|

Re: .NET java thread count keeps growing

Pavel Tupitsyn
Denis,

Can't say for sure, did not investigate this yet. I'll try to find time
next week and get back to you.

On Sat, Sep 22, 2018 at 4:23 AM Denis Magda <[hidden email]> wrote:

> Pavel,
>
> Do you think we can get it fixed in 2.7 time frame?
>
> --
> Denis
>
> ---------- Forwarded message ---------
> From: Ilya Kasnacheev <[hidden email]>
> Date: Tue, Sep 18, 2018 at 9:30 AM
> Subject: Re: .NET java thread count keeps growing
> To: <[hidden email]>
>
>
> Hello!
>
> I can observe the problem that you are describing. I have created a ticket
> https://issues.apache.org/jira/browse/IGNITE-9638
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 17 сент. 2018 г. в 20:21, Alew <[hidden email]>:
>
>> Hi, I found a way to reproduce the issue.
>>
>> I think it is related to .net thread identity.
>>
>> New .net thread leads to new java thread.
>>
>>
>>
>> On 17/09/2018 15:04, Ilya Kasnacheev wrote:
>>
>> Hello!
>>
>> You seem to have 3000 threads of form "Thread-NNNN", which is not what
>> Ignite usually uses, and they're all empty.
>> This is mysterious so I urge you to share the sample reproducer (or,
>> barring that, code snippet) that leads to such behavior.
>>
>> By the way, do you observe any errors in the log?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> вс, 16 сент. 2018 г. в 4:09, Alew <[hidden email]>:
>>
>>> Hi!
>>>
>>> I have growing java thread count. The application crashes with OOME. The
>>> application is very simple, only reading values in a cache.
>>>
>>> Any suggestion to debug this issue?
>>>
>>> Ignite 2.4
>>>
>>>
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: .NET java thread count keeps growing

Ilya Kasnacheev
Hello!

Can we get back to this question? It rears its ugly head again:
https://stackoverflow.com/questions/57513576/apache-ignite-spawns-too-much-threads/57523214

Regards,
--
Ilya Kasnacheev


вс, 23 сент. 2018 г. в 15:20, Pavel Tupitsyn <[hidden email]>:

> Denis,
>
> Can't say for sure, did not investigate this yet. I'll try to find time
> next week and get back to you.
>
> On Sat, Sep 22, 2018 at 4:23 AM Denis Magda <[hidden email]> wrote:
>
> > Pavel,
> >
> > Do you think we can get it fixed in 2.7 time frame?
> >
> > --
> > Denis
> >
> > ---------- Forwarded message ---------
> > From: Ilya Kasnacheev <[hidden email]>
> > Date: Tue, Sep 18, 2018 at 9:30 AM
> > Subject: Re: .NET java thread count keeps growing
> > To: <[hidden email]>
> >
> >
> > Hello!
> >
> > I can observe the problem that you are describing. I have created a
> ticket
> > https://issues.apache.org/jira/browse/IGNITE-9638
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 17 сент. 2018 г. в 20:21, Alew <[hidden email]>:
> >
> >> Hi, I found a way to reproduce the issue.
> >>
> >> I think it is related to .net thread identity.
> >>
> >> New .net thread leads to new java thread.
> >>
> >>
> >>
> >> On 17/09/2018 15:04, Ilya Kasnacheev wrote:
> >>
> >> Hello!
> >>
> >> You seem to have 3000 threads of form "Thread-NNNN", which is not what
> >> Ignite usually uses, and they're all empty.
> >> This is mysterious so I urge you to share the sample reproducer (or,
> >> barring that, code snippet) that leads to such behavior.
> >>
> >> By the way, do you observe any errors in the log?
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> вс, 16 сент. 2018 г. в 4:09, Alew <[hidden email]>:
> >>
> >>> Hi!
> >>>
> >>> I have growing java thread count. The application crashes with OOME.
> The
> >>> application is very simple, only reading values in a cache.
> >>>
> >>> Any suggestion to debug this issue?
> >>>
> >>> Ignite 2.4
> >>>
> >>>
> >>>
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: .NET java thread count keeps growing

Raymond Wilson
We have observed a similar problem but have not characterised it well
enough to report on the list yet.

We are using IA .Net 2.7.5.

Sent from my iPhone

> On 17/08/2019, at 1:45 PM, Ilya Kasnacheev <[hidden email]> wrote:
>
> Hello!
>
> Can we get back to this question? It rears its ugly head again:
> https://stackoverflow.com/questions/57513576/apache-ignite-spawns-too-much-threads/57523214
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вс, 23 сент. 2018 г. в 15:20, Pavel Tupitsyn <[hidden email]>:
>
>> Denis,
>>
>> Can't say for sure, did not investigate this yet. I'll try to find time
>> next week and get back to you.
>>
>>> On Sat, Sep 22, 2018 at 4:23 AM Denis Magda <[hidden email]> wrote:
>>>
>>> Pavel,
>>>
>>> Do you think we can get it fixed in 2.7 time frame?
>>>
>>> --
>>> Denis
>>>
>>> ---------- Forwarded message ---------
>>> From: Ilya Kasnacheev <[hidden email]>
>>> Date: Tue, Sep 18, 2018 at 9:30 AM
>>> Subject: Re: .NET java thread count keeps growing
>>> To: <[hidden email]>
>>>
>>>
>>> Hello!
>>>
>>> I can observe the problem that you are describing. I have created a
>> ticket
>>> https://issues.apache.org/jira/browse/IGNITE-9638
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 17 сент. 2018 г. в 20:21, Alew <[hidden email]>:
>>>
>>>> Hi, I found a way to reproduce the issue.
>>>>
>>>> I think it is related to .net thread identity.
>>>>
>>>> New .net thread leads to new java thread.
>>>>
>>>>
>>>>
>>>> On 17/09/2018 15:04, Ilya Kasnacheev wrote:
>>>>
>>>> Hello!
>>>>
>>>> You seem to have 3000 threads of form "Thread-NNNN", which is not what
>>>> Ignite usually uses, and they're all empty.
>>>> This is mysterious so I urge you to share the sample reproducer (or,
>>>> barring that, code snippet) that leads to such behavior.
>>>>
>>>> By the way, do you observe any errors in the log?
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> вс, 16 сент. 2018 г. в 4:09, Alew <[hidden email]>:
>>>>
>>>>> Hi!
>>>>>
>>>>> I have growing java thread count. The application crashes with OOME.
>> The
>>>>> application is very simple, only reading values in a cache.
>>>>>
>>>>> Any suggestion to debug this issue?
>>>>>
>>>>> Ignite 2.4
>>>>>
>>>>>
>>>>>
>>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: .NET java thread count keeps growing

dmagda
Ilya, thanks for bringing up the discussion.

Pavel, did you have a chance to asses the problem? Is it specific to
asynchronous CQ (as reported on SO) or spans across many APIs?

-
Denis


On Sat, Aug 17, 2019 at 5:16 PM Raymond Wilson <[hidden email]>
wrote:

> We have observed a similar problem but have not characterised it well
> enough to report on the list yet.
>
> We are using IA .Net 2.7.5.
>
> Sent from my iPhone
>
> > On 17/08/2019, at 1:45 PM, Ilya Kasnacheev <[hidden email]>
> wrote:
> >
> > Hello!
> >
> > Can we get back to this question? It rears its ugly head again:
> >
> https://stackoverflow.com/questions/57513576/apache-ignite-spawns-too-much-threads/57523214
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > вс, 23 сент. 2018 г. в 15:20, Pavel Tupitsyn <[hidden email]>:
> >
> >> Denis,
> >>
> >> Can't say for sure, did not investigate this yet. I'll try to find time
> >> next week and get back to you.
> >>
> >>> On Sat, Sep 22, 2018 at 4:23 AM Denis Magda <[hidden email]> wrote:
> >>>
> >>> Pavel,
> >>>
> >>> Do you think we can get it fixed in 2.7 time frame?
> >>>
> >>> --
> >>> Denis
> >>>
> >>> ---------- Forwarded message ---------
> >>> From: Ilya Kasnacheev <[hidden email]>
> >>> Date: Tue, Sep 18, 2018 at 9:30 AM
> >>> Subject: Re: .NET java thread count keeps growing
> >>> To: <[hidden email]>
> >>>
> >>>
> >>> Hello!
> >>>
> >>> I can observe the problem that you are describing. I have created a
> >> ticket
> >>> https://issues.apache.org/jira/browse/IGNITE-9638
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> пн, 17 сент. 2018 г. в 20:21, Alew <[hidden email]>:
> >>>
> >>>> Hi, I found a way to reproduce the issue.
> >>>>
> >>>> I think it is related to .net thread identity.
> >>>>
> >>>> New .net thread leads to new java thread.
> >>>>
> >>>>
> >>>>
> >>>> On 17/09/2018 15:04, Ilya Kasnacheev wrote:
> >>>>
> >>>> Hello!
> >>>>
> >>>> You seem to have 3000 threads of form "Thread-NNNN", which is not what
> >>>> Ignite usually uses, and they're all empty.
> >>>> This is mysterious so I urge you to share the sample reproducer (or,
> >>>> barring that, code snippet) that leads to such behavior.
> >>>>
> >>>> By the way, do you observe any errors in the log?
> >>>>
> >>>> Regards,
> >>>> --
> >>>> Ilya Kasnacheev
> >>>>
> >>>>
> >>>> вс, 16 сент. 2018 г. в 4:09, Alew <[hidden email]>:
> >>>>
> >>>>> Hi!
> >>>>>
> >>>>> I have growing java thread count. The application crashes with OOME.
> >> The
> >>>>> application is very simple, only reading values in a cache.
> >>>>>
> >>>>> Any suggestion to debug this issue?
> >>>>>
> >>>>> Ignite 2.4
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: .NET java thread count keeps growing

Pavel Tupitsyn
Denis, the problem is described in
https://issues.apache.org/jira/browse/IGNITE-9638
It is not specific to Continuous Queries.
It occurs when user code starts new threads a lot and calls Ignite APIs
from there (which is not common, since idiomatic .NET code should use
built-in thread pool instead).

I investigated it thoroughly last year and, unfortunately, did not find a
clean, cross-platform solution - see my comment in Jira.
We'll have to develop a fix for every platform (Linux, Windows, Mac)
separately in C or C++, compile native binaries and ship them as part of
Ignite.NET or as a dependency.





On Mon, Aug 19, 2019 at 2:04 PM Denis Magda <[hidden email]> wrote:

> Ilya, thanks for bringing up the discussion.
>
> Pavel, did you have a chance to asses the problem? Is it specific to
> asynchronous CQ (as reported on SO) or spans across many APIs?
>
> -
> Denis
>
>
> On Sat, Aug 17, 2019 at 5:16 PM Raymond Wilson <[hidden email]
> >
> wrote:
>
> > We have observed a similar problem but have not characterised it well
> > enough to report on the list yet.
> >
> > We are using IA .Net 2.7.5.
> >
> > Sent from my iPhone
> >
> > > On 17/08/2019, at 1:45 PM, Ilya Kasnacheev <[hidden email]>
> > wrote:
> > >
> > > Hello!
> > >
> > > Can we get back to this question? It rears its ugly head again:
> > >
> >
> https://stackoverflow.com/questions/57513576/apache-ignite-spawns-too-much-threads/57523214
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > вс, 23 сент. 2018 г. в 15:20, Pavel Tupitsyn <[hidden email]>:
> > >
> > >> Denis,
> > >>
> > >> Can't say for sure, did not investigate this yet. I'll try to find
> time
> > >> next week and get back to you.
> > >>
> > >>> On Sat, Sep 22, 2018 at 4:23 AM Denis Magda <[hidden email]>
> wrote:
> > >>>
> > >>> Pavel,
> > >>>
> > >>> Do you think we can get it fixed in 2.7 time frame?
> > >>>
> > >>> --
> > >>> Denis
> > >>>
> > >>> ---------- Forwarded message ---------
> > >>> From: Ilya Kasnacheev <[hidden email]>
> > >>> Date: Tue, Sep 18, 2018 at 9:30 AM
> > >>> Subject: Re: .NET java thread count keeps growing
> > >>> To: <[hidden email]>
> > >>>
> > >>>
> > >>> Hello!
> > >>>
> > >>> I can observe the problem that you are describing. I have created a
> > >> ticket
> > >>> https://issues.apache.org/jira/browse/IGNITE-9638
> > >>>
> > >>> Regards,
> > >>> --
> > >>> Ilya Kasnacheev
> > >>>
> > >>>
> > >>> пн, 17 сент. 2018 г. в 20:21, Alew <[hidden email]>:
> > >>>
> > >>>> Hi, I found a way to reproduce the issue.
> > >>>>
> > >>>> I think it is related to .net thread identity.
> > >>>>
> > >>>> New .net thread leads to new java thread.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 17/09/2018 15:04, Ilya Kasnacheev wrote:
> > >>>>
> > >>>> Hello!
> > >>>>
> > >>>> You seem to have 3000 threads of form "Thread-NNNN", which is not
> what
> > >>>> Ignite usually uses, and they're all empty.
> > >>>> This is mysterious so I urge you to share the sample reproducer (or,
> > >>>> barring that, code snippet) that leads to such behavior.
> > >>>>
> > >>>> By the way, do you observe any errors in the log?
> > >>>>
> > >>>> Regards,
> > >>>> --
> > >>>> Ilya Kasnacheev
> > >>>>
> > >>>>
> > >>>> вс, 16 сент. 2018 г. в 4:09, Alew <[hidden email]>:
> > >>>>
> > >>>>> Hi!
> > >>>>>
> > >>>>> I have growing java thread count. The application crashes with
> OOME.
> > >> The
> > >>>>> application is very simple, only reading values in a cache.
> > >>>>>
> > >>>>> Any suggestion to debug this issue?
> > >>>>>
> > >>>>> Ignite 2.4
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: .NET java thread count keeps growing

Pavel Tupitsyn
Good news, everyone, I've found a straightforward fix for this,
cross-platform and in pure C#.
Will be included in Ignite 2.8. Details in the ticket.

Thanks

On Mon, Aug 19, 2019 at 11:20 PM Pavel Tupitsyn <[hidden email]>
wrote:

> Denis, the problem is described in
> https://issues.apache.org/jira/browse/IGNITE-9638
> It is not specific to Continuous Queries.
> It occurs when user code starts new threads a lot and calls Ignite APIs
> from there (which is not common, since idiomatic .NET code should use
> built-in thread pool instead).
>
> I investigated it thoroughly last year and, unfortunately, did not find a
> clean, cross-platform solution - see my comment in Jira.
> We'll have to develop a fix for every platform (Linux, Windows, Mac)
> separately in C or C++, compile native binaries and ship them as part of
> Ignite.NET or as a dependency.
>
>
>
>
>
> On Mon, Aug 19, 2019 at 2:04 PM Denis Magda <[hidden email]> wrote:
>
>> Ilya, thanks for bringing up the discussion.
>>
>> Pavel, did you have a chance to asses the problem? Is it specific to
>> asynchronous CQ (as reported on SO) or spans across many APIs?
>>
>> -
>> Denis
>>
>>
>> On Sat, Aug 17, 2019 at 5:16 PM Raymond Wilson <
>> [hidden email]>
>> wrote:
>>
>> > We have observed a similar problem but have not characterised it well
>> > enough to report on the list yet.
>> >
>> > We are using IA .Net 2.7.5.
>> >
>> > Sent from my iPhone
>> >
>> > > On 17/08/2019, at 1:45 PM, Ilya Kasnacheev <[hidden email]
>> >
>> > wrote:
>> > >
>> > > Hello!
>> > >
>> > > Can we get back to this question? It rears its ugly head again:
>> > >
>> >
>> https://stackoverflow.com/questions/57513576/apache-ignite-spawns-too-much-threads/57523214
>> > >
>> > > Regards,
>> > > --
>> > > Ilya Kasnacheev
>> > >
>> > >
>> > > вс, 23 сент. 2018 г. в 15:20, Pavel Tupitsyn <[hidden email]>:
>> > >
>> > >> Denis,
>> > >>
>> > >> Can't say for sure, did not investigate this yet. I'll try to find
>> time
>> > >> next week and get back to you.
>> > >>
>> > >>> On Sat, Sep 22, 2018 at 4:23 AM Denis Magda <[hidden email]>
>> wrote:
>> > >>>
>> > >>> Pavel,
>> > >>>
>> > >>> Do you think we can get it fixed in 2.7 time frame?
>> > >>>
>> > >>> --
>> > >>> Denis
>> > >>>
>> > >>> ---------- Forwarded message ---------
>> > >>> From: Ilya Kasnacheev <[hidden email]>
>> > >>> Date: Tue, Sep 18, 2018 at 9:30 AM
>> > >>> Subject: Re: .NET java thread count keeps growing
>> > >>> To: <[hidden email]>
>> > >>>
>> > >>>
>> > >>> Hello!
>> > >>>
>> > >>> I can observe the problem that you are describing. I have created a
>> > >> ticket
>> > >>> https://issues.apache.org/jira/browse/IGNITE-9638
>> > >>>
>> > >>> Regards,
>> > >>> --
>> > >>> Ilya Kasnacheev
>> > >>>
>> > >>>
>> > >>> пн, 17 сент. 2018 г. в 20:21, Alew <[hidden email]>:
>> > >>>
>> > >>>> Hi, I found a way to reproduce the issue.
>> > >>>>
>> > >>>> I think it is related to .net thread identity.
>> > >>>>
>> > >>>> New .net thread leads to new java thread.
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> On 17/09/2018 15:04, Ilya Kasnacheev wrote:
>> > >>>>
>> > >>>> Hello!
>> > >>>>
>> > >>>> You seem to have 3000 threads of form "Thread-NNNN", which is not
>> what
>> > >>>> Ignite usually uses, and they're all empty.
>> > >>>> This is mysterious so I urge you to share the sample reproducer
>> (or,
>> > >>>> barring that, code snippet) that leads to such behavior.
>> > >>>>
>> > >>>> By the way, do you observe any errors in the log?
>> > >>>>
>> > >>>> Regards,
>> > >>>> --
>> > >>>> Ilya Kasnacheev
>> > >>>>
>> > >>>>
>> > >>>> вс, 16 сент. 2018 г. в 4:09, Alew <[hidden email]>:
>> > >>>>
>> > >>>>> Hi!
>> > >>>>>
>> > >>>>> I have growing java thread count. The application crashes with
>> OOME.
>> > >> The
>> > >>>>> application is very simple, only reading values in a cache.
>> > >>>>>
>> > >>>>> Any suggestion to debug this issue?
>> > >>>>>
>> > >>>>> Ignite 2.4
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>
>> > >>
>> >
>>
>