Apache Ignite Developers - Legacy Mail Archive

Create separate thread pool for query execution

Classic

List

Threaded

15 messages Options

Vladimir Ozerov

Create separate thread pool for query execution

Igniters,

Currently if several long-running queries are submitted, it may stall all
cache operations because all theads in the system pool will be busy with
queries.

I think it makes sense to move queries execution into separate dedicated
thread pool. As we have timeouts for core pool threads now, it should not
affect memory consumption or startup time anyhow. Thoughts?

Vladimir.

Sergi

Re: Create separate thread pool for query execution

+1

Sergi

2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]>:

> Igniters,
>
> Currently if several long-running queries are submitted, it may stall all
> cache operations because all theads in the system pool will be busy with
> queries.
>
> I think it makes sense to move queries execution into separate dedicated
> thread pool. As we have timeouts for core pool threads now, it should not
> affect memory consumption or startup time anyhow. Thoughts?
>
> Vladimir.
>

dsetrakyan

Re: Create separate thread pool for query execution

How about long running compute? Do we need a separate thread pool for it as
well?

On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <[hidden email]>
wrote:

> +1
>
> Sergi
>
> 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]>:
>
> > Igniters,
> >
> > Currently if several long-running queries are submitted, it may stall all
> > cache operations because all theads in the system pool will be busy with
> > queries.
> >
> > I think it makes sense to move queries execution into separate dedicated
> > thread pool. As we have timeouts for core pool threads now, it should not
> > affect memory consumption or startup time anyhow. Thoughts?
> >
> > Vladimir.
> >
>

Romain Gilles

Re: Create separate thread pool for query execution

Hi igniters,
I'm not against to create a thread pool for each features but I have some difficulty to minimize the number of threads required to start ignite... If you add too many thread pools does this will not increase the number of threads at startup?

Thanks.

Romain

________________________________
De: Dmitriy Setrakyan <[hidden email]>
Envoyé: 23 oct. 2016 23:00
À: [hidden email]
Objet: Re: Create separate thread pool for query execution

How about long running compute? Do we need a separate thread pool for it as
well?

On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <[hidden email]>
wrote:

"Misys" is the trade name of the Misys group of companies. This email and any attachments have been scanned for known viruses using multiple scanners. This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person. This email does not constitute the commencement of legal relations between you and Misys. Please refer to the executed contract between you and the relevant member of the Misys group for the identity of the contracting party with which you are dealing.

Vladimir Ozerov

Re: Create separate thread pool for query execution

In reply to this post by dsetrakyan

Dima,
We already have one - public pool.

24 окт. 2016 г. 0:00 пользователь "Dmitriy Setrakyan" <[hidden email]>
написал:

> How about long running compute? Do we need a separate thread pool for it as
> well?
>
> On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > +1
> >
> > Sergi
> >
> > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> >
> > > Igniters,
> > >
> > > Currently if several long-running queries are submitted, it may stall
> all
> > > cache operations because all theads in the system pool will be busy
> with
> > > queries.
> > >
> > > I think it makes sense to move queries execution into separate
> dedicated
> > > thread pool. As we have timeouts for core pool threads now, it should
> not
> > > affect memory consumption or startup time anyhow. Thoughts?
> > >
> > > Vladimir.
> > >
> >
>

Vladimir Ozerov

Re: Create separate thread pool for query execution

In reply to this post by Romain Gilles

Romain,
We do not pre-start threads in our pools on Ignite start. Moreover, idle
threads are stopped after some timeout.

24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <[hidden email]>
написал:

Hi igniters,
I'm not against to create a thread pool for each features but I have some
difficulty to minimize the number of threads required to start ignite... If
you add too many thread pools does this will not increase the number of
threads at startup?

Thanks.

Romain

________________________________
De: Dmitriy Setrakyan <[hidden email]>
Envoyé: 23 oct. 2016 23:00
À: [hidden email]
Objet: Re: Create separate thread pool for query execution

How about long running compute? Do we need a separate thread pool for it as
well?

On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <[hidden email]>
wrote:

> +1
>
> Sergi
>
> 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]>:
>
> > Igniters,
> >
> > Currently if several long-running queries are submitted, it may stall
all
> > cache operations because all theads in the system pool will be busy with
> > queries.
> >
> > I think it makes sense to move queries execution into separate dedicated
> > thread pool. As we have timeouts for core pool threads now, it should
not
> > affect memory consumption or startup time anyhow. Thoughts?
> >
> > Vladimir.
> >
>
"Misys" is the trade name of the Misys group of companies. This email and
any attachments have been scanned for known viruses using multiple
scanners. This email message is intended for the named recipient only. It
may be privileged and/or confidential. If you are not the named recipient
of this email please notify us immediately and do not copy it or use it for
any purpose, nor disclose its contents to any other person. This email does
not constitute the commencement of legal relations between you and Misys.
Please refer to the executed contract between you and the relevant member
of the Misys group for the identity of the contracting party with which you
are dealing.

Sergey Kozlov

Re: Create separate thread pool for query execution

Hi

It seems we need a set of dedicated pools for various purposes. Could we
design a general apporach for this like following:
- define dedicated pools in the grid configuration
- run a query/compute/etc in the particular pool

On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Romain,
> We do not pre-start threads in our pools on Ignite start. Moreover, idle
> threads are stopped after some timeout.
>
> 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <
> [hidden email]>
> написал:
>
> Hi igniters,
> I'm not against to create a thread pool for each features but I have some
> difficulty to minimize the number of threads required to start ignite... If
> you add too many thread pools does this will not increase the number of
> threads at startup?
>
> Thanks.
>
> Romain
>
>
> ________________________________
> De: Dmitriy Setrakyan <[hidden email]>
> Envoyé: 23 oct. 2016 23:00
> À: [hidden email]
> Objet: Re: Create separate thread pool for query execution
>
> How about long running compute? Do we need a separate thread pool for it as
> well?
>
> On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <[hidden email]>
> wrote:
>
> > +1
> >
> > Sergi
> >
> > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> >
> > > Igniters,
> > >
> > > Currently if several long-running queries are submitted, it may stall
> all
> > > cache operations because all theads in the system pool will be busy
> with
> > > queries.
> > >
> > > I think it makes sense to move queries execution into separate
> dedicated
> > > thread pool. As we have timeouts for core pool threads now, it should
> not
> > > affect memory consumption or startup time anyhow. Thoughts?
> > >
> > > Vladimir.
> > >
> >
> "Misys" is the trade name of the Misys group of companies. This email and
> any attachments have been scanned for known viruses using multiple
> scanners. This email message is intended for the named recipient only. It
> may be privileged and/or confidential. If you are not the named recipient
> of this email please notify us immediately and do not copy it or use it for
> any purpose, nor disclose its contents to any other person. This email does
> not constitute the commencement of legal relations between you and Misys.
> Please refer to the executed contract between you and the relevant member
> of the Misys group for the identity of the contracting party with which you
> are dealing.
>

--
Sergey Kozlov
GridGain Systems
www.gridgain.com

dsetrakyan

Re: Create separate thread pool for query execution

Sergey, I think we already have a general approach with 3 or 4 thread pools
we have today.

As far as the query thread pool, how many threads should be in it by
default? What happens is the query is run from compute task - will the
execution be shifted from the public to the query thread pool?

D.

On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov <[hidden email]> wrote:

> Hi
>
> It seems we need a set of dedicated pools for various purposes. Could we
> design a general apporach for this like following:
> - define dedicated pools in the grid configuration
> - run a query/compute/etc in the particular pool
>
> On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Romain,
> > We do not pre-start threads in our pools on Ignite start. Moreover, idle
> > threads are stopped after some timeout.
> >
> > 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <
> > [hidden email]>
> > написал:
> >
> > Hi igniters,
> > I'm not against to create a thread pool for each features but I have some
> > difficulty to minimize the number of threads required to start ignite...
> If
> > you add too many thread pools does this will not increase the number of
> > threads at startup?
> >
> > Thanks.
> >
> > Romain
> >
> >
> > ________________________________
> > De: Dmitriy Setrakyan <[hidden email]>
> > Envoyé: 23 oct. 2016 23:00
> > À: [hidden email]
> > Objet: Re: Create separate thread pool for query execution
> >
> > How about long running compute? Do we need a separate thread pool for it
> as
> > well?
> >
> > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <
> [hidden email]>
> > wrote:
> >
> > > +1
> > >
> > > Sergi
> > >
> > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> > >
> > > > Igniters,
> > > >
> > > > Currently if several long-running queries are submitted, it may stall
> > all
> > > > cache operations because all theads in the system pool will be busy
> > with
> > > > queries.
> > > >
> > > > I think it makes sense to move queries execution into separate
> > dedicated
> > > > thread pool. As we have timeouts for core pool threads now, it should
> > not
> > > > affect memory consumption or startup time anyhow. Thoughts?
> > > >
> > > > Vladimir.
> > > >
> > >
> > "Misys" is the trade name of the Misys group of companies. This email and
> > any attachments have been scanned for known viruses using multiple
> > scanners. This email message is intended for the named recipient only. It
> > may be privileged and/or confidential. If you are not the named recipient
> > of this email please notify us immediately and do not copy it or use it
> for
> > any purpose, nor disclose its contents to any other person. This email
> does
> > not constitute the commencement of legal relations between you and Misys.
> > Please refer to the executed contract between you and the relevant member
> > of the Misys group for the identity of the contracting party with which
> you
> > are dealing.
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

yzhdanov

Re: Create separate thread pool for query execution

> As far as the query thread pool, how many threads should be in it by
> default? What happens is the query is run from compute task - will the
> execution be shifted from the public to the query thread pool?

Pool management should be the same as for other pools.

Remote executions should be executed in query pool. Local should run
synchronously.

--Yakov

2016-10-24 11:53 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:

> Sergey, I think we already have a general approach with 3 or 4 thread pools
> we have today.
>
> As far as the query thread pool, how many threads should be in it by
> default? What happens is the query is run from compute task - will the
> execution be shifted from the public to the query thread pool?
>
> D.
>
> On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov <[hidden email]>
> wrote:
>
> > Hi
> >
> > It seems we need a set of dedicated pools for various purposes. Could we
> > design a general apporach for this like following:
> > - define dedicated pools in the grid configuration
> > - run a query/compute/etc in the particular pool
> >
> > On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Romain,
> > > We do not pre-start threads in our pools on Ignite start. Moreover,
> idle
> > > threads are stopped after some timeout.
> > >
> > > 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <
> > > [hidden email]>
> > > написал:
> > >
> > > Hi igniters,
> > > I'm not against to create a thread pool for each features but I have
> some
> > > difficulty to minimize the number of threads required to start
> ignite...
> > If
> > > you add too many thread pools does this will not increase the number of
> > > threads at startup?
> > >
> > > Thanks.
> > >
> > > Romain
> > >
> > >
> > > ________________________________
> > > De: Dmitriy Setrakyan <[hidden email]>
> > > Envoyé: 23 oct. 2016 23:00
> > > À: [hidden email]
> > > Objet: Re: Create separate thread pool for query execution
> > >
> > > How about long running compute? Do we need a separate thread pool for
> it
> > as
> > > well?
> > >
> > > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <
> > [hidden email]>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Sergi
> > > >
> > > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> > > >
> > > > > Igniters,
> > > > >
> > > > > Currently if several long-running queries are submitted, it may
> stall
> > > all
> > > > > cache operations because all theads in the system pool will be busy
> > > with
> > > > > queries.
> > > > >
> > > > > I think it makes sense to move queries execution into separate
> > > dedicated
> > > > > thread pool. As we have timeouts for core pool threads now, it
> should
> > > not
> > > > > affect memory consumption or startup time anyhow. Thoughts?
> > > > >
> > > > > Vladimir.
> > > > >
> > > >
> > > "Misys" is the trade name of the Misys group of companies. This email
> and
> > > any attachments have been scanned for known viruses using multiple
> > > scanners. This email message is intended for the named recipient only.
> It
> > > may be privileged and/or confidential. If you are not the named
> recipient
> > > of this email please notify us immediately and do not copy it or use it
> > for
> > > any purpose, nor disclose its contents to any other person. This email
> > does
> > > not constitute the commencement of legal relations between you and
> Misys.
> > > Please refer to the executed contract between you and the relevant
> member
> > > of the Misys group for the identity of the contracting party with which
> > you
> > > are dealing.
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>

dsetrakyan

Re: Create separate thread pool for query execution

I think this makes sense. Do we have a ticket filed?

On Mon, Oct 24, 2016 at 2:17 AM, Yakov Zhdanov <[hidden email]> wrote:

> > As far as the query thread pool, how many threads should be in it by
> > default? What happens is the query is run from compute task - will the
> > execution be shifted from the public to the query thread pool?
>
> Pool management should be the same as for other pools.
>
> Remote executions should be executed in query pool. Local should run
> synchronously.
>
> --Yakov
>
> 2016-10-24 11:53 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
>
> > Sergey, I think we already have a general approach with 3 or 4 thread
> pools
> > we have today.
> >
> > As far as the query thread pool, how many threads should be in it by
> > default? What happens is the query is run from compute task - will the
> > execution be shifted from the public to the query thread pool?
> >
> > D.
> >
> > On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov <[hidden email]>
> > wrote:
> >
> > > Hi
> > >
> > > It seems we need a set of dedicated pools for various purposes. Could
> we
> > > design a general apporach for this like following:
> > > - define dedicated pools in the grid configuration
> > > - run a query/compute/etc in the particular pool
> > >
> > > On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <[hidden email]
> >
> > > wrote:
> > >
> > > > Romain,
> > > > We do not pre-start threads in our pools on Ignite start. Moreover,
> > idle
> > > > threads are stopped after some timeout.
> > > >
> > > > 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <
> > > > [hidden email]>
> > > > написал:
> > > >
> > > > Hi igniters,
> > > > I'm not against to create a thread pool for each features but I have
> > some
> > > > difficulty to minimize the number of threads required to start
> > ignite...
> > > If
> > > > you add too many thread pools does this will not increase the number
> of
> > > > threads at startup?
> > > >
> > > > Thanks.
> > > >
> > > > Romain
> > > >
> > > >
> > > > ________________________________
> > > > De: Dmitriy Setrakyan <[hidden email]>
> > > > Envoyé: 23 oct. 2016 23:00
> > > > À: [hidden email]
> > > > Objet: Re: Create separate thread pool for query execution
> > > >
> > > > How about long running compute? Do we need a separate thread pool for
> > it
> > > as
> > > > well?
> > > >
> > > > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Sergi
> > > > >
> > > > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]>:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > Currently if several long-running queries are submitted, it may
> > stall
> > > > all
> > > > > > cache operations because all theads in the system pool will be
> busy
> > > > with
> > > > > > queries.
> > > > > >
> > > > > > I think it makes sense to move queries execution into separate
> > > > dedicated
> > > > > > thread pool. As we have timeouts for core pool threads now, it
> > should
> > > > not
> > > > > > affect memory consumption or startup time anyhow. Thoughts?
> > > > > >
> > > > > > Vladimir.
> > > > > >
> > > > >
> > > > "Misys" is the trade name of the Misys group of companies. This email
> > and
> > > > any attachments have been scanned for known viruses using multiple
> > > > scanners. This email message is intended for the named recipient
> only.
> > It
> > > > may be privileged and/or confidential. If you are not the named
> > recipient
> > > > of this email please notify us immediately and do not copy it or use
> it
> > > for
> > > > any purpose, nor disclose its contents to any other person. This
> email
> > > does
> > > > not constitute the commencement of legal relations between you and
> > Misys.
> > > > Please refer to the executed contract between you and the relevant
> > member
> > > > of the Misys group for the identity of the contracting party with
> which
> > > you
> > > > are dealing.
> > > >
> > >
> > >
> > >
> > > --
> > > Sergey Kozlov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>

Vladimir Ozerov

Re: Create separate thread pool for query execution

Yes, https://issues.apache.org/jira/browse/IGNITE-4105

On Mon, Oct 24, 2016 at 7:22 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> I think this makes sense. Do we have a ticket filed?
>
> On Mon, Oct 24, 2016 at 2:17 AM, Yakov Zhdanov <[hidden email]>
> wrote:
>
> > > As far as the query thread pool, how many threads should be in it by
> > > default? What happens is the query is run from compute task - will the
> > > execution be shifted from the public to the query thread pool?
> >
> > Pool management should be the same as for other pools.
> >
> > Remote executions should be executed in query pool. Local should run
> > synchronously.
> >
> > --Yakov
> >
> > 2016-10-24 11:53 GMT+03:00 Dmitriy Setrakyan <[hidden email]>:
> >
> > > Sergey, I think we already have a general approach with 3 or 4 thread
> > pools
> > > we have today.
> > >
> > > As far as the query thread pool, how many threads should be in it by
> > > default? What happens is the query is run from compute task - will the
> > > execution be shifted from the public to the query thread pool?
> > >
> > > D.
> > >
> > > On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov <[hidden email]>
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > It seems we need a set of dedicated pools for various purposes. Could
> > we
> > > > design a general apporach for this like following:
> > > > - define dedicated pools in the grid configuration
> > > > - run a query/compute/etc in the particular pool
> > > >
> > > > On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Romain,
> > > > > We do not pre-start threads in our pools on Ignite start. Moreover,
> > > idle
> > > > > threads are stopped after some timeout.
> > > > >
> > > > > 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <
> > > > > [hidden email]>
> > > > > написал:
> > > > >
> > > > > Hi igniters,
> > > > > I'm not against to create a thread pool for each features but I
> have
> > > some
> > > > > difficulty to minimize the number of threads required to start
> > > ignite...
> > > > If
> > > > > you add too many thread pools does this will not increase the
> number
> > of
> > > > > threads at startup?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Romain
> > > > >
> > > > >
> > > > > ________________________________
> > > > > De: Dmitriy Setrakyan <[hidden email]>
> > > > > Envoyé: 23 oct. 2016 23:00
> > > > > À: [hidden email]
> > > > > Objet: Re: Create separate thread pool for query execution
> > > > >
> > > > > How about long running compute? Do we need a separate thread pool
> for
> > > it
> > > > as
> > > > > well?
> > > > >
> > > > > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <
> > > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Sergi
> > > > > >
> > > > > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]
> >:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > Currently if several long-running queries are submitted, it may
> > > stall
> > > > > all
> > > > > > > cache operations because all theads in the system pool will be
> > busy
> > > > > with
> > > > > > > queries.
> > > > > > >
> > > > > > > I think it makes sense to move queries execution into separate
> > > > > dedicated
> > > > > > > thread pool. As we have timeouts for core pool threads now, it
> > > should
> > > > > not
> > > > > > > affect memory consumption or startup time anyhow. Thoughts?
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > >
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> > only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > it
> > > > for
> > > > > any purpose, nor disclose its contents to any other person. This
> > email
> > > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > which
> > > > you
> > > > > are dealing.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sergey Kozlov
> > > > GridGain Systems
> > > > www.gridgain.com
> > > >
> > >
> >
>

Andrey Kornev

Re: Create separate thread pool for query execution

In reply to this post by dsetrakyan

Guys,
I feel very uneasy about proliferation of thread pools in Ignite. I suggest we take step back.
Most of the users (including yours truly) have no idea how to correctly configure the existing thread pools, what those thread pools are for (like, management thread pool, or marshaller cache thread pool, or async callback thread pool, or peer class loading thread pool, or - my personal favorite - utility cache thread pool, just to name a few), and why they should worry about them altogether.
In reality, there should probably be only 2 parameters exposed at the configuration level: the size of Ignite's internal (or "system") thread pool, and the size of the user thread pool. Or alternatively, one number could indicate the total number of thread available to Ignite (hard upper bound) and the other would be a ratio of system threads to user threads. Either way, it's Ignite's internal business how to manage the thread pools given the constraints. For example, Ignite may choose to allocate the system threads across twenty thread pools, or maybe just one -- the users do not care. What users do care about is the size of the user thread pool because it's the one that executes their code. I know it's not the case in the current version, but going forward, Ignite must ensure that all user code (including all sorts of listeners and callbacks) is executed on the user threads.
In any case, it's the user who is ultimately in charge of figuring out the correct thread pool sizes, and I believe having just 2 knobs to turn vs. 7 or 8 as is the case today, would make things a lot simpler and reduce the guess work.
Speaking of query execution, it feels natural to have it executed on the user thread pool, as the query is a user-triggered action executing a user's code (even though the code is not Java in this case, but something that looks rather like SQL).
Regards,
Andrey
_____________________________
From: Dmitriy Setrakyan <[hidden email]<mailto:[hidden email]>>
Sent: Monday, October 24, 2016 9:23 AM
Subject: Re: Create separate thread pool for query execution
To: <[hidden email]<mailto:[hidden email]>>

I think this makes sense. Do we have a ticket filed?

On Mon, Oct 24, 2016 at 2:17 AM, Yakov Zhdanov <[hidden email]<mailto:[hidden email]>> wrote:

> > As far as the query thread pool, how many threads should be in it by
> > default? What happens is the query is run from compute task - will the
> > execution be shifted from the public to the query thread pool?
>
> Pool management should be the same as for other pools.
>
> Remote executions should be executed in query pool. Local should run
> synchronously.
>
> --Yakov
>
> 2016-10-24 11:53 GMT+03:00 Dmitriy Setrakyan <[hidden email]<mailto:[hidden email]>>:
>
> > Sergey, I think we already have a general approach with 3 or 4 thread
> pools
> > we have today.
> >
> > As far as the query thread pool, how many threads should be in it by
> > default? What happens is the query is run from compute task - will the
> > execution be shifted from the public to the query thread pool?
> >
> > D.
> >
> > On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov <[hidden email]<mailto:[hidden email]>>
> > wrote:
> >
> > > Hi
> > >
> > > It seems we need a set of dedicated pools for various purposes. Could
> we
> > > design a general apporach for this like following:
> > > - define dedicated pools in the grid configuration
> > > - run a query/compute/etc in the particular pool
> > >
> > > On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <[hidden email]<mailto:[hidden email]>
> >
> > > wrote:
> > >
> > > > Romain,
> > > > We do not pre-start threads in our pools on Ignite start. Moreover,
> > idle
> > > > threads are stopped after some timeout.
> > > >
> > > > 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <
> > > > [hidden email]<mailto:[hidden email]>>
> > > > написал:
> > > >
> > > > Hi igniters,
> > > > I'm not against to create a thread pool for each features but I have
> > some
> > > > difficulty to minimize the number of threads required to start
> > ignite...
> > > If
> > > > you add too many thread pools does this will not increase the number
> of
> > > > threads at startup?
> > > >
> > > > Thanks.
> > > >
> > > > Romain
> > > >
> > > >
> > > > ________________________________
> > > > De: Dmitriy Setrakyan <[hidden email]<mailto:[hidden email]>>
> > > > Envoye: 23 oct. 2016 23:00
> > > > A: [hidden email]<mailto:[hidden email]>
> > > > Objet: Re: Create separate thread pool for query execution
> > > >
> > > > How about long running compute? Do we need a separate thread pool for
> > it
> > > as
> > > > well?
> > > >
> > > > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <
> > > [hidden email]<mailto:[hidden email]>>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Sergi
> > > > >
> > > > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]<mailto:[hidden email]>>:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > Currently if several long-running queries are submitted, it may
> > stall
> > > > all
> > > > > > cache operations because all theads in the system pool will be
> busy
> > > > with
> > > > > > queries.
> > > > > >
> > > > > > I think it makes sense to move queries execution into separate
> > > > dedicated
> > > > > > thread pool. As we have timeouts for core pool threads now, it
> > should
> > > > not
> > > > > > affect memory consumption or startup time anyhow. Thoughts?
> > > > > >
> > > > > > Vladimir.
> > > > > >
> > > > >
> > > > "Misys" is the trade name of the Misys group of companies. This email
> > and
> > > > any attachments have been scanned for known viruses using multiple
> > > > scanners. This email message is intended for the named recipient
> only.
> > It
> > > > may be privileged and/or confidential. If you are not the named
> > recipient
> > > > of this email please notify us immediately and do not copy it or use
> it
> > > for
> > > > any purpose, nor disclose its contents to any other person. This
> email
> > > does
> > > > not constitute the commencement of legal relations between you and
> > Misys.
> > > > Please refer to the executed contract between you and the relevant
> > member
> > > > of the Misys group for the identity of the contracting party with
> which
> > > you
> > > > are dealing.
> > > >
> > >
> > >
> > >
> > > --
> > > Sergey Kozlov
> > > GridGain Systems
> > > www.gridgain.com<http://www.gridgain.com>
> > >
> >
>

Vladimir Ozerov

Re: Create separate thread pool for query execution

Andrey,

Ignite already works the way you described except of listeners/callbacks.
Only size of each thread pool is exposed to configuration. And each one
already have sensible default value, so normally users do not think of it
at all. For some pools we also have timeouts, but they will be deprecated
soon.

Queries cannot be executed in system pool because they can block cache
operations. And they cannot be executed in public pool because in this case
it will be impossible to run queries from closures and tasks.

Vladimir.

On Wed, Oct 26, 2016 at 8:37 AM, Andrey Kornev <[hidden email]>
wrote:

> Guys,
> I feel very uneasy about proliferation of thread pools in Ignite. I
> suggest we take step back.
> Most of the users (including yours truly) have no idea how to correctly
> configure the existing thread pools, what those thread pools are for (like,
> management thread pool, or marshaller cache thread pool, or async callback
> thread pool, or peer class loading thread pool, or - my personal favorite -
> utility cache thread pool, just to name a few), and why they should worry
> about them altogether.
> In reality, there should probably be only 2 parameters exposed at the
> configuration level: the size of Ignite's internal (or "system") thread
> pool, and the size of the user thread pool. Or alternatively, one number
> could indicate the total number of thread available to Ignite (hard upper
> bound) and the other would be a ratio of system threads to user threads.
> Either way, it's Ignite's internal business how to manage the thread pools
> given the constraints. For example, Ignite may choose to allocate the
> system threads across twenty thread pools, or maybe just one -- the users
> do not care. What users do care about is the size of the user thread pool
> because it's the one that executes their code. I know it's not the case in
> the current version, but going forward, Ignite must ensure that all user
> code (including all sorts of listeners and callbacks) is executed on the
> user threads.
> In any case, it's the user who is ultimately in charge of figuring out the
> correct thread pool sizes, and I believe having just 2 knobs to turn vs. 7
> or 8 as is the case today, would make things a lot simpler and reduce the
> guess work.
> Speaking of query execution, it feels natural to have it executed on the
> user thread pool, as the query is a user-triggered action executing a
> user's code (even though the code is not Java in this case, but something
> that looks rather like SQL).
> Regards,
> Andrey
> _____________________________
> From: Dmitriy Setrakyan <[hidden email]<mailto:
> [hidden email]>>
> Sent: Monday, October 24, 2016 9:23 AM
> Subject: Re: Create separate thread pool for query execution
> To: <[hidden email]<mailto:[hidden email]>>
>
>
> I think this makes sense. Do we have a ticket filed?
>
> On Mon, Oct 24, 2016 at 2:17 AM, Yakov Zhdanov <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> > > As far as the query thread pool, how many threads should be in it by
> > > default? What happens is the query is run from compute task - will the
> > > execution be shifted from the public to the query thread pool?
> >
> > Pool management should be the same as for other pools.
> >
> > Remote executions should be executed in query pool. Local should run
> > synchronously.
> >
> > --Yakov
> >
> > 2016-10-24 11:53 GMT+03:00 Dmitriy Setrakyan <[hidden email]
> <mailto:[hidden email]>>:
> >
> > > Sergey, I think we already have a general approach with 3 or 4 thread
> > pools
> > > we have today.
> > >
> > > As far as the query thread pool, how many threads should be in it by
> > > default? What happens is the query is run from compute task - will the
> > > execution be shifted from the public to the query thread pool?
> > >
> > > D.
> > >
> > > On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov <[hidden email]
> <mailto:[hidden email]>>
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > It seems we need a set of dedicated pools for various purposes. Could
> > we
> > > > design a general apporach for this like following:
> > > > - define dedicated pools in the grid configuration
> > > > - run a query/compute/etc in the particular pool
> > > >
> > > > On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <
> [hidden email]<mailto:[hidden email]>
> > >
> > > > wrote:
> > > >
> > > > > Romain,
> > > > > We do not pre-start threads in our pools on Ignite start. Moreover,
> > > idle
> > > > > threads are stopped after some timeout.
> > > > >
> > > > > 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <
> > > > > [hidden email]<mailto:[hidden email]>>
> > > > > написал:
> > > > >
> > > > > Hi igniters,
> > > > > I'm not against to create a thread pool for each features but I
> have
> > > some
> > > > > difficulty to minimize the number of threads required to start
> > > ignite...
> > > > If
> > > > > you add too many thread pools does this will not increase the
> number
> > of
> > > > > threads at startup?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Romain
> > > > >
> > > > >
> > > > > ________________________________
> > > > > De: Dmitriy Setrakyan <[hidden email]<mailto:
> [hidden email]>>
> > > > > Envoye: 23 oct. 2016 23:00
> > > > > A: [hidden email]<mailto:[hidden email]>
> > > > > Objet: Re: Create separate thread pool for query execution
> > > > >
> > > > > How about long running compute? Do we need a separate thread pool
> for
> > > it
> > > > as
> > > > > well?
> > > > >
> > > > > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <
> > > > [hidden email]<mailto:[hidden email]>>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Sergi
> > > > > >
> > > > > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <[hidden email]
> <mailto:[hidden email]>>:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > Currently if several long-running queries are submitted, it may
> > > stall
> > > > > all
> > > > > > > cache operations because all theads in the system pool will be
> > busy
> > > > > with
> > > > > > > queries.
> > > > > > >
> > > > > > > I think it makes sense to move queries execution into separate
> > > > > dedicated
> > > > > > > thread pool. As we have timeouts for core pool threads now, it
> > > should
> > > > > not
> > > > > > > affect memory consumption or startup time anyhow. Thoughts?
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > >
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> > only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > it
> > > > for
> > > > > any purpose, nor disclose its contents to any other person. This
> > email
> > > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > which
> > > > you
> > > > > are dealing.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sergey Kozlov
> > > > GridGain Systems
> > > > www.gridgain.com<http://www.gridgain.com>
> > > >
> > >
> >
>
>
>

Andrey Kornev

Re: Create separate thread pool for query execution

Vladimir,

Thanks for your reply!

Unfortunately, most of the time users end up having to think about configuring the thread pools, because the "sensible" default value for most of the thread pools is the number of processors multiplied by 2. With modern production-grade multicore machines, the default thread pools size is simply too high.

Another problem with the current approach to thread management is that some resources inevitably go wasted. For example, if one thread pool runs out of threads it won't be able to borrow from another thread pool even though that one may have plenty of threads sitting idle.

And even if there were really good reasons for maintaining all those thread pools, complete lack of any documentation makes it pretty difficult (if not impossible) to configure the thread pools correctly.

Finally, your explanation as to why another thread pool - dedicated exclusively to queries - is necessary, doesn't explain anything, but rather raises more questions:

- why would a query block cache operations? Even if it absolutely must block, so what? In this scenario, how is a query different from another "regular" cache operation that may also block some other cache operations? Why special treatment?

- why wouldn't it possible to run queries from the Ignite closures and tasks? I don't see any connection.

Thanks

Andrey

________________________________
From: Vladimir Ozerov <[hidden email]>
Sent: Tuesday, October 25, 2016 11:40 PM
To: [hidden email]
Subject: Re: Create separate thread pool for query execution

Andrey,

Ignite already works the way you described except of listeners/callbacks.
Only size of each thread pool is exposed to configuration. And each one
already have sensible default value, so normally users do not think of it
at all. For some pools we also have timeouts, but they will be deprecated
soon.

Queries cannot be executed in system pool because they can block cache
operations. And they cannot be executed in public pool because in this case
it will be impossible to run queries from closures and tasks.

Vladimir.

On Wed, Oct 26, 2016 at 8:37 AM, Andrey Kornev <[hidden email]>
wrote:

Vladimir Ozerov

Re: Create separate thread pool for query execution

Andrey,

I am not aware of any serious issues with thread pools configuration. For
cache pool ("system pool") [cores * 2] makes sense, because usually cache
operations are fast and small, but contains blocking operations around
(e.g. moving task from blocking queue to the thread). Sometimes [cores * 2]
could be too much for public pool, but again - I do not know any serious
problems with it except of slight performance degradation.

The main idea of many thread pool is to allow users to perform synchronous
operations to one component from within another. For sure, there is a
general solution to the problem - reactive style or continuations. When
every potentially blocking operation releases current thread and allow it
to perform other stuff. Something like ComputeJobContinuation.holdcc/callcc,
but performed implicitly. In this case it would be enough to have a single
thread pool for everything. The problem is that this approach is too
complex for both users and Ignite developers. This is the main reason why
we have multiple pools - essentially we create separate execution circuits
for different components, thus allowing them interact with each other in
blocking fashion. Personally, I am completely satisfied with this approach,
because it is simple for users.

What important here, is that we are not trying to support each and every
use case. Instead, we are trying to cover some widely-used scenarios, such
as "call cache from compute".

Queries cannot be run in public pool, because tasks and closures are also
executed in that pool. It could lead to starvation when query is started
from compute job/closure.

Vladimir.

On Thu, Oct 27, 2016 at 4:40 AM, Andrey Kornev <[hidden email]>
wrote:

> Vladimir,
>
>
> Thanks for your reply!
>
>
> Unfortunately, most of the time users end up having to think about
> configuring the thread pools, because the "sensible" default value for most
> of the thread pools is the number of processors multiplied by 2. With
> modern production-grade multicore machines, the default thread pools size
> is simply too high.
>
>
> Another problem with the current approach to thread management is that
> some resources inevitably go wasted. For example, if one thread pool runs
> out of threads it won't be able to borrow from another thread pool even
> though that one may have plenty of threads sitting idle.
>
>
> And even if there were really good reasons for maintaining all those
> thread pools, complete lack of any documentation makes it pretty difficult
> (if not impossible) to configure the thread pools correctly.
>
>
> Finally, your explanation as to why another thread pool - dedicated
> exclusively to queries - is necessary, doesn't explain anything, but rather
> raises more questions:
>
> - why would a query block cache operations? Even if it absolutely must
> block, so what? In this scenario, how is a query different from another
> "regular" cache operation that may also block some other cache operations?
> Why special treatment?
>
> - why wouldn't it possible to run queries from the Ignite closures and
> tasks? I don't see any connection.
>
>
> Thanks
>
> Andrey
>
> ________________________________
> From: Vladimir Ozerov <[hidden email]>
> Sent: Tuesday, October 25, 2016 11:40 PM
> To: [hidden email]
> Subject: Re: Create separate thread pool for query execution
>
> Andrey,
>
> Ignite already works the way you described except of listeners/callbacks.
> Only size of each thread pool is exposed to configuration. And each one
> already have sensible default value, so normally users do not think of it
> at all. For some pools we also have timeouts, but they will be deprecated
> soon.
>
> Queries cannot be executed in system pool because they can block cache
> operations. And they cannot be executed in public pool because in this case
> it will be impossible to run queries from closures and tasks.
>
> Vladimir.
>
> On Wed, Oct 26, 2016 at 8:37 AM, Andrey Kornev <[hidden email]>
> wrote:
>
> > Guys,
> > I feel very uneasy about proliferation of thread pools in Ignite. I
> > suggest we take step back.
> > Most of the users (including yours truly) have no idea how to correctly
> > configure the existing thread pools, what those thread pools are for
> (like,
> > management thread pool, or marshaller cache thread pool, or async
> callback
> > thread pool, or peer class loading thread pool, or - my personal
> favorite -
> > utility cache thread pool, just to name a few), and why they should worry
> > about them altogether.
> > In reality, there should probably be only 2 parameters exposed at the
> > configuration level: the size of Ignite's internal (or "system") thread
> > pool, and the size of the user thread pool. Or alternatively, one number
> > could indicate the total number of thread available to Ignite (hard upper
> > bound) and the other would be a ratio of system threads to user threads.
> > Either way, it's Ignite's internal business how to manage the thread
> pools
> > given the constraints. For example, Ignite may choose to allocate the
> > system threads across twenty thread pools, or maybe just one -- the users
> > do not care. What users do care about is the size of the user thread pool
> > because it's the one that executes their code. I know it's not the case
> in
> > the current version, but going forward, Ignite must ensure that all user
> > code (including all sorts of listeners and callbacks) is executed on the
> > user threads.
> > In any case, it's the user who is ultimately in charge of figuring out
> the
> > correct thread pool sizes, and I believe having just 2 knobs to turn vs.
> 7
> > or 8 as is the case today, would make things a lot simpler and reduce the
> > guess work.
> > Speaking of query execution, it feels natural to have it executed on the
> > user thread pool, as the query is a user-triggered action executing a
> > user's code (even though the code is not Java in this case, but something
> > that looks rather like SQL).
> > Regards,
> > Andrey
> > _____________________________
> > From: Dmitriy Setrakyan <[hidden email]<mailto:
> > [hidden email]>>
> > Sent: Monday, October 24, 2016 9:23 AM
> > Subject: Re: Create separate thread pool for query execution
> > To: <[hidden email]<mailto:[hidden email]>>
> >
> >
> > I think this makes sense. Do we have a ticket filed?
> >
> > On Mon, Oct 24, 2016 at 2:17 AM, Yakov Zhdanov <[hidden email]
> > <mailto:[hidden email]>> wrote:
> >
> > > > As far as the query thread pool, how many threads should be in it by
> > > > default? What happens is the query is run from compute task - will
> the
> > > > execution be shifted from the public to the query thread pool?
> > >
> > > Pool management should be the same as for other pools.
> > >
> > > Remote executions should be executed in query pool. Local should run
> > > synchronously.
> > >
> > > --Yakov
> > >
> > > 2016-10-24 11:53 GMT+03:00 Dmitriy Setrakyan <[hidden email]
> > <mailto:[hidden email]>>:
> > >
> > > > Sergey, I think we already have a general approach with 3 or 4 thread
> > > pools
> > > > we have today.
> > > >
> > > > As far as the query thread pool, how many threads should be in it by
> > > > default? What happens is the query is run from compute task - will
> the
> > > > execution be shifted from the public to the query thread pool?
> > > >
> > > > D.
> > > >
> > > > On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov <[hidden email]
> > <mailto:[hidden email]>>
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > It seems we need a set of dedicated pools for various purposes.
> Could
> > > we
> > > > > design a general apporach for this like following:
> > > > > - define dedicated pools in the grid configuration
> > > > > - run a query/compute/etc in the particular pool
> > > > >
> > > > > On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <
> > [hidden email]<mailto:[hidden email]>
> > > >
> > > > > wrote:
> > > > >
> > > > > > Romain,
> > > > > > We do not pre-start threads in our pools on Ignite start.
> Moreover,
> > > > idle
> > > > > > threads are stopped after some timeout.
> > > > > >
> > > > > > 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" <
> > > > > > [hidden email]<mailto:[hidden email]>>
> > > > > > написал:
> > > > > >
> > > > > > Hi igniters,
> > > > > > I'm not against to create a thread pool for each features but I
> > have
> > > > some
> > > > > > difficulty to minimize the number of threads required to start
> > > > ignite...
> > > > > If
> > > > > > you add too many thread pools does this will not increase the
> > number
> > > of
> > > > > > threads at startup?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Romain
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > De: Dmitriy Setrakyan <[hidden email]<mailto:
> > [hidden email]>>
> > > > > > Envoye: 23 oct. 2016 23:00
> > > > > > A: [hidden email]<mailto:[hidden email]>
> > > > > > Objet: Re: Create separate thread pool for query execution
> > > > > >
> > > > > > How about long running compute? Do we need a separate thread pool
> > for
> > > > it
> > > > > as
> > > > > > well?
> > > > > >
> > > > > > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <
> > > > > [hidden email]<mailto:[hidden email]>>
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Sergi
> > > > > > >
> > > > > > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <
> [hidden email]
> > <mailto:[hidden email]>>:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > Currently if several long-running queries are submitted, it
> may
> > > > stall
> > > > > > all
> > > > > > > > cache operations because all theads in the system pool will
> be
> > > busy
> > > > > > with
> > > > > > > > queries.
> > > > > > > >
> > > > > > > > I think it makes sense to move queries execution into
> separate
> > > > > > dedicated
> > > > > > > > thread pool. As we have timeouts for core pool threads now,
> it
> > > > should
> > > > > > not
> > > > > > > > affect memory consumption or startup time anyhow. Thoughts?
> > > > > > > >
> > > > > > > > Vladimir.
> > > > > > > >
> > > > > > >
> > > > > > "Misys" is the trade name of the Misys group of companies. This
> > email
> > > > and
> > > > > > any attachments have been scanned for known viruses using
> multiple
> > > > > > scanners. This email message is intended for the named recipient
> > > only.
> > > > It
> > > > > > may be privileged and/or confidential. If you are not the named
> > > > recipient
> > > > > > of this email please notify us immediately and do not copy it or
> > use
> > > it
> > > > > for
> > > > > > any purpose, nor disclose its contents to any other person. This
> > > email
> > > > > does
> > > > > > not constitute the commencement of legal relations between you
> and
> > > > Misys.
> > > > > > Please refer to the executed contract between you and the
> relevant
> > > > member
> > > > > > of the Misys group for the identity of the contracting party with
> > > which
> > > > > you
> > > > > > are dealing.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sergey Kozlov
> > > > > GridGain Systems
> > > > > www.gridgain.com<http://www.gridgain.com>
> > > > >
> > > >
> > >
> >
> >
> >
>