Re: Runtime.availableProcessors() returns hardware's CPU count which is the issue with Ignite in Kubernetes

[yzhdanov]

Cross-posting to dev list.

Guys,

Suggestion below makes sense to me. Filed a ticket
https://issues.apache.org/jira/browse/IGNITE-7310

Perhaps, Arseny would like to provide a PR himself ;)

--Yakov

2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk <[hidden email]>:

> Hi guys.
>
> Ignite configures all thread pools, selectors, etc. basing on Runtime.availableProcessors()
> which seems not correct in containerized environment. In Kubernetes with
> Docker that method returns CPU count of a Node/machine, which is 64 in our
> particular case. But those 64 CPU and their timings are shared between
> other stuff on the node like other Pods and services. Appropriate value of
> available cores for Pod is usually configured as CPU Resource and estimated
> basing on different things taking performance into account. General idea,
> if you want to run several Pods on the same node, they all should request
> less resources then the node provides. So, we give 4-8 cores for Ignite
> instance in Kubernetes, but Ignite's thread pools are configured like they
> get all 64 CPUs, and in turn we get a lot of threads for the Pod with 4-8
> cores available.
>
> Now we manually set appropriate values for all available properties which
> relate to thread pools.
>
> Would it be correct to have one environment variable, say
> IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for those
> configurations and by default equals to Runtime.availableProcessors()?
>
> Thanks.
>
> ​
> Arseny Kovalchuk
>
> Senior Software Engineer at Synesis
> skype: arseny.kovalchuk
> mobile: [phone]
> ​LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>​
>

[Stanislav Lukyanov]

Hi Arseny,

This behavior of the `Runtime.availableProcessors()` is actually a recognized issue of the Hotspot, see [1]. It was fixed not that long ago for JDK 9 and 8uX, and I can see correct values returned in my Docker environment with JDK 8u151, although I believe it depends on a specific way a container is configured.

Which Java version do you use? Can you try your code on JDK 8u151?

BTW see also [2] and [3] on more stuff to be fixed in JDK for better container support.

Thanks,
Stan

[1] https://bugs.openjdk.java.net/browse/JDK-6515172
[2] https://bugs.openjdk.java.net/browse/JDK-8146115
[3] https://bugs.openjdk.java.net/browse/JDK-8182070


From: Yakov Zhdanov
Sent: 26 декабря 2017 г. 16:05
To: [hidden email]; [hidden email]
Subject: Re: Runtime.availableProcessors() returns hardware's CPU count whichis the issue with Ignite in Kubernetes

Cross-posting to dev list.

Guys,

Suggestion below makes sense to me. Filed a ticket
https://issues.apache.org/jira/browse/IGNITE-7310

Perhaps, Arseny would like to provide a PR himself ;)

--Yakov

2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk <[hidden email]>:

> Hi guys.
>
> Ignite configures all thread pools, selectors, etc. basing on Runtime.availableProcessors()
> which seems not correct in containerized environment. In Kubernetes with
> Docker that method returns CPU count of a Node/machine, which is 64 in our
> particular case. But those 64 CPU and their timings are shared between
> other stuff on the node like other Pods and services. Appropriate value of
> available cores for Pod is usually configured as CPU Resource and estimated
> basing on different things taking performance into account. General idea,
> if you want to run several Pods on the same node, they all should request
> less resources then the node provides. So, we give 4-8 cores for Ignite
> instance in Kubernetes, but Ignite's thread pools are configured like they
> get all 64 CPUs, and in turn we get a lot of threads for the Pod with 4-8
> cores available.
>
> Now we manually set appropriate values for all available properties which
> relate to thread pools.
>
> Would it be correct to have one environment variable, say
> IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for those
> configurations and by default equals to Runtime.availableProcessors()?
>
> Thanks.
>
> ​
> Arseny Kovalchuk
>
> Senior Software Engineer at Synesis
> skype: arseny.kovalchuk
> mobile: [phone]
> ​LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>​
>

[Ilya Lantukh]

Hi Yakov,

I think that property IGNITE_NODES_PER_HOST, as you suggested, would be
confusing, because users might want to reduce amount of available resources
for ignite node not only because they run multiple nodes per host, but also
because they run other software. Also, in my opinion all types of system
resources (CPU, memory, network) shouldn't be scaled using the same value.

So I'd prefer to have IGNITE_CONCURRENCY_LEVEL or
IGNITE_AVAILABLE_PROCESSORS, as it was originally suggested.

On Tue, Dec 26, 2017 at 4:05 PM, Yakov Zhdanov <[hidden email]> wrote:

> Cross-posting to dev list.
>
> Guys,
>
> Suggestion below makes sense to me. Filed a ticket
> https://issues.apache.org/jira/browse/IGNITE-7310
>
> Perhaps, Arseny would like to provide a PR himself ;)
>
> --Yakov
>
> 2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk <[hidden email]>:
>
> > Hi guys.
> >
> > Ignite configures all thread pools, selectors, etc. basing on
> Runtime.availableProcessors()
> > which seems not correct in containerized environment. In Kubernetes with
> > Docker that method returns CPU count of a Node/machine, which is 64 in
> our
> > particular case. But those 64 CPU and their timings are shared between
> > other stuff on the node like other Pods and services. Appropriate value
> of
> > available cores for Pod is usually configured as CPU Resource and
> estimated
> > basing on different things taking performance into account. General idea,
> > if you want to run several Pods on the same node, they all should request
> > less resources then the node provides. So, we give 4-8 cores for Ignite
> > instance in Kubernetes, but Ignite's thread pools are configured like
> they
> > get all 64 CPUs, and in turn we get a lot of threads for the Pod with 4-8
> > cores available.
> >
> > Now we manually set appropriate values for all available properties which
> > relate to thread pools.
> >
> > Would it be correct to have one environment variable, say
> > IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for
> those
> > configurations and by default equals to Runtime.availableProcessors()?
> >
> > Thanks.
> >
> > ​
> > Arseny Kovalchuk
> >
> > Senior Software Engineer at Synesis
> > skype: arseny.kovalchuk
> > mobile: [phone]
> > ​LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>​
> >
>

--
Best regards,
Ilya

[Yakov Zhdanov-2]

Ilya, agree. I like IGNITE_AVAILABLE_CPU more.

Yakov Zhdanov,
www.gridgain.com

2017-12-26 16:36 GMT+03:00 Ilya Lantukh <[hidden email]>:

> Hi Yakov,
>
> I think that property IGNITE_NODES_PER_HOST, as you suggested, would be
> confusing, because users might want to reduce amount of available resources
> for ignite node not only because they run multiple nodes per host, but also
> because they run other software. Also, in my opinion all types of system
> resources (CPU, memory, network) shouldn't be scaled using the same value.
>
> So I'd prefer to have IGNITE_CONCURRENCY_LEVEL or
> IGNITE_AVAILABLE_PROCESSORS, as it was originally suggested.
>
> On Tue, Dec 26, 2017 at 4:05 PM, Yakov Zhdanov <[hidden email]>
> wrote:
>
>> Cross-posting to dev list.
>>
>> Guys,
>>
>> Suggestion below makes sense to me. Filed a ticket
>> https://issues.apache.org/jira/browse/IGNITE-7310
>>
>> Perhaps, Arseny would like to provide a PR himself ;)
>>
>> --Yakov
>>
>> 2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk <[hidden email]
>> >:
>>
>> > Hi guys.
>> >
>> > Ignite configures all thread pools, selectors, etc. basing on
>> Runtime.availableProcessors()
>> > which seems not correct in containerized environment. In Kubernetes with
>> > Docker that method returns CPU count of a Node/machine, which is 64 in
>> our
>> > particular case. But those 64 CPU and their timings are shared between
>> > other stuff on the node like other Pods and services. Appropriate value
>> of
>> > available cores for Pod is usually configured as CPU Resource and
>> estimated
>> > basing on different things taking performance into account. General
>> idea,
>> > if you want to run several Pods on the same node, they all should
>> request
>> > less resources then the node provides. So, we give 4-8 cores for Ignite
>> > instance in Kubernetes, but Ignite's thread pools are configured like
>> they
>> > get all 64 CPUs, and in turn we get a lot of threads for the Pod with
>> 4-8
>> > cores available.
>> >
>> > Now we manually set appropriate values for all available properties
>> which
>> > relate to thread pools.
>> >
>> > Would it be correct to have one environment variable, say
>> > IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for
>> those
>> > configurations and by default equals to Runtime.availableProcessors()?
>> >
>> > Thanks.
>> >
>> > ​
>> > Arseny Kovalchuk
>> >
>> > Senior Software Engineer at Synesis
>> > skype: arseny.kovalchuk
>> > mobile: [phone]
>> > ​LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>​
>> >
>>
>
>
>
> --
> Best regards,
> Ilya
>

[Arseny Kovalchuk]

Hi Stanislav.

We use OpenJDK and use Alpine Linux based images. See java version below.
In our environment availableProcessors returns CPU's for the host.

Did you mean to try Oracle's JDK 8u151?

arseny@kovalchuka-ubuntu:~/kipod-x$ ku exec ignite-instance-0 -ti bash
bash-4.4# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (IcedTea 3.6.0) (Alpine 8.151.12-r0)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
bash-4.4# jjs
jjs> print(java.lang.Runtime.runtime.availableProcessors());
40
jjs>


​
Arseny Kovalchuk

Senior Software Engineer at Synesis
skype: arseny.kovalchuk
mobile: [phone]
​LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>​

On 26 December 2017 at 16:56, Yakov Zhdanov <[hidden email]> wrote:

> Ilya, agree. I like IGNITE_AVAILABLE_CPU more.
>
> Yakov Zhdanov,
> www.gridgain.com
>
> 2017-12-26 16:36 GMT+03:00 Ilya Lantukh <[hidden email]>:
>
>> Hi Yakov,
>>
>> I think that property IGNITE_NODES_PER_HOST, as you suggested, would be
>> confusing, because users might want to reduce amount of available resources
>> for ignite node not only because they run multiple nodes per host, but also
>> because they run other software. Also, in my opinion all types of system
>> resources (CPU, memory, network) shouldn't be scaled using the same value.
>>
>> So I'd prefer to have IGNITE_CONCURRENCY_LEVEL or
>> IGNITE_AVAILABLE_PROCESSORS, as it was originally suggested.
>>
>> On Tue, Dec 26, 2017 at 4:05 PM, Yakov Zhdanov <[hidden email]>
>> wrote:
>>
>>> Cross-posting to dev list.
>>>
>>> Guys,
>>>
>>> Suggestion below makes sense to me. Filed a ticket
>>> https://issues.apache.org/jira/browse/IGNITE-7310
>>>
>>> Perhaps, Arseny would like to provide a PR himself ;)
>>>
>>> --Yakov
>>>
>>> 2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk <[hidden email]
>>> >:
>>>
>>> > Hi guys.
>>> >
>>> > Ignite configures all thread pools, selectors, etc. basing on
>>> Runtime.availableProcessors()
>>> > which seems not correct in containerized environment. In Kubernetes
>>> with
>>> > Docker that method returns CPU count of a Node/machine, which is 64 in
>>> our
>>> > particular case. But those 64 CPU and their timings are shared between
>>> > other stuff on the node like other Pods and services. Appropriate
>>> value of
>>> > available cores for Pod is usually configured as CPU Resource and
>>> estimated
>>> > basing on different things taking performance into account. General
>>> idea,
>>> > if you want to run several Pods on the same node, they all should
>>> request
>>> > less resources then the node provides. So, we give 4-8 cores for Ignite
>>> > instance in Kubernetes, but Ignite's thread pools are configured like
>>> they
>>> > get all 64 CPUs, and in turn we get a lot of threads for the Pod with
>>> 4-8
>>> > cores available.
>>> >
>>> > Now we manually set appropriate values for all available properties
>>> which
>>> > relate to thread pools.
>>> >
>>> > Would it be correct to have one environment variable, say
>>> > IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for
>>> those
>>> > configurations and by default equals to Runtime.availableProcessors()?
>>> >
>>> > Thanks.
>>> >
>>> > ​
>>> > Arseny Kovalchuk
>>> >
>>> > Senior Software Engineer at Synesis
>>> > skype: arseny.kovalchuk
>>> > mobile: [phone] <+375%2029%20666-16-16>
>>> > ​LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>​
>>> >
>>>
>>
>>
>>
>> --
>> Best regards,
>> Ilya
>>
>
>

[Stanislav Lukyanov]

Hi Arseny,

Both OpenJDK and Oracle JDK 8u151 should do.
I’ve checked and it seems that it is indeed about the specific way of invoking Docker. By default it doesn’t restrict container to particular CPUs, but you can make it do that by using `--cpuset-cpus` flag.

Example:
Without cpuset (Docker host has 4 CPUs)
    >docker run -it java-docker bash
    root@8a2cd9d06695:/usr/test# java -version
    openjdk version "1.8.0_151"
    OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
    OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
    root@8a2cd9d06695:/usr/test# jjs
    jjs> print(java.lang.Runtime.runtime.availableProcessors());
    4

With cpuset specifying the first two CPUs
    >docker run -it --cpuset-cpus=0,1 java-docker bash
    root@7c2723a9819e:/usr/test# jjs
    jjs> print(java.lang.Runtime.runtime.availableProcessors());
    2

Note also that by using cpuset you change the settings for the whole container, not just for the Ignite, so that other Java pools in the same JVM, e.g. parallel stream executor, should also work better.

Unfortunately, I’m not familiar with Kubernetes, but the manual [1] says that you can enable the use of cpusets via a static CPU management policy.

Hope that helps,
Stan

[1] https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/

From: Arseny Kovalchuk
Sent: 26 декабря 2017 г. 17:37
To: [hidden email]
Cc: [hidden email]
Subject: Re: Runtime.availableProcessors() returns hardware's CPU count whichis the issue with Ignite in Kubernetes

Hi Stanislav.

We use OpenJDK and use Alpine Linux based images. See java version below. In our environment availableProcessors returns CPU's for the host. 

Did you mean to try Oracle's JDK 8u151?
arseny@kovalchuka-ubuntu:~/kipod-x$ ku exec ignite-instance-0 -ti bash
bash-4.4# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (IcedTea 3.6.0) (Alpine 8.151.12-r0)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
bash-4.4# jjs
jjs> print(java.lang.Runtime.runtime.availableProcessors());
40
jjs> 



​
Arseny Kovalchuk

Senior Software Engineer at Synesis
skype: arseny.kovalchuk
mobile: [phone]
​LinkedIn Profile​

On 26 December 2017 at 16:56, Yakov Zhdanov <[hidden email]> wrote:
Ilya, agree. I like IGNITE_AVAILABLE_CPU more.


Yakov Zhdanov,
www.gridgain.com

2017-12-26 16:36 GMT+03:00 Ilya Lantukh <[hidden email]>:
Hi Yakov,

I think that property IGNITE_NODES_PER_HOST, as you suggested, would be confusing, because users might want to reduce amount of available resources for ignite node not only because they run multiple nodes per host, but also because they run other software. Also, in my opinion all types of system resources (CPU, memory, network) shouldn't be scaled using the same value.

So I'd prefer to have IGNITE_CONCURRENCY_LEVEL or IGNITE_AVAILABLE_PROCESSORS, as it was originally suggested.

On Tue, Dec 26, 2017 at 4:05 PM, Yakov Zhdanov <[hidden email]> wrote:
Cross-posting to dev list.

Guys,

Suggestion below makes sense to me. Filed a ticket
https://issues.apache.org/jira/browse/IGNITE-7310

Perhaps, Arseny would like to provide a PR himself ;)

--Yakov

2017-12-26 14:32 GMT+03:00 Arseny Kovalchuk <[hidden email]>:

> Hi guys.
>
> Ignite configures all thread pools, selectors, etc. basing on Runtime.availableProcessors()
> which seems not correct in containerized environment. In Kubernetes with
> Docker that method returns CPU count of a Node/machine, which is 64 in our
> particular case. But those 64 CPU and their timings are shared between
> other stuff on the node like other Pods and services. Appropriate value of
> available cores for Pod is usually configured as CPU Resource and estimated
> basing on different things taking performance into account. General idea,
> if you want to run several Pods on the same node, they all should request
> less resources then the node provides. So, we give 4-8 cores for Ignite
> instance in Kubernetes, but Ignite's thread pools are configured like they
> get all 64 CPUs, and in turn we get a lot of threads for the Pod with 4-8
> cores available.
>
> Now we manually set appropriate values for all available properties which
> relate to thread pools.
>
> Would it be correct to have one environment variable, say
> IGNITE_CONCURRENCY_LEVEL which will be used as a reference value for those
> configurations and by default equals to Runtime.availableProcessors()?
>
> Thanks.
>
> ​
> Arseny Kovalchuk
>
> Senior Software Engineer at Synesis
> skype: arseny.kovalchuk
> mobile: [phone]
> ​LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>​
>

--
Best regards,
Ilya