Ignite monitoring

[irudyak]

Hi guys,

While experimenting with large Ignite clusters I found that lack of
monitoring is rather critical problem. I know that Ignite provides number
of JMX MBeans to monitor custom metrics in addition to host system metrics
(CPU, IO, RAM, ....). The problem is, there are no out of the box solution
to monitor all this.

Thus you have to manually setup some kind of monitoring tool like Graphite,
Grafana, Ganglia and etc. Which involves setting up monitoring agents on
all the nodes, uploading JMX agent on all the nodes, selecting appropriate
metrics from the plenty of JMX MBeans and preparing config files, tuning
Ignite shell scripts to include "java agent" in java launch command. Lots
of work and pain, each time you want to create new Ignite cluster.

Probably it makes sense to have all these out of the box, by slightly
modifying existing and providing additional shell scripts, to bootstrap all
monitoring infrastructure?

Igor Rudyak

[yzhdanov]

Igor, can you please share the changes to scripts you did to support
monitoring? Can it be done by defining and exporting JAVA_OPTS env variable
and then launching ignite.sh?

Thanks!

--Yakov

2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:

> Hi guys,
>
> While experimenting with large Ignite clusters I found that lack of
> monitoring is rather critical problem. I know that Ignite provides number
> of JMX MBeans to monitor custom metrics in addition to host system metrics
> (CPU, IO, RAM, ....). The problem is, there are no out of the box solution
> to monitor all this.
>
> Thus you have to manually setup some kind of monitoring tool like Graphite,
> Grafana, Ganglia and etc. Which involves setting up monitoring agents on
> all the nodes, uploading JMX agent on all the nodes, selecting appropriate
> metrics from the plenty of JMX MBeans and preparing config files, tuning
> Ignite shell scripts to include "java agent" in java launch command. Lots
> of work and pain, each time you want to create new Ignite cluster.
>
> Probably it makes sense to have all these out of the box, by slightly
> modifying existing and providing additional shell scripts, to bootstrap all
> monitoring infrastructure?
>
> Igor Rudyak
>

[irudyak]

Yakov, as for now I just have well structured scripts to setup Ganglia
agent on Ignite hosts to monitor system metrics like CPU, RAM, IO and etc
(this scripts already included in Ignite 1.6).

Also experimented with displaying JVM metrics by providing java agent and
specifying MBeans to collect metrics from. But it's rather draft version.
The second problem is, there are plenty of MBeans in Ignite - I just don't
know which to select from.

Anyway, the original idea was to check with the community if it makes sense
to have such monitoring functionality out of the box.

Igor Rudyak



On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <[hidden email]> wrote:

> Igor, can you please share the changes to scripts you did to support
> monitoring? Can it be done by defining and exporting JAVA_OPTS env variable
> and then launching ignite.sh?
>
> Thanks!
>
> --Yakov
>
> 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:
>
> > Hi guys,
> >
> > While experimenting with large Ignite clusters I found that lack of
> > monitoring is rather critical problem. I know that Ignite provides number
> > of JMX MBeans to monitor custom metrics in addition to host system
> metrics
> > (CPU, IO, RAM, ....). The problem is, there are no out of the box
> solution
> > to monitor all this.
> >
> > Thus you have to manually setup some kind of monitoring tool like
> Graphite,
> > Grafana, Ganglia and etc. Which involves setting up monitoring agents on
> > all the nodes, uploading JMX agent on all the nodes, selecting
> appropriate
> > metrics from the plenty of JMX MBeans and preparing config files, tuning
> > Ignite shell scripts to include "java agent" in java launch command. Lots
> > of work and pain, each time you want to create new Ignite cluster.
> >
> > Probably it makes sense to have all these out of the box, by slightly
> > modifying existing and providing additional shell scripts, to bootstrap
> all
> > monitoring infrastructure?
> >
> > Igor Rudyak
> >
>

[Vladimir Ozerov]

Igor,

I think that built-in monitoring facility will add great value to the
product. We have to deal with user performance issues pretty often, and it
is always a kind of pain to get to the bottom of the problem. We have to
ask users for configuration, logs, system config, etc, etc.. Instead, it
would be great if we had a single big "switch". If user has performance
issue, he turns it on, then perform problematic operations, and then dumps
all collected data. We can collect dozens of things:
1) OS/JVM information
2) Ignite configs, logs, etc..
3) Performance data (CPU, RAM, IO)
4) Metrics
5) JMX data (both Ignite and JVM)
6) Some internal tracing (SQL query plans, how long it takes messages to
pass between nodes, etc.)

I think the most important part here is good infrastructure (interfaces)
and API. So that we can start with something very simple, like collecting
configs from all nodes, or starting/stopping shell commands, and then
gradually add more and more tracing facilities.

Thoughts?

Vladimir.


On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <[hidden email]> wrote:

> Yakov, as for now I just have well structured scripts to setup Ganglia
> agent on Ignite hosts to monitor system metrics like CPU, RAM, IO and etc
> (this scripts already included in Ignite 1.6).
>
> Also experimented with displaying JVM metrics by providing java agent and
> specifying MBeans to collect metrics from. But it's rather draft version.
> The second problem is, there are plenty of MBeans in Ignite - I just don't
> know which to select from.
>
> Anyway, the original idea was to check with the community if it makes sense
> to have such monitoring functionality out of the box.
>
> Igor Rudyak
>
>
>
> On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <[hidden email]>
> wrote:
>
> > Igor, can you please share the changes to scripts you did to support
> > monitoring? Can it be done by defining and exporting JAVA_OPTS env
> variable
> > and then launching ignite.sh?
> >
> > Thanks!
> >
> > --Yakov
> >
> > 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:
> >
> > > Hi guys,
> > >
> > > While experimenting with large Ignite clusters I found that lack of
> > > monitoring is rather critical problem. I know that Ignite provides
> number
> > > of JMX MBeans to monitor custom metrics in addition to host system
> > metrics
> > > (CPU, IO, RAM, ....). The problem is, there are no out of the box
> > solution
> > > to monitor all this.
> > >
> > > Thus you have to manually setup some kind of monitoring tool like
> > Graphite,
> > > Grafana, Ganglia and etc. Which involves setting up monitoring agents
> on
> > > all the nodes, uploading JMX agent on all the nodes, selecting
> > appropriate
> > > metrics from the plenty of JMX MBeans and preparing config files,
> tuning
> > > Ignite shell scripts to include "java agent" in java launch command.
> Lots
> > > of work and pain, each time you want to create new Ignite cluster.
> > >
> > > Probably it makes sense to have all these out of the box, by slightly
> > > modifying existing and providing additional shell scripts, to bootstrap
> > all
> > > monitoring infrastructure?
> > >
> > > Igor Rudyak
> > >
> >
>

[dsetrakyan]

I think we can add this functionality to Ignite web console, no?

On Mon, Jul 18, 2016 at 11:08 AM, Vladimir Ozerov <[hidden email]>
wrote:

> Igor,
>
> I think that built-in monitoring facility will add great value to the
> product. We have to deal with user performance issues pretty often, and it
> is always a kind of pain to get to the bottom of the problem. We have to
> ask users for configuration, logs, system config, etc, etc.. Instead, it
> would be great if we had a single big "switch". If user has performance
> issue, he turns it on, then perform problematic operations, and then dumps
> all collected data. We can collect dozens of things:
> 1) OS/JVM information
> 2) Ignite configs, logs, etc..
> 3) Performance data (CPU, RAM, IO)
> 4) Metrics
> 5) JMX data (both Ignite and JVM)
> 6) Some internal tracing (SQL query plans, how long it takes messages to
> pass between nodes, etc.)
>
> I think the most important part here is good infrastructure (interfaces)
> and API. So that we can start with something very simple, like collecting
> configs from all nodes, or starting/stopping shell commands, and then
> gradually add more and more tracing facilities.
>
> Thoughts?
>
> Vladimir.
>
>
> On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <[hidden email]> wrote:
>
> > Yakov, as for now I just have well structured scripts to setup Ganglia
> > agent on Ignite hosts to monitor system metrics like CPU, RAM, IO and etc
> > (this scripts already included in Ignite 1.6).
> >
> > Also experimented with displaying JVM metrics by providing java agent and
> > specifying MBeans to collect metrics from. But it's rather draft version.
> > The second problem is, there are plenty of MBeans in Ignite - I just
> don't
> > know which to select from.
> >
> > Anyway, the original idea was to check with the community if it makes
> sense
> > to have such monitoring functionality out of the box.
> >
> > Igor Rudyak
> >
> >
> >
> > On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <[hidden email]>
> > wrote:
> >
> > > Igor, can you please share the changes to scripts you did to support
> > > monitoring? Can it be done by defining and exporting JAVA_OPTS env
> > variable
> > > and then launching ignite.sh?
> > >
> > > Thanks!
> > >
> > > --Yakov
> > >
> > > 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:
> > >
> > > > Hi guys,
> > > >
> > > > While experimenting with large Ignite clusters I found that lack of
> > > > monitoring is rather critical problem. I know that Ignite provides
> > number
> > > > of JMX MBeans to monitor custom metrics in addition to host system
> > > metrics
> > > > (CPU, IO, RAM, ....). The problem is, there are no out of the box
> > > solution
> > > > to monitor all this.
> > > >
> > > > Thus you have to manually setup some kind of monitoring tool like
> > > Graphite,
> > > > Grafana, Ganglia and etc. Which involves setting up monitoring agents
> > on
> > > > all the nodes, uploading JMX agent on all the nodes, selecting
> > > appropriate
> > > > metrics from the plenty of JMX MBeans and preparing config files,
> > tuning
> > > > Ignite shell scripts to include "java agent" in java launch command.
> > Lots
> > > > of work and pain, each time you want to create new Ignite cluster.
> > > >
> > > > Probably it makes sense to have all these out of the box, by slightly
> > > > modifying existing and providing additional shell scripts, to
> bootstrap
> > > all
> > > > monitoring infrastructure?
> > > >
> > > > Igor Rudyak
> > > >
> > >
> >
>

[Alexey Kuznetsov-2]

I think we should have some general API and we could call it from web
console and/or from other places.

18 Июл 2016 г. 20:18 пользователь "Dmitriy Setrakyan" <[hidden email]>
написал:

> I think we can add this functionality to Ignite web console, no?
>
> On Mon, Jul 18, 2016 at 11:08 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Igor,
> >
> > I think that built-in monitoring facility will add great value to the
> > product. We have to deal with user performance issues pretty often, and
> it
> > is always a kind of pain to get to the bottom of the problem. We have to
> > ask users for configuration, logs, system config, etc, etc.. Instead, it
> > would be great if we had a single big "switch". If user has performance
> > issue, he turns it on, then perform problematic operations, and then
> dumps
> > all collected data. We can collect dozens of things:
> > 1) OS/JVM information
> > 2) Ignite configs, logs, etc..
> > 3) Performance data (CPU, RAM, IO)
> > 4) Metrics
> > 5) JMX data (both Ignite and JVM)
> > 6) Some internal tracing (SQL query plans, how long it takes messages to
> > pass between nodes, etc.)
> >
> > I think the most important part here is good infrastructure (interfaces)
> > and API. So that we can start with something very simple, like collecting
> > configs from all nodes, or starting/stopping shell commands, and then
> > gradually add more and more tracing facilities.
> >
> > Thoughts?
> >
> > Vladimir.
> >
> >
> > On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <[hidden email]> wrote:
> >
> > > Yakov, as for now I just have well structured scripts to setup Ganglia
> > > agent on Ignite hosts to monitor system metrics like CPU, RAM, IO and
> etc
> > > (this scripts already included in Ignite 1.6).
> > >
> > > Also experimented with displaying JVM metrics by providing java agent
> and
> > > specifying MBeans to collect metrics from. But it's rather draft
> version.
> > > The second problem is, there are plenty of MBeans in Ignite - I just
> > don't
> > > know which to select from.
> > >
> > > Anyway, the original idea was to check with the community if it makes
> > sense
> > > to have such monitoring functionality out of the box.
> > >
> > > Igor Rudyak
> > >
> > >
> > >
> > > On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <[hidden email]>
> > > wrote:
> > >
> > > > Igor, can you please share the changes to scripts you did to support
> > > > monitoring? Can it be done by defining and exporting JAVA_OPTS env
> > > variable
> > > > and then launching ignite.sh?
> > > >
> > > > Thanks!
> > > >
> > > > --Yakov
> > > >
> > > > 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > While experimenting with large Ignite clusters I found that lack of
> > > > > monitoring is rather critical problem. I know that Ignite provides
> > > number
> > > > > of JMX MBeans to monitor custom metrics in addition to host system
> > > > metrics
> > > > > (CPU, IO, RAM, ....). The problem is, there are no out of the box
> > > > solution
> > > > > to monitor all this.
> > > > >
> > > > > Thus you have to manually setup some kind of monitoring tool like
> > > > Graphite,
> > > > > Grafana, Ganglia and etc. Which involves setting up monitoring
> agents
> > > on
> > > > > all the nodes, uploading JMX agent on all the nodes, selecting
> > > > appropriate
> > > > > metrics from the plenty of JMX MBeans and preparing config files,
> > > tuning
> > > > > Ignite shell scripts to include "java agent" in java launch
> command.
> > > Lots
> > > > > of work and pain, each time you want to create new Ignite cluster.
> > > > >
> > > > > Probably it makes sense to have all these out of the box, by
> slightly
> > > > > modifying existing and providing additional shell scripts, to
> > bootstrap
> > > > all
> > > > > monitoring infrastructure?
> > > > >
> > > > > Igor Rudyak
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

On Mon, Jul 18, 2016 at 8:45 PM, Alexey Kuznetsov <[hidden email]>
wrote:

> I think we should have some general API and we could call it from web
> console and/or from other places.
>

Agree. The server side support should be sufficient to enable different
monitoring connections, including command-line visor, web console, or JMX
beans.

> 18 Июл 2016 г. 20:18 пользователь "Dmitriy Setrakyan" <
> [hidden email]>
> написал:
>
> > I think we can add this functionality to Ignite web console, no?
> >
> > On Mon, Jul 18, 2016 at 11:08 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Igor,
> > >
> > > I think that built-in monitoring facility will add great value to the
> > > product. We have to deal with user performance issues pretty often, and
> > it
> > > is always a kind of pain to get to the bottom of the problem. We have
> to
> > > ask users for configuration, logs, system config, etc, etc.. Instead,
> it
> > > would be great if we had a single big "switch". If user has performance
> > > issue, he turns it on, then perform problematic operations, and then
> > dumps
> > > all collected data. We can collect dozens of things:
> > > 1) OS/JVM information
> > > 2) Ignite configs, logs, etc..
> > > 3) Performance data (CPU, RAM, IO)
> > > 4) Metrics
> > > 5) JMX data (both Ignite and JVM)
> > > 6) Some internal tracing (SQL query plans, how long it takes messages
> to
> > > pass between nodes, etc.)
> > >
> > > I think the most important part here is good infrastructure
> (interfaces)
> > > and API. So that we can start with something very simple, like
> collecting
> > > configs from all nodes, or starting/stopping shell commands, and then
> > > gradually add more and more tracing facilities.
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> > >
> > > On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <[hidden email]>
> wrote:
> > >
> > > > Yakov, as for now I just have well structured scripts to setup
> Ganglia
> > > > agent on Ignite hosts to monitor system metrics like CPU, RAM, IO and
> > etc
> > > > (this scripts already included in Ignite 1.6).
> > > >
> > > > Also experimented with displaying JVM metrics by providing java agent
> > and
> > > > specifying MBeans to collect metrics from. But it's rather draft
> > version.
> > > > The second problem is, there are plenty of MBeans in Ignite - I just
> > > don't
> > > > know which to select from.
> > > >
> > > > Anyway, the original idea was to check with the community if it makes
> > > sense
> > > > to have such monitoring functionality out of the box.
> > > >
> > > > Igor Rudyak
> > > >
> > > >
> > > >
> > > > On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <[hidden email]>
> > > > wrote:
> > > >
> > > > > Igor, can you please share the changes to scripts you did to
> support
> > > > > monitoring? Can it be done by defining and exporting JAVA_OPTS env
> > > > variable
> > > > > and then launching ignite.sh?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --Yakov
> > > > >
> > > > > 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > While experimenting with large Ignite clusters I found that lack
> of
> > > > > > monitoring is rather critical problem. I know that Ignite
> provides
> > > > number
> > > > > > of JMX MBeans to monitor custom metrics in addition to host
> system
> > > > > metrics
> > > > > > (CPU, IO, RAM, ....). The problem is, there are no out of the box
> > > > > solution
> > > > > > to monitor all this.
> > > > > >
> > > > > > Thus you have to manually setup some kind of monitoring tool like
> > > > > Graphite,
> > > > > > Grafana, Ganglia and etc. Which involves setting up monitoring
> > agents
> > > > on
> > > > > > all the nodes, uploading JMX agent on all the nodes, selecting
> > > > > appropriate
> > > > > > metrics from the plenty of JMX MBeans and preparing config files,
> > > > tuning
> > > > > > Ignite shell scripts to include "java agent" in java launch
> > command.
> > > > Lots
> > > > > > of work and pain, each time you want to create new Ignite
> cluster.
> > > > > >
> > > > > > Probably it makes sense to have all these out of the box, by
> > slightly
> > > > > > modifying existing and providing additional shell scripts, to
> > > bootstrap
> > > > > all
> > > > > > monitoring infrastructure?
> > > > > >
> > > > > > Igor Rudyak
> > > > > >
> > > > >
> > > >
> > >
> >
>

[irudyak]

Are there any documentation regarding how to use Ignite web console? How to
add new metrics to monitor?

On Mon, Jul 18, 2016 at 10:47 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Mon, Jul 18, 2016 at 8:45 PM, Alexey Kuznetsov <[hidden email]
> >
> wrote:
>
> > I think we should have some general API and we could call it from web
> > console and/or from other places.
> >
>
> Agree. The server side support should be sufficient to enable different
> monitoring connections, including command-line visor, web console, or JMX
> beans.
>
>
> > 18 Июл 2016 г. 20:18 пользователь "Dmitriy Setrakyan" <
> > [hidden email]>
> > написал:
> >
> > > I think we can add this functionality to Ignite web console, no?
> > >
> > > On Mon, Jul 18, 2016 at 11:08 AM, Vladimir Ozerov <
> [hidden email]>
> > > wrote:
> > >
> > > > Igor,
> > > >
> > > > I think that built-in monitoring facility will add great value to the
> > > > product. We have to deal with user performance issues pretty often,
> and
> > > it
> > > > is always a kind of pain to get to the bottom of the problem. We have
> > to
> > > > ask users for configuration, logs, system config, etc, etc.. Instead,
> > it
> > > > would be great if we had a single big "switch". If user has
> performance
> > > > issue, he turns it on, then perform problematic operations, and then
> > > dumps
> > > > all collected data. We can collect dozens of things:
> > > > 1) OS/JVM information
> > > > 2) Ignite configs, logs, etc..
> > > > 3) Performance data (CPU, RAM, IO)
> > > > 4) Metrics
> > > > 5) JMX data (both Ignite and JVM)
> > > > 6) Some internal tracing (SQL query plans, how long it takes messages
> > to
> > > > pass between nodes, etc.)
> > > >
> > > > I think the most important part here is good infrastructure
> > (interfaces)
> > > > and API. So that we can start with something very simple, like
> > collecting
> > > > configs from all nodes, or starting/stopping shell commands, and then
> > > > gradually add more and more tracing facilities.
> > > >
> > > > Thoughts?
> > > >
> > > > Vladimir.
> > > >
> > > >
> > > > On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <[hidden email]>
> > wrote:
> > > >
> > > > > Yakov, as for now I just have well structured scripts to setup
> > Ganglia
> > > > > agent on Ignite hosts to monitor system metrics like CPU, RAM, IO
> and
> > > etc
> > > > > (this scripts already included in Ignite 1.6).
> > > > >
> > > > > Also experimented with displaying JVM metrics by providing java
> agent
> > > and
> > > > > specifying MBeans to collect metrics from. But it's rather draft
> > > version.
> > > > > The second problem is, there are plenty of MBeans in Ignite - I
> just
> > > > don't
> > > > > know which to select from.
> > > > >
> > > > > Anyway, the original idea was to check with the community if it
> makes
> > > > sense
> > > > > to have such monitoring functionality out of the box.
> > > > >
> > > > > Igor Rudyak
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Igor, can you please share the changes to scripts you did to
> > support
> > > > > > monitoring? Can it be done by defining and exporting JAVA_OPTS
> env
> > > > > variable
> > > > > > and then launching ignite.sh?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > --Yakov
> > > > > >
> > > > > > 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:
> > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > While experimenting with large Ignite clusters I found that
> lack
> > of
> > > > > > > monitoring is rather critical problem. I know that Ignite
> > provides
> > > > > number
> > > > > > > of JMX MBeans to monitor custom metrics in addition to host
> > system
> > > > > > metrics
> > > > > > > (CPU, IO, RAM, ....). The problem is, there are no out of the
> box
> > > > > > solution
> > > > > > > to monitor all this.
> > > > > > >
> > > > > > > Thus you have to manually setup some kind of monitoring tool
> like
> > > > > > Graphite,
> > > > > > > Grafana, Ganglia and etc. Which involves setting up monitoring
> > > agents
> > > > > on
> > > > > > > all the nodes, uploading JMX agent on all the nodes, selecting
> > > > > > appropriate
> > > > > > > metrics from the plenty of JMX MBeans and preparing config
> files,
> > > > > tuning
> > > > > > > Ignite shell scripts to include "java agent" in java launch
> > > command.
> > > > > Lots
> > > > > > > of work and pain, each time you want to create new Ignite
> > cluster.
> > > > > > >
> > > > > > > Probably it makes sense to have all these out of the box, by
> > > slightly
> > > > > > > modifying existing and providing additional shell scripts, to
> > > > bootstrap
> > > > > > all
> > > > > > > monitoring infrastructure?
> > > > > > >
> > > > > > > Igor Rudyak
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Andrey Novikov-3]

Igor,

Ignite does not have monitoring in Web Console, only Configuration and SQL.
But command-line Visor in Ignite have some monitoring features and can be
used  to add new.

On Wed, Jul 20, 2016 at 1:19 AM, Igor Rudyak <[hidden email]> wrote:

> Are there any documentation regarding how to use Ignite web console? How to
> add new metrics to monitor?
>
> On Mon, Jul 18, 2016 at 10:47 AM, Dmitriy Setrakyan <[hidden email]
> >
> wrote:
>
> > On Mon, Jul 18, 2016 at 8:45 PM, Alexey Kuznetsov <
> [hidden email]
> > >
> > wrote:
> >
> > > I think we should have some general API and we could call it from web
> > > console and/or from other places.
> > >
> >
> > Agree. The server side support should be sufficient to enable different
> > monitoring connections, including command-line visor, web console, or JMX
> > beans.
> >
> >
> > > 18 Июл 2016 г. 20:18 пользователь "Dmitriy Setrakyan" <
> > > [hidden email]>
> > > написал:
> > >
> > > > I think we can add this functionality to Ignite web console, no?
> > > >
> > > > On Mon, Jul 18, 2016 at 11:08 AM, Vladimir Ozerov <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Igor,
> > > > >
> > > > > I think that built-in monitoring facility will add great value to
> the
> > > > > product. We have to deal with user performance issues pretty often,
> > and
> > > > it
> > > > > is always a kind of pain to get to the bottom of the problem. We
> have
> > > to
> > > > > ask users for configuration, logs, system config, etc, etc..
> Instead,
> > > it
> > > > > would be great if we had a single big "switch". If user has
> > performance
> > > > > issue, he turns it on, then perform problematic operations, and
> then
> > > > dumps
> > > > > all collected data. We can collect dozens of things:
> > > > > 1) OS/JVM information
> > > > > 2) Ignite configs, logs, etc..
> > > > > 3) Performance data (CPU, RAM, IO)
> > > > > 4) Metrics
> > > > > 5) JMX data (both Ignite and JVM)
> > > > > 6) Some internal tracing (SQL query plans, how long it takes
> messages
> > > to
> > > > > pass between nodes, etc.)
> > > > >
> > > > > I think the most important part here is good infrastructure
> > > (interfaces)
> > > > > and API. So that we can start with something very simple, like
> > > collecting
> > > > > configs from all nodes, or starting/stopping shell commands, and
> then
> > > > > gradually add more and more tracing facilities.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vladimir.
> > > > >
> > > > >
> > > > > On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Yakov, as for now I just have well structured scripts to setup
> > > Ganglia
> > > > > > agent on Ignite hosts to monitor system metrics like CPU, RAM, IO
> > and
> > > > etc
> > > > > > (this scripts already included in Ignite 1.6).
> > > > > >
> > > > > > Also experimented with displaying JVM metrics by providing java
> > agent
> > > > and
> > > > > > specifying MBeans to collect metrics from. But it's rather draft
> > > > version.
> > > > > > The second problem is, there are plenty of MBeans in Ignite - I
> > just
> > > > > don't
> > > > > > know which to select from.
> > > > > >
> > > > > > Anyway, the original idea was to check with the community if it
> > makes
> > > > > sense
> > > > > > to have such monitoring functionality out of the box.
> > > > > >
> > > > > > Igor Rudyak
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <
> > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Igor, can you please share the changes to scripts you did to
> > > support
> > > > > > > monitoring? Can it be done by defining and exporting JAVA_OPTS
> > env
> > > > > > variable
> > > > > > > and then launching ignite.sh?
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > --Yakov
> > > > > > >
> > > > > > > 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:
> > > > > > >
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > > While experimenting with large Ignite clusters I found that
> > lack
> > > of
> > > > > > > > monitoring is rather critical problem. I know that Ignite
> > > provides
> > > > > > number
> > > > > > > > of JMX MBeans to monitor custom metrics in addition to host
> > > system
> > > > > > > metrics
> > > > > > > > (CPU, IO, RAM, ....). The problem is, there are no out of the
> > box
> > > > > > > solution
> > > > > > > > to monitor all this.
> > > > > > > >
> > > > > > > > Thus you have to manually setup some kind of monitoring tool
> > like
> > > > > > > Graphite,
> > > > > > > > Grafana, Ganglia and etc. Which involves setting up
> monitoring
> > > > agents
> > > > > > on
> > > > > > > > all the nodes, uploading JMX agent on all the nodes,
> selecting
> > > > > > > appropriate
> > > > > > > > metrics from the plenty of JMX MBeans and preparing config
> > files,
> > > > > > tuning
> > > > > > > > Ignite shell scripts to include "java agent" in java launch
> > > > command.
> > > > > > Lots
> > > > > > > > of work and pain, each time you want to create new Ignite
> > > cluster.
> > > > > > > >
> > > > > > > > Probably it makes sense to have all these out of the box, by
> > > > slightly
> > > > > > > > modifying existing and providing additional shell scripts, to
> > > > > bootstrap
> > > > > > > all
> > > > > > > > monitoring infrastructure?
> > > > > > > >
> > > > > > > > Igor Rudyak
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[irudyak]

Andrey,

Is Web Console you are talking about the same thing as GridGain Web Console
(http://ignite.apache.org/addons.html#web-console)? If yes it has
monitoring tab which allows to monitor some JVM and cache metrics (
https://console.gridgain.com/monitoring).

As for me, Visor command line tool doesn't look like good candidate for
monitoring. Monitoring tool should provide rather friendly UI with graphs,
tables and etc. to simplify monitoring of the system. While command line
tool like Visor is more about system administration/management.

Igor Rudyak


On Wed, Jul 20, 2016 at 2:46 AM, Andrey Novikov <[hidden email]> wrote:

> Igor,
>
> Ignite does not have monitoring in Web Console, only Configuration and SQL.
> But command-line Visor in Ignite have some monitoring features and can be
> used  to add new.
>
> On Wed, Jul 20, 2016 at 1:19 AM, Igor Rudyak <[hidden email]> wrote:
>
> > Are there any documentation regarding how to use Ignite web console? How
> to
> > add new metrics to monitor?
> >
> > On Mon, Jul 18, 2016 at 10:47 AM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > wrote:
> >
> > > On Mon, Jul 18, 2016 at 8:45 PM, Alexey Kuznetsov <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > I think we should have some general API and we could call it from web
> > > > console and/or from other places.
> > > >
> > >
> > > Agree. The server side support should be sufficient to enable different
> > > monitoring connections, including command-line visor, web console, or
> JMX
> > > beans.
> > >
> > >
> > > > 18 Июл 2016 г. 20:18 пользователь "Dmitriy Setrakyan" <
> > > > [hidden email]>
> > > > написал:
> > > >
> > > > > I think we can add this functionality to Ignite web console, no?
> > > > >
> > > > > On Mon, Jul 18, 2016 at 11:08 AM, Vladimir Ozerov <
> > > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Igor,
> > > > > >
> > > > > > I think that built-in monitoring facility will add great value to
> > the
> > > > > > product. We have to deal with user performance issues pretty
> often,
> > > and
> > > > > it
> > > > > > is always a kind of pain to get to the bottom of the problem. We
> > have
> > > > to
> > > > > > ask users for configuration, logs, system config, etc, etc..
> > Instead,
> > > > it
> > > > > > would be great if we had a single big "switch". If user has
> > > performance
> > > > > > issue, he turns it on, then perform problematic operations, and
> > then
> > > > > dumps
> > > > > > all collected data. We can collect dozens of things:
> > > > > > 1) OS/JVM information
> > > > > > 2) Ignite configs, logs, etc..
> > > > > > 3) Performance data (CPU, RAM, IO)
> > > > > > 4) Metrics
> > > > > > 5) JMX data (both Ignite and JVM)
> > > > > > 6) Some internal tracing (SQL query plans, how long it takes
> > messages
> > > > to
> > > > > > pass between nodes, etc.)
> > > > > >
> > > > > > I think the most important part here is good infrastructure
> > > > (interfaces)
> > > > > > and API. So that we can start with something very simple, like
> > > > collecting
> > > > > > configs from all nodes, or starting/stopping shell commands, and
> > then
> > > > > > gradually add more and more tracing facilities.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Vladimir.
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <[hidden email]
> >
> > > > wrote:
> > > > > >
> > > > > > > Yakov, as for now I just have well structured scripts to setup
> > > > Ganglia
> > > > > > > agent on Ignite hosts to monitor system metrics like CPU, RAM,
> IO
> > > and
> > > > > etc
> > > > > > > (this scripts already included in Ignite 1.6).
> > > > > > >
> > > > > > > Also experimented with displaying JVM metrics by providing java
> > > agent
> > > > > and
> > > > > > > specifying MBeans to collect metrics from. But it's rather
> draft
> > > > > version.
> > > > > > > The second problem is, there are plenty of MBeans in Ignite - I
> > > just
> > > > > > don't
> > > > > > > know which to select from.
> > > > > > >
> > > > > > > Anyway, the original idea was to check with the community if it
> > > makes
> > > > > > sense
> > > > > > > to have such monitoring functionality out of the box.
> > > > > > >
> > > > > > > Igor Rudyak
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Igor, can you please share the changes to scripts you did to
> > > > support
> > > > > > > > monitoring? Can it be done by defining and exporting
> JAVA_OPTS
> > > env
> > > > > > > variable
> > > > > > > > and then launching ignite.sh?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > --Yakov
> > > > > > > >
> > > > > > > > 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]>:
> > > > > > > >
> > > > > > > > > Hi guys,
> > > > > > > > >
> > > > > > > > > While experimenting with large Ignite clusters I found that
> > > lack
> > > > of
> > > > > > > > > monitoring is rather critical problem. I know that Ignite
> > > > provides
> > > > > > > number
> > > > > > > > > of JMX MBeans to monitor custom metrics in addition to host
> > > > system
> > > > > > > > metrics
> > > > > > > > > (CPU, IO, RAM, ....). The problem is, there are no out of
> the
> > > box
> > > > > > > > solution
> > > > > > > > > to monitor all this.
> > > > > > > > >
> > > > > > > > > Thus you have to manually setup some kind of monitoring
> tool
> > > like
> > > > > > > > Graphite,
> > > > > > > > > Grafana, Ganglia and etc. Which involves setting up
> > monitoring
> > > > > agents
> > > > > > > on
> > > > > > > > > all the nodes, uploading JMX agent on all the nodes,
> > selecting
> > > > > > > > appropriate
> > > > > > > > > metrics from the plenty of JMX MBeans and preparing config
> > > files,
> > > > > > > tuning
> > > > > > > > > Ignite shell scripts to include "java agent" in java launch
> > > > > command.
> > > > > > > Lots
> > > > > > > > > of work and pain, each time you want to create new Ignite
> > > > cluster.
> > > > > > > > >
> > > > > > > > > Probably it makes sense to have all these out of the box,
> by
> > > > > slightly
> > > > > > > > > modifying existing and providing additional shell scripts,
> to
> > > > > > bootstrap
> > > > > > > > all
> > > > > > > > > monitoring infrastructure?
> > > > > > > > >
> > > > > > > > > Igor Rudyak
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[dsetrakyan]

On Wed, Jul 20, 2016 at 7:35 PM, Igor Rudyak <[hidden email]> wrote:

> Andrey,
>
> Is Web Console you are talking about the same thing as GridGain Web Console
> (http://ignite.apache.org/addons.html#web-console)? If yes it has
> monitoring tab which allows to monitor some JVM and cache metrics (
> https://console.gridgain.com/monitoring).
>

Not exactly. GridGain web console is based on Ignite web console, but
GridGain added more features to it, management tab is one of those features.

You should feel free to add a monitoring tab as part of Ignite web console
or work with GridGain directly to add more features to the management tab.

> Igor Rudyak
>
>
> On Wed, Jul 20, 2016 at 2:46 AM, Andrey Novikov <[hidden email]>
> wrote:
>
> > Igor,
> >
> > Ignite does not have monitoring in Web Console, only Configuration and
> SQL.
> > But command-line Visor in Ignite have some monitoring features and can be
> > used  to add new.
> >
> > On Wed, Jul 20, 2016 at 1:19 AM, Igor Rudyak <[hidden email]> wrote:
> >
> > > Are there any documentation regarding how to use Ignite web console?
> How
> > to
> > > add new metrics to monitor?
> > >
> > > On Mon, Jul 18, 2016 at 10:47 AM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > On Mon, Jul 18, 2016 at 8:45 PM, Alexey Kuznetsov <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > I think we should have some general API and we could call it from
> web
> > > > > console and/or from other places.
> > > > >
> > > >
> > > > Agree. The server side support should be sufficient to enable
> different
> > > > monitoring connections, including command-line visor, web console, or
> > JMX
> > > > beans.
> > > >
> > > >
> > > > > 18 Июл 2016 г. 20:18 пользователь "Dmitriy Setrakyan" <
> > > > > [hidden email]>
> > > > > написал:
> > > > >
> > > > > > I think we can add this functionality to Ignite web console, no?
> > > > > >
> > > > > > On Mon, Jul 18, 2016 at 11:08 AM, Vladimir Ozerov <
> > > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Igor,
> > > > > > >
> > > > > > > I think that built-in monitoring facility will add great value
> to
> > > the
> > > > > > > product. We have to deal with user performance issues pretty
> > often,
> > > > and
> > > > > > it
> > > > > > > is always a kind of pain to get to the bottom of the problem.
> We
> > > have
> > > > > to
> > > > > > > ask users for configuration, logs, system config, etc, etc..
> > > Instead,
> > > > > it
> > > > > > > would be great if we had a single big "switch". If user has
> > > > performance
> > > > > > > issue, he turns it on, then perform problematic operations, and
> > > then
> > > > > > dumps
> > > > > > > all collected data. We can collect dozens of things:
> > > > > > > 1) OS/JVM information
> > > > > > > 2) Ignite configs, logs, etc..
> > > > > > > 3) Performance data (CPU, RAM, IO)
> > > > > > > 4) Metrics
> > > > > > > 5) JMX data (both Ignite and JVM)
> > > > > > > 6) Some internal tracing (SQL query plans, how long it takes
> > > messages
> > > > > to
> > > > > > > pass between nodes, etc.)
> > > > > > >
> > > > > > > I think the most important part here is good infrastructure
> > > > > (interfaces)
> > > > > > > and API. So that we can start with something very simple, like
> > > > > collecting
> > > > > > > configs from all nodes, or starting/stopping shell commands,
> and
> > > then
> > > > > > > gradually add more and more tracing facilities.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 14, 2016 at 11:36 PM, Igor Rudyak <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Yakov, as for now I just have well structured scripts to
> setup
> > > > > Ganglia
> > > > > > > > agent on Ignite hosts to monitor system metrics like CPU,
> RAM,
> > IO
> > > > and
> > > > > > etc
> > > > > > > > (this scripts already included in Ignite 1.6).
> > > > > > > >
> > > > > > > > Also experimented with displaying JVM metrics by providing
> java
> > > > agent
> > > > > > and
> > > > > > > > specifying MBeans to collect metrics from. But it's rather
> > draft
> > > > > > version.
> > > > > > > > The second problem is, there are plenty of MBeans in Ignite
> - I
> > > > just
> > > > > > > don't
> > > > > > > > know which to select from.
> > > > > > > >
> > > > > > > > Anyway, the original idea was to check with the community if
> it
> > > > makes
> > > > > > > sense
> > > > > > > > to have such monitoring functionality out of the box.
> > > > > > > >
> > > > > > > > Igor Rudyak
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 14, 2016 at 1:05 AM, Yakov Zhdanov <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Igor, can you please share the changes to scripts you did
> to
> > > > > support
> > > > > > > > > monitoring? Can it be done by defining and exporting
> > JAVA_OPTS
> > > > env
> > > > > > > > variable
> > > > > > > > > and then launching ignite.sh?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > --Yakov
> > > > > > > > >
> > > > > > > > > 2016-07-13 22:45 GMT+03:00 Igor Rudyak <[hidden email]
> >:
> > > > > > > > >
> > > > > > > > > > Hi guys,
> > > > > > > > > >
> > > > > > > > > > While experimenting with large Ignite clusters I found
> that
> > > > lack
> > > > > of
> > > > > > > > > > monitoring is rather critical problem. I know that Ignite
> > > > > provides
> > > > > > > > number
> > > > > > > > > > of JMX MBeans to monitor custom metrics in addition to
> host
> > > > > system
> > > > > > > > > metrics
> > > > > > > > > > (CPU, IO, RAM, ....). The problem is, there are no out of
> > the
> > > > box
> > > > > > > > > solution
> > > > > > > > > > to monitor all this.
> > > > > > > > > >
> > > > > > > > > > Thus you have to manually setup some kind of monitoring
> > tool
> > > > like
> > > > > > > > > Graphite,
> > > > > > > > > > Grafana, Ganglia and etc. Which involves setting up
> > > monitoring
> > > > > > agents
> > > > > > > > on
> > > > > > > > > > all the nodes, uploading JMX agent on all the nodes,
> > > selecting
> > > > > > > > > appropriate
> > > > > > > > > > metrics from the plenty of JMX MBeans and preparing
> config
> > > > files,
> > > > > > > > tuning
> > > > > > > > > > Ignite shell scripts to include "java agent" in java
> launch
> > > > > > command.
> > > > > > > > Lots
> > > > > > > > > > of work and pain, each time you want to create new Ignite
> > > > > cluster.
> > > > > > > > > >
> > > > > > > > > > Probably it makes sense to have all these out of the box,
> > by
> > > > > > slightly
> > > > > > > > > > modifying existing and providing additional shell
> scripts,
> > to
> > > > > > > bootstrap
> > > > > > > > > all
> > > > > > > > > > monitoring infrastructure?
> > > > > > > > > >
> > > > > > > > > > Igor Rudyak
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>