[IEP-35] Monitoring & Profiling. Current API Analysis.

[Nikolay Izhikov-2]

Hello, Igniters.

For now, Ignite lack of good monitoring subsystem and profiling tools.
I think it's time to fix it!
I've created IEP-35 [1] for it.

Problems description:

1. Existing metrics are not documented(This is huge issue for our users, I know it, for sure).
2. Currently, there are different APIs for monitoring: Java API, JMX, Visor task, SQL System views. But,
the cover Ignite API partially.
3. Many Ignite subsystems and internal processes are hidden from user and administrator. Ignite is a black box for them.

I want to initiate first part of discussion: Current Ignite APIs for monitoring.

*What APIs are available for what IGNITE subsystems*?

I made some analysis for current APIs.
Please, validate my conclusion:

Monitoring completely unavailable:

        * Compute Grid(Visor tasks are broken)
        * Service Grid
        * Data streamers
        * Distributed Data Structures
        * Ignite messaging (Ignite#message)
        * 3-d party storage
        * ContinuousQuery
        * MVCC transactions
        * ML - *What should be available?*
        * Explicit locks

Monitoring API available:

        * PDS + offheap memory
                * Ignite#dataRegionMetrics
                * Ignite#dataStorageMetrics
                * Ignite#persistentStoreMetrics
        * Cache
                * IgniteCache#mxBean
                * IgniteCache#localMxBean
                * SQL system view: CACHE_GROUPS
                * SQL system view: LOCAL_CACHE_GROUPS_IO
                * SQL system view: CACHES

        * Queries
                * IgniteCache#queryMetrics
                * IgniteCache#queryDetailMetrics
                * SQL system view: LOCAL_SQL_QUERY_HISTORY
                * SQL system view: LOCAL_SQL_RUNNING_QUERIES
        * SQL
                * SQL system view: SCHEMAS
                * SQL system view: TABLES
                * SQL system view: INDEXES
        * Transactions
                * JMX - TransactionMetricsMxBean
                * JMX - TransactionMXBean
        * ThinClients
                * JMX - ClientProcessorMXBean

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392

[daradurvs]

Hi, Nikolay,

From my point of view, this is one of the most useful IEP for Ignite
end-users. I often face with a case when Ignite is a black-box for
administrators and devops engineers.

I'd like to have a feature to monitor PME's queue, to view its size
and events type. The same is actual for Service Grid deployments
queue.

Should we add this to EIP?

On Fri, Apr 19, 2019 at 3:49 PM Nikolay Izhikov <[hidden email]> wrote:

>
> Hello, Igniters.
>
> For now, Ignite lack of good monitoring subsystem and profiling tools.
> I think it's time to fix it!
> I've created IEP-35 [1] for it.
>
> Problems description:
>
> 1. Existing metrics are not documented(This is huge issue for our users, I know it, for sure).
> 2. Currently, there are different APIs for monitoring: Java API, JMX, Visor task, SQL System views. But,
> the cover Ignite API partially.
> 3. Many Ignite subsystems and internal processes are hidden from user and administrator. Ignite is a black box for them.
>
> I want to initiate first part of discussion: Current Ignite APIs for monitoring.
>
> *What APIs are available for what IGNITE subsystems*?
>
> I made some analysis for current APIs.
> Please, validate my conclusion:
>
> Monitoring completely unavailable:
>
>         *       Compute Grid(Visor tasks are broken)
>         *       Service Grid
>         *       Data streamers
>         *       Distributed Data Structures
>         *       Ignite messaging (Ignite#message)
>         *       3-d party storage
>         *       ContinuousQuery
>         *       MVCC transactions
>         *       ML - *What should be available?*
>         *       Explicit locks
>
> Monitoring API available:
>
>         * PDS + offheap memory
>                 * Ignite#dataRegionMetrics
>                 * Ignite#dataStorageMetrics
>                 * Ignite#persistentStoreMetrics
>         * Cache
>                 * IgniteCache#mxBean
>                 * IgniteCache#localMxBean
>                 * SQL system view: CACHE_GROUPS
>                 * SQL system view: LOCAL_CACHE_GROUPS_IO
>                 * SQL system view: CACHES
>
>         * Queries
>                 * IgniteCache#queryMetrics
>                 * IgniteCache#queryDetailMetrics
>                 * SQL system view: LOCAL_SQL_QUERY_HISTORY
>                 * SQL system view: LOCAL_SQL_RUNNING_QUERIES
>         * SQL
>                 * SQL system view: SCHEMAS
>                 * SQL system view: TABLES
>                 * SQL system view: INDEXES
>         * Transactions
>                 * JMX - TransactionMetricsMxBean
>                 * JMX - TransactionMXBean
>         * ThinClients
>                 * JMX - ClientProcessorMXBean
>
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392

--
Best Regards, Vyacheslav D.

[Denis Garus]

Hello!
We should not forget about monitoring security events.
From security, every fail of authentication or authorization is an
important event that may indicate an attack or configuration mistake.
In any case, grid admins should have tools to get these events on time.

вс, 21 апр. 2019 г. в 22:26, Vyacheslav Daradur <[hidden email]>:

> Hi, Nikolay,
>
> From my point of view, this is one of the most useful IEP for Ignite
> end-users. I often face with a case when Ignite is a black-box for
> administrators and devops engineers.
>
> I'd like to have a feature to monitor PME's queue, to view its size
> and events type. The same is actual for Service Grid deployments
> queue.
>
> Should we add this to EIP?
>
> On Fri, Apr 19, 2019 at 3:49 PM Nikolay Izhikov <[hidden email]>
> wrote:
> >
> > Hello, Igniters.
> >
> > For now, Ignite lack of good monitoring subsystem and profiling tools.
> > I think it's time to fix it!
> > I've created IEP-35 [1] for it.
> >
> > Problems description:
> >
> > 1. Existing metrics are not documented(This is huge issue for our users,
> I know it, for sure).
> > 2. Currently, there are different APIs for monitoring: Java API, JMX,
> Visor task, SQL System views. But,
> > the cover Ignite API partially.
> > 3. Many Ignite subsystems and internal processes are hidden from user
> and administrator. Ignite is a black box for them.
> >
> > I want to initiate first part of discussion: Current Ignite APIs for
> monitoring.
> >
> > *What APIs are available for what IGNITE subsystems*?
> >
> > I made some analysis for current APIs.
> > Please, validate my conclusion:
> >
> > Monitoring completely unavailable:
> >
> >         *       Compute Grid(Visor tasks are broken)
> >         *       Service Grid
> >         *       Data streamers
> >         *       Distributed Data Structures
> >         *       Ignite messaging (Ignite#message)
> >         *       3-d party storage
> >         *       ContinuousQuery
> >         *       MVCC transactions
> >         *       ML - *What should be available?*
> >         *       Explicit locks
> >
> > Monitoring API available:
> >
> >         * PDS + offheap memory
> >                 * Ignite#dataRegionMetrics
> >                 * Ignite#dataStorageMetrics
> >                 * Ignite#persistentStoreMetrics
> >         * Cache
> >                 * IgniteCache#mxBean
> >                 * IgniteCache#localMxBean
> >                 * SQL system view: CACHE_GROUPS
> >                 * SQL system view: LOCAL_CACHE_GROUPS_IO
> >                 * SQL system view: CACHES
> >
> >         * Queries
> >                 * IgniteCache#queryMetrics
> >                 * IgniteCache#queryDetailMetrics
> >                 * SQL system view: LOCAL_SQL_QUERY_HISTORY
> >                 * SQL system view: LOCAL_SQL_RUNNING_QUERIES
> >         * SQL
> >                 * SQL system view: SCHEMAS
> >                 * SQL system view: TABLES
> >                 * SQL system view: INDEXES
> >         * Transactions
> >                 * JMX - TransactionMetricsMxBean
> >                 * JMX - TransactionMXBean
> >         * ThinClients
> >                 * JMX - ClientProcessorMXBean
> >
> > [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
>
>
>
> --
> Best Regards, Vyacheslav D.
>

[Nikolay Izhikov-2]

Hello, Denis, Vyacheslav.

In Phase 1 I want to focus on monitoring subsystem and public API related activities.
I think Ignite should be able to show all entities that are started by the end user.

In subsequent phases, we can add monitoring of Ignite internal processes.
Anyway, I've added internal processes you mention in IEP.


В Пн, 22/04/2019 в 13:02 +0300, Denis Garus пишет:

> Hello!
> We should not forget about monitoring security events.
> From security, every fail of authentication or authorization is an
> important event that may indicate an attack or configuration mistake.
> In any case, grid admins should have tools to get these events on time.
>
> вс, 21 апр. 2019 г. в 22:26, Vyacheslav Daradur <[hidden email]>:
>
> > Hi, Nikolay,
> >
> > From my point of view, this is one of the most useful IEP for Ignite
> > end-users. I often face with a case when Ignite is a black-box for
> > administrators and devops engineers.
> >
> > I'd like to have a feature to monitor PME's queue, to view its size
> > and events type. The same is actual for Service Grid deployments
> > queue.
> >
> > Should we add this to EIP?
> >
> > On Fri, Apr 19, 2019 at 3:49 PM Nikolay Izhikov <[hidden email]>
> > wrote:
> > >
> > > Hello, Igniters.
> > >
> > > For now, Ignite lack of good monitoring subsystem and profiling tools.
> > > I think it's time to fix it!
> > > I've created IEP-35 [1] for it.
> > >
> > > Problems description:
> > >
> > > 1. Existing metrics are not documented(This is huge issue for our users,
> >
> > I know it, for sure).
> > > 2. Currently, there are different APIs for monitoring: Java API, JMX,
> >
> > Visor task, SQL System views. But,
> > > the cover Ignite API partially.
> > > 3. Many Ignite subsystems and internal processes are hidden from user
> >
> > and administrator. Ignite is a black box for them.
> > >
> > > I want to initiate first part of discussion: Current Ignite APIs for
> >
> > monitoring.
> > >
> > > *What APIs are available for what IGNITE subsystems*?
> > >
> > > I made some analysis for current APIs.
> > > Please, validate my conclusion:
> > >
> > > Monitoring completely unavailable:
> > >
> > >         *       Compute Grid(Visor tasks are broken)
> > >         *       Service Grid
> > >         *       Data streamers
> > >         *       Distributed Data Structures
> > >         *       Ignite messaging (Ignite#message)
> > >         *       3-d party storage
> > >         *       ContinuousQuery
> > >         *       MVCC transactions
> > >         *       ML - *What should be available?*
> > >         *       Explicit locks
> > >
> > > Monitoring API available:
> > >
> > >         * PDS + offheap memory
> > >                 * Ignite#dataRegionMetrics
> > >                 * Ignite#dataStorageMetrics
> > >                 * Ignite#persistentStoreMetrics
> > >         * Cache
> > >                 * IgniteCache#mxBean
> > >                 * IgniteCache#localMxBean
> > >                 * SQL system view: CACHE_GROUPS
> > >                 * SQL system view: LOCAL_CACHE_GROUPS_IO
> > >                 * SQL system view: CACHES
> > >
> > >         * Queries
> > >                 * IgniteCache#queryMetrics
> > >                 * IgniteCache#queryDetailMetrics
> > >                 * SQL system view: LOCAL_SQL_QUERY_HISTORY
> > >                 * SQL system view: LOCAL_SQL_RUNNING_QUERIES
> > >         * SQL
> > >                 * SQL system view: SCHEMAS
> > >                 * SQL system view: TABLES
> > >                 * SQL system view: INDEXES
> > >         * Transactions
> > >                 * JMX - TransactionMetricsMxBean
> > >                 * JMX - TransactionMXBean
> > >         * ThinClients
> > >                 * JMX - ClientProcessorMXBean
> > >
> > > [1]
> >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
> >