[DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager

[Nikolay Izhikov-2]

Hello, Igniters.

Since the last release `RunningQueryManager` [1] was added.
It used to track a running query.

In IEP-35 [2] SystemView API was added.
SystemView API supposed to be used to track all kinds of internal Ignite objects.

I think this RunningQueryManager should be replaced [3] with the more unified SystemView API.

Any objections?

[1] https://issues.apache.org/jira/browse/IGNITE-10754
[2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
[3] https://issues.apache.org/jira/browse/IGNITE-12223
[4] https://issues.apache.org/jira/browse/IGNITE-12224

[Ivan Pavlukhin]

Hi Nikolay,

Actually I do not fully understand what is SystemView API. I have not
found it in IEP [1] (I searched for words "system" and "view").

RunningQueryManager is a component responsible for tracking running
queries internally. This info is exposed to users as SQL view via
SqlSystemViewRunningQueries. In the same package you can find a plenty
of other SQL views.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392

вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov <[hidden email]>:

>
> Hello, Igniters.
>
> Since the last release `RunningQueryManager` [1] was added.
> It used to track a running query.
>
> In IEP-35 [2] SystemView API was added.
> SystemView API supposed to be used to track all kinds of internal Ignite objects.
>
> I think this RunningQueryManager should be replaced [3] with the more unified SystemView API.
>
> Any objections?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-10754
> [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> [3] https://issues.apache.org/jira/browse/IGNITE-12223
> [4] https://issues.apache.org/jira/browse/IGNITE-12224

--
Best regards,
Ivan Pavlukhin

[Nikolay Izhikov-2]

Hello, Ivan.

Thanks for feedback.

Initial IEP [1] naming was changed during code review.
I updated the IEP [1] with the current naming.

Can you take a look and check is all clear now?

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392

В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет:

> Hi Nikolay,
>
> Actually I do not fully understand what is SystemView API. I have not
> found it in IEP [1] (I searched for words "system" and "view").
>
> RunningQueryManager is a component responsible for tracking running
> queries internally. This info is exposed to users as SQL view via
> SqlSystemViewRunningQueries. In the same package you can find a plenty
> of other SQL views.
>
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
>
> вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov <[hidden email]>:
> >
> > Hello, Igniters.
> >
> > Since the last release `RunningQueryManager` [1] was added.
> > It used to track a running query.
> >
> > In IEP-35 [2] SystemView API was added.
> > SystemView API supposed to be used to track all kinds of internal Ignite objects.
> >
> > I think this RunningQueryManager should be replaced [3] with the more unified SystemView API.
> >
> > Any objections?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-10754
> > [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > [3] https://issues.apache.org/jira/browse/IGNITE-12223
> > [4] https://issues.apache.org/jira/browse/IGNITE-12224
>
>
>

[Ivan Pavlukhin]

Nikolay,

I checked the IEP [1]. Now it is more clear for me about SystemView
API. Follow-up questions:
1. How system views are going to be exposed? Is there any difference
in comparison to other metrics?
2. What should be done to adopt RunningQueryManager to SystemView API?

Also some bits for my understanding. I do not have a clear intuition
what is the difference between metrics and system views? For example,
how a system view is different from a metric holding a collection of
values? And why they were introduced as a separate class?

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392

чт, 3 окт. 2019 г. в 16:37, Nikolay Izhikov <[hidden email]>:

>
> Hello, Ivan.
>
> Thanks for feedback.
>
> Initial IEP [1] naming was changed during code review.
> I updated the IEP [1] with the current naming.
>
> Can you take a look and check is all clear now?
>
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
>
> В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет:
> > Hi Nikolay,
> >
> > Actually I do not fully understand what is SystemView API. I have not
> > found it in IEP [1] (I searched for words "system" and "view").
> >
> > RunningQueryManager is a component responsible for tracking running
> > queries internally. This info is exposed to users as SQL view via
> > SqlSystemViewRunningQueries. In the same package you can find a plenty
> > of other SQL views.
> >
> > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> >
> > вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov <[hidden email]>:
> > >
> > > Hello, Igniters.
> > >
> > > Since the last release `RunningQueryManager` [1] was added.
> > > It used to track a running query.
> > >
> > > In IEP-35 [2] SystemView API was added.
> > > SystemView API supposed to be used to track all kinds of internal Ignite objects.
> > >
> > > I think this RunningQueryManager should be replaced [3] with the more unified SystemView API.
> > >
> > > Any objections?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-10754
> > > [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > [3] https://issues.apache.org/jira/browse/IGNITE-12223
> > > [4] https://issues.apache.org/jira/browse/IGNITE-12224
> >
> >
> >

--
Best regards,
Ivan Pavlukhin

[Nikolay Izhikov-2]

Hello, Ivan.

> 1. How system views are going to be exposed? Is there any difference
> in comparison to other metrics?

We have a `SystemViewExporterSpi`.
Built-in implementations are `JmxSystemViewExporterSpi` and `SqlViewExporterSpi`.

> 2. What should be done to adopt RunningQueryManager to SystemView API?

I think we should replace `RunningQueryManager` with the special SystemView implementation.

> what is the difference between metrics and system views?

Actually, it's a very good question :)

System view is a collection of internal Ignite objects exported to a user.
Each system view is a table.

Metric is a value representing some instantaneous state of the internal Ignite object.
So its a "cell" of table.

We need metrics to build charts and history of processes.
We need system views to known what objects exist in node and its params.

В Пт, 04/10/2019 в 11:51 +0300, Ivan Pavlukhin пишет:

> Nikolay,
>
> I checked the IEP [1]. Now it is more clear for me about SystemView
> API. Follow-up questions:
> 1. How system views are going to be exposed? Is there any difference
> in comparison to other metrics?
> 2. What should be done to adopt RunningQueryManager to SystemView API?
>
> Also some bits for my understanding. I do not have a clear intuition
> what is the difference between metrics and system views? For example,
> how a system view is different from a metric holding a collection of
> values? And why they were introduced as a separate class?
>
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
>
> чт, 3 окт. 2019 г. в 16:37, Nikolay Izhikov <[hidden email]>:
> >
> > Hello, Ivan.
> >
> > Thanks for feedback.
> >
> > Initial IEP [1] naming was changed during code review.
> > I updated the IEP [1] with the current naming.
> >
> > Can you take a look and check is all clear now?
> >
> > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> >
> > В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет:
> > > Hi Nikolay,
> > >
> > > Actually I do not fully understand what is SystemView API. I have not
> > > found it in IEP [1] (I searched for words "system" and "view").
> > >
> > > RunningQueryManager is a component responsible for tracking running
> > > queries internally. This info is exposed to users as SQL view via
> > > SqlSystemViewRunningQueries. In the same package you can find a plenty
> > > of other SQL views.
> > >
> > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > >
> > > вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov <[hidden email]>:
> > > >
> > > > Hello, Igniters.
> > > >
> > > > Since the last release `RunningQueryManager` [1] was added.
> > > > It used to track a running query.
> > > >
> > > > In IEP-35 [2] SystemView API was added.
> > > > SystemView API supposed to be used to track all kinds of internal Ignite objects.
> > > >
> > > > I think this RunningQueryManager should be replaced [3] with the more unified SystemView API.
> > > >
> > > > Any objections?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-10754
> > > > [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > [3] https://issues.apache.org/jira/browse/IGNITE-12223
> > > > [4] https://issues.apache.org/jira/browse/IGNITE-12224
> > >
> > >
> > >
>
>
>

[Ivan Pavlukhin]

Nikolay,

Thank you for sharing knowledge.

> I think we should replace `RunningQueryManager` with the special SystemView implementation.

Not sure that I got the intention and abstraction here. For me a
straightforward approach here is to keep RunningQueryManager as is and
use a new API to expose it's content to monitoring system.
RunningQueryManager is responsible for tracking running queries (and
query history). All in all, other views expose info from other
managers and processors (e.g. IgniteTxManager, GridTaskProcessor,
SchemaManager). Have I missed something?

пт, 4 окт. 2019 г. в 14:12, Nikolay Izhikov <[hidden email]>:

>
> Hello, Ivan.
>
> > 1. How system views are going to be exposed? Is there any difference
> > in comparison to other metrics?
>
> We have a `SystemViewExporterSpi`.
> Built-in implementations are `JmxSystemViewExporterSpi` and `SqlViewExporterSpi`.
>
> > 2. What should be done to adopt RunningQueryManager to SystemView API?
>
> I think we should replace `RunningQueryManager` with the special SystemView implementation.
>
> > what is the difference between metrics and system views?
>
> Actually, it's a very good question :)
>
> System view is a collection of internal Ignite objects exported to a user.
> Each system view is a table.
>
> Metric is a value representing some instantaneous state of the internal Ignite object.
> So its a "cell" of table.
>
> We need metrics to build charts and history of processes.
> We need system views to known what objects exist in node and its params.
>
> В Пт, 04/10/2019 в 11:51 +0300, Ivan Pavlukhin пишет:
> > Nikolay,
> >
> > I checked the IEP [1]. Now it is more clear for me about SystemView
> > API. Follow-up questions:
> > 1. How system views are going to be exposed? Is there any difference
> > in comparison to other metrics?
> > 2. What should be done to adopt RunningQueryManager to SystemView API?
> >
> > Also some bits for my understanding. I do not have a clear intuition
> > what is the difference between metrics and system views? For example,
> > how a system view is different from a metric holding a collection of
> > values? And why they were introduced as a separate class?
> >
> > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> >
> > чт, 3 окт. 2019 г. в 16:37, Nikolay Izhikov <[hidden email]>:
> > >
> > > Hello, Ivan.
> > >
> > > Thanks for feedback.
> > >
> > > Initial IEP [1] naming was changed during code review.
> > > I updated the IEP [1] with the current naming.
> > >
> > > Can you take a look and check is all clear now?
> > >
> > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > >
> > > В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет:
> > > > Hi Nikolay,
> > > >
> > > > Actually I do not fully understand what is SystemView API. I have not
> > > > found it in IEP [1] (I searched for words "system" and "view").
> > > >
> > > > RunningQueryManager is a component responsible for tracking running
> > > > queries internally. This info is exposed to users as SQL view via
> > > > SqlSystemViewRunningQueries. In the same package you can find a plenty
> > > > of other SQL views.
> > > >
> > > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > >
> > > > вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov <[hidden email]>:
> > > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > Since the last release `RunningQueryManager` [1] was added.
> > > > > It used to track a running query.
> > > > >
> > > > > In IEP-35 [2] SystemView API was added.
> > > > > SystemView API supposed to be used to track all kinds of internal Ignite objects.
> > > > >
> > > > > I think this RunningQueryManager should be replaced [3] with the more unified SystemView API.
> > > > >
> > > > > Any objections?
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10754
> > > > > [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > > [3] https://issues.apache.org/jira/browse/IGNITE-12223
> > > > > [4] https://issues.apache.org/jira/browse/IGNITE-12224
> > > >
> > > >
> > > >
> >
> >
> >

--
Best regards,
Ivan Pavlukhin

[Nikolay Izhikov-2]

Ivan.

> RunningQueryManager is responsible for tracking running queries (and query history)

As I understand, RunningQueryManager track query only for an export.
So we don't need explicit entity for that, we already have System Views.

В Пт, 04/10/2019 в 17:40 +0300, Ivan Pavlukhin пишет:

> Nikolay,
>
> Thank you for sharing knowledge.
>
> > I think we should replace `RunningQueryManager` with the special SystemView implementation.
>
> Not sure that I got the intention and abstraction here. For me a
> straightforward approach here is to keep RunningQueryManager as is and
> use a new API to expose it's content to monitoring system.
> RunningQueryManager is responsible for tracking running queries (and
> query history). All in all, other views expose info from other
> managers and processors (e.g. IgniteTxManager, GridTaskProcessor,
> SchemaManager). Have I missed something?
>
> пт, 4 окт. 2019 г. в 14:12, Nikolay Izhikov <[hidden email]>:
> >
> > Hello, Ivan.
> >
> > > 1. How system views are going to be exposed? Is there any difference
> > > in comparison to other metrics?
> >
> > We have a `SystemViewExporterSpi`.
> > Built-in implementations are `JmxSystemViewExporterSpi` and `SqlViewExporterSpi`.
> >
> > > 2. What should be done to adopt RunningQueryManager to SystemView API?
> >
> > I think we should replace `RunningQueryManager` with the special SystemView implementation.
> >
> > > what is the difference between metrics and system views?
> >
> > Actually, it's a very good question :)
> >
> > System view is a collection of internal Ignite objects exported to a user.
> > Each system view is a table.
> >
> > Metric is a value representing some instantaneous state of the internal Ignite object.
> > So its a "cell" of table.
> >
> > We need metrics to build charts and history of processes.
> > We need system views to known what objects exist in node and its params.
> >
> > В Пт, 04/10/2019 в 11:51 +0300, Ivan Pavlukhin пишет:
> > > Nikolay,
> > >
> > > I checked the IEP [1]. Now it is more clear for me about SystemView
> > > API. Follow-up questions:
> > > 1. How system views are going to be exposed? Is there any difference
> > > in comparison to other metrics?
> > > 2. What should be done to adopt RunningQueryManager to SystemView API?
> > >
> > > Also some bits for my understanding. I do not have a clear intuition
> > > what is the difference between metrics and system views? For example,
> > > how a system view is different from a metric holding a collection of
> > > values? And why they were introduced as a separate class?
> > >
> > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > >
> > > чт, 3 окт. 2019 г. в 16:37, Nikolay Izhikov <[hidden email]>:
> > > >
> > > > Hello, Ivan.
> > > >
> > > > Thanks for feedback.
> > > >
> > > > Initial IEP [1] naming was changed during code review.
> > > > I updated the IEP [1] with the current naming.
> > > >
> > > > Can you take a look and check is all clear now?
> > > >
> > > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > >
> > > > В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет:
> > > > > Hi Nikolay,
> > > > >
> > > > > Actually I do not fully understand what is SystemView API. I have not
> > > > > found it in IEP [1] (I searched for words "system" and "view").
> > > > >
> > > > > RunningQueryManager is a component responsible for tracking running
> > > > > queries internally. This info is exposed to users as SQL view via
> > > > > SqlSystemViewRunningQueries. In the same package you can find a plenty
> > > > > of other SQL views.
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > >
> > > > > вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov <[hidden email]>:
> > > > > >
> > > > > > Hello, Igniters.
> > > > > >
> > > > > > Since the last release `RunningQueryManager` [1] was added.
> > > > > > It used to track a running query.
> > > > > >
> > > > > > In IEP-35 [2] SystemView API was added.
> > > > > > SystemView API supposed to be used to track all kinds of internal Ignite objects.
> > > > > >
> > > > > > I think this RunningQueryManager should be replaced [3] with the more unified SystemView API.
> > > > > >
> > > > > > Any objections?
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10754
> > > > > > [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > > > [3] https://issues.apache.org/jira/browse/IGNITE-12223
> > > > > > [4] https://issues.apache.org/jira/browse/IGNITE-12224
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
>
>
>

[Ivan Pavlukhin]

Nikolay,

> As I understand, RunningQueryManager track query only for an export.

Not quite. Also it is responsible for explicit query cancellation and
running queries cancellation on node stop. I do not think that a view
should be responsible for it.

пт, 4 окт. 2019 г. в 18:17, Nikolay Izhikov <[hidden email]>:

>
> Ivan.
>
> > RunningQueryManager is responsible for tracking running queries (and query history)
>
> As I understand, RunningQueryManager track query only for an export.
> So we don't need explicit entity for that, we already have System Views.
>
> В Пт, 04/10/2019 в 17:40 +0300, Ivan Pavlukhin пишет:
> > Nikolay,
> >
> > Thank you for sharing knowledge.
> >
> > > I think we should replace `RunningQueryManager` with the special SystemView implementation.
> >
> > Not sure that I got the intention and abstraction here. For me a
> > straightforward approach here is to keep RunningQueryManager as is and
> > use a new API to expose it's content to monitoring system.
> > RunningQueryManager is responsible for tracking running queries (and
> > query history). All in all, other views expose info from other
> > managers and processors (e.g. IgniteTxManager, GridTaskProcessor,
> > SchemaManager). Have I missed something?
> >
> > пт, 4 окт. 2019 г. в 14:12, Nikolay Izhikov <[hidden email]>:
> > >
> > > Hello, Ivan.
> > >
> > > > 1. How system views are going to be exposed? Is there any difference
> > > > in comparison to other metrics?
> > >
> > > We have a `SystemViewExporterSpi`.
> > > Built-in implementations are `JmxSystemViewExporterSpi` and `SqlViewExporterSpi`.
> > >
> > > > 2. What should be done to adopt RunningQueryManager to SystemView API?
> > >
> > > I think we should replace `RunningQueryManager` with the special SystemView implementation.
> > >
> > > > what is the difference between metrics and system views?
> > >
> > > Actually, it's a very good question :)
> > >
> > > System view is a collection of internal Ignite objects exported to a user.
> > > Each system view is a table.
> > >
> > > Metric is a value representing some instantaneous state of the internal Ignite object.
> > > So its a "cell" of table.
> > >
> > > We need metrics to build charts and history of processes.
> > > We need system views to known what objects exist in node and its params.
> > >
> > > В Пт, 04/10/2019 в 11:51 +0300, Ivan Pavlukhin пишет:
> > > > Nikolay,
> > > >
> > > > I checked the IEP [1]. Now it is more clear for me about SystemView
> > > > API. Follow-up questions:
> > > > 1. How system views are going to be exposed? Is there any difference
> > > > in comparison to other metrics?
> > > > 2. What should be done to adopt RunningQueryManager to SystemView API?
> > > >
> > > > Also some bits for my understanding. I do not have a clear intuition
> > > > what is the difference between metrics and system views? For example,
> > > > how a system view is different from a metric holding a collection of
> > > > values? And why they were introduced as a separate class?
> > > >
> > > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > >
> > > > чт, 3 окт. 2019 г. в 16:37, Nikolay Izhikov <[hidden email]>:
> > > > >
> > > > > Hello, Ivan.
> > > > >
> > > > > Thanks for feedback.
> > > > >
> > > > > Initial IEP [1] naming was changed during code review.
> > > > > I updated the IEP [1] with the current naming.
> > > > >
> > > > > Can you take a look and check is all clear now?
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > >
> > > > > В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет:
> > > > > > Hi Nikolay,
> > > > > >
> > > > > > Actually I do not fully understand what is SystemView API. I have not
> > > > > > found it in IEP [1] (I searched for words "system" and "view").
> > > > > >
> > > > > > RunningQueryManager is a component responsible for tracking running
> > > > > > queries internally. This info is exposed to users as SQL view via
> > > > > > SqlSystemViewRunningQueries. In the same package you can find a plenty
> > > > > > of other SQL views.
> > > > > >
> > > > > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > > >
> > > > > > вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov <[hidden email]>:
> > > > > > >
> > > > > > > Hello, Igniters.
> > > > > > >
> > > > > > > Since the last release `RunningQueryManager` [1] was added.
> > > > > > > It used to track a running query.
> > > > > > >
> > > > > > > In IEP-35 [2] SystemView API was added.
> > > > > > > SystemView API supposed to be used to track all kinds of internal Ignite objects.
> > > > > > >
> > > > > > > I think this RunningQueryManager should be replaced [3] with the more unified SystemView API.
> > > > > > >
> > > > > > > Any objections?
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10754
> > > > > > > [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > > > > [3] https://issues.apache.org/jira/browse/IGNITE-12223
> > > > > > > [4] https://issues.apache.org/jira/browse/IGNITE-12224
> > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> >
> >
> >

--
Best regards,
Ivan Pavlukhin