Apache Ignite Developers - Legacy Mail Archive

New Monitoring System: items to complete before GA

Classic

List

Threaded

4 messages Options

dmagda

New Monitoring System: items to complete before GA

Igniters,

There was a private discussion between Nikolay Izhikov, Andrey Gura, Alex
Goncharuk and I some time ago where we attempted to outline a list of tasks
to complete before announcing the feature in the GA state (removing the
experimental flag from the new APIs and deprecating the legacy ones).
Folks, with your permission, let me share the list first and next we can
talk out each item in detail separately:

1. Add ability to enable\disable subset of the metrics for collection:
https://issues.apache.org/jira/browse/IGNITE-11927
Am I right that the task will let us perform a desired configuration
MetricExporterSPI and its specific implementation? Frankly, I couldn't
figure out how to do that
with "setExportFilter(Predicate<ReadOnlyMetricRegistry> filter);" method --
probably, that's just the matter of documentation.

2. Extension of IgniteSpiContext in order to allow Ignite developers to
implement their own exporters (with no usage of the internal API).

3. Adding/extending/removing public metrics related to the API
facade. Andrey, please elaborate on this item if needed.

Also, I had a chance to experience the API in wearing a hat of an
application developer while preparing for the Ignite 2.8 webinar. Please
take into account my feedback below. Overall, I was impressed with the way
the new system is designed and operates. All I had to do is just to
register exporters and connect my favorite tool to the cluster. Kudos to
Nikolay, Andrey and everyone else involved. So, that's my list and I ready
to create JIRA tickets whenever is relevant:

4. After registering an exporter with a node only a subset of metrics gets
updated. For instance, "GetTime_X" set of metrics belonging to a specific
cache get updated all the times while CacheGets/Puts/Hits require me to
enable "setStatisticsEnable(true)" for the cache. This is extremely
confusing. Any way we can remove this ambiguity? Guess the same applies for
memory and persistence metrics - there are old ones that might require to
turn on "metricsEnabled" flag for data regions.

5. The system view that should track compute tasks (views.tasks) is always
empty and I can see a number of tasks running via the compute.jobs registry
or in "views.views" mxbean. Again, confusing from the application developer
standpoint.

6. Ignite MBeans are listed under "org.apache.{node_id}" but should be
placed under "org.apache.ignite.{node_id}". Guess this is left as is for
now to preserve backward compatibility.

7. You see hundreds of metrics once connect through JMX and it's hard to
sort out the valuable ones. The documentation can do a great job outlining
the right ones but we can also add descriptions to MXBeans explaining what
a metric is used for.

8. Failed to use LogExporterSpi:
https://issues.apache.org/jira/browse/IGNITE-12880

-
Denis

Nikolay Izhikov-2

Re: New Monitoring System: items to complete before GA

> 1. Add ability to enable\disable subset of the metrics for collection:
> https://issues.apache.org/jira/browse/IGNITE-11927
> Am I right that the task will let us perform a desired configuration
MetricExporterSPI and its specific implementation?

This improvement should allow Ignite users to disable or enable
`MetricsRegistry` globally on the node.
When some `MetricRegistry` disabled then registry:
* Do not consume resources(CPU, RAM) to keep metrics values up to
date(don’t perform increment, decrement operations, etc)
* Can’t be viewed in any exporter.

> Frankly, I couldn't figure out how to do that with
"setExportFilter(Predicate<ReadOnlyMetricRegistry> filter);"

With filter user can exclude arbitrary registry from arbitrary exporter.
Example of filtering out all cache metrics from JMX:

```
JmxMetricExporterSpi jmxSpi = new JmxMetricExporterSpi();
jmxSpi.setExportFilter(mreg -> !mreg.name().startsWith(«cache»));

cfg.setMetricExporterSpi(jmxSpi);
```

> 2. Extension of IgniteSpiContext in order to allow Ignite developers to
implement their own exporters

Please, clarify, your idea.
Exporters can be implemented without usage of the internal API.
You can look into JmxMetricExporterSpi as an example.

> 3. Adding/extending/removing public metrics related to the API facade.

I think we can start with a discussion and finishing Java API to read
metrics.
It implemented in my PR [1].

> 4. After registering an exporter with a node only a subset of metrics
gets updated. This is extremely confusing. Any way we can remove this
ambiguity?

Yes, it a bad thing.
And it’s known restriction of the current implementation that was
introduced to preserve backward compatibility.
It seems we can remove this restriction after IGNITE-11927 would be
implemented.

> 5. The system view that should track compute tasks (views.tasks) is
always empty

TASKS reflects the running task that was started on the local node.
You can find an example of using this view in tests [2]
It seems we should add examples for the views.

The view to observe compute JOBS(parts of the task that are executed on the
remote node) was added by me in [3]
It’s not a part of 2.8.0 release but should be available in 2.8.1

> 6. Ignite MBeans are listed under "org.apache.{node_id}" but should be
placed under "org.apache.ignite.{node_id}".

Please, create a ticket and I will try to fix it.

> we can also add descriptions to MXBeans explaining what a metric is used
for.

Great idea! Let’s do it, for sure!
Please, create a ticket for it

> 8. Failed to use LogExporterSpi:

On my plate, I will fix it.

[1] https://github.com/apache/ignite/pull/7283
[2]
https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/metric/SystemViewSelfTest.java#L236
[3] https://issues.apache.org/jira/browse/IGNITE-12745

ср, 8 апр. 2020 г. в 23:10, Denis Magda <[hidden email]>:

> Igniters,
>
> There was a private discussion between Nikolay Izhikov, Andrey Gura, Alex
> Goncharuk and I some time ago where we attempted to outline a list of tasks
> to complete before announcing the feature in the GA state (removing the
> experimental flag from the new APIs and deprecating the legacy ones).
> Folks, with your permission, let me share the list first and next we can
> talk out each item in detail separately:
>
> 1. Add ability to enable\disable subset of the metrics for collection:
> https://issues.apache.org/jira/browse/IGNITE-11927
> Am I right that the task will let us perform a desired configuration
> MetricExporterSPI and its specific implementation? Frankly, I couldn't
> figure out how to do that
> with "setExportFilter(Predicate<ReadOnlyMetricRegistry> filter);" method --
> probably, that's just the matter of documentation.
>
> 2. Extension of IgniteSpiContext in order to allow Ignite developers to
> implement their own exporters (with no usage of the internal API).
>
> 3. Adding/extending/removing public metrics related to the API
> facade. Andrey, please elaborate on this item if needed.
>
> Also, I had a chance to experience the API in wearing a hat of an
> application developer while preparing for the Ignite 2.8 webinar. Please
> take into account my feedback below. Overall, I was impressed with the way
> the new system is designed and operates. All I had to do is just to
> register exporters and connect my favorite tool to the cluster. Kudos to
> Nikolay, Andrey and everyone else involved. So, that's my list and I ready
> to create JIRA tickets whenever is relevant:
>
> 4. After registering an exporter with a node only a subset of metrics gets
> updated. For instance, "GetTime_X" set of metrics belonging to a specific
> cache get updated all the times while CacheGets/Puts/Hits require me to
> enable "setStatisticsEnable(true)" for the cache. This is extremely
> confusing. Any way we can remove this ambiguity? Guess the same applies for
> memory and persistence metrics - there are old ones that might require to
> turn on "metricsEnabled" flag for data regions.
>
> 5. The system view that should track compute tasks (views.tasks) is always
> empty and I can see a number of tasks running via the compute.jobs
> registry
> or in "views.views" mxbean. Again, confusing from the application developer
> standpoint.
>
> 6. Ignite MBeans are listed under "org.apache.{node_id}" but should be
> placed under "org.apache.ignite.{node_id}". Guess this is left as is for
> now to preserve backward compatibility.
>
> 7. You see hundreds of metrics once connect through JMX and it's hard to
> sort out the valuable ones. The documentation can do a great job outlining
> the right ones but we can also add descriptions to MXBeans explaining what
> a metric is used for.
>
> 8. Failed to use LogExporterSpi:
> https://issues.apache.org/jira/browse/IGNITE-12880
>
> -
> Denis
>

dmagda

Re: New Monitoring System: items to complete before GA

Nikolay, thanks for joining the discussion.

Please give me a few days to respond and create tickets. A bit hectic week.

-
Denis

On Sat, Apr 18, 2020 at 6:10 AM Nikolay Izhikov <[hidden email]> wrote:

> > 1. Add ability to enable\disable subset of the metrics for collection:
> > https://issues.apache.org/jira/browse/IGNITE-11927
> > Am I right that the task will let us perform a desired configuration
> MetricExporterSPI and its specific implementation?
>
> This improvement should allow Ignite users to disable or enable
> `MetricsRegistry` globally on the node.
> When some `MetricRegistry` disabled then registry:
> * Do not consume resources(CPU, RAM) to keep metrics values up to
> date(don’t perform increment, decrement operations, etc)
> * Can’t be viewed in any exporter.
>
> > Frankly, I couldn't figure out how to do that with
> "setExportFilter(Predicate<ReadOnlyMetricRegistry> filter);"
>
> With filter user can exclude arbitrary registry from arbitrary exporter.
> Example of filtering out all cache metrics from JMX:
>
> ```
> JmxMetricExporterSpi jmxSpi = new JmxMetricExporterSpi();
> jmxSpi.setExportFilter(mreg -> !mreg.name().startsWith(«cache»));
>
> cfg.setMetricExporterSpi(jmxSpi);
> ```
>
> > 2. Extension of IgniteSpiContext in order to allow Ignite developers to
> implement their own exporters
>
> Please, clarify, your idea.
> Exporters can be implemented without usage of the internal API.
> You can look into JmxMetricExporterSpi as an example.
>
> > 3. Adding/extending/removing public metrics related to the API facade.
>
> I think we can start with a discussion and finishing Java API to read
> metrics.
> It implemented in my PR [1].
>
> > 4. After registering an exporter with a node only a subset of metrics
> gets updated. This is extremely confusing. Any way we can remove this
> ambiguity?
>
> Yes, it a bad thing.
> And it’s known restriction of the current implementation that was
> introduced to preserve backward compatibility.
> It seems we can remove this restriction after IGNITE-11927 would be
> implemented.
>
> > 5. The system view that should track compute tasks (views.tasks) is
> always empty
>
> TASKS reflects the running task that was started on the local node.
> You can find an example of using this view in tests [2]
> It seems we should add examples for the views.
>
> The view to observe compute JOBS(parts of the task that are executed on the
> remote node) was added by me in [3]
> It’s not a part of 2.8.0 release but should be available in 2.8.1
>
> > 6. Ignite MBeans are listed under "org.apache.{node_id}" but should be
> placed under "org.apache.ignite.{node_id}".
>
> Please, create a ticket and I will try to fix it.
>
> > we can also add descriptions to MXBeans explaining what a metric is used
> for.
>
> Great idea! Let’s do it, for sure!
> Please, create a ticket for it
>
> > 8. Failed to use LogExporterSpi:
>
> On my plate, I will fix it.
>
> [1] https://github.com/apache/ignite/pull/7283
> [2]
>
> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/metric/SystemViewSelfTest.java#L236
> [3] https://issues.apache.org/jira/browse/IGNITE-12745
>
>
> ср, 8 апр. 2020 г. в 23:10, Denis Magda <[hidden email]>:
>
> > Igniters,
> >
> > There was a private discussion between Nikolay Izhikov, Andrey Gura, Alex
> > Goncharuk and I some time ago where we attempted to outline a list of
> tasks
> > to complete before announcing the feature in the GA state (removing the
> > experimental flag from the new APIs and deprecating the legacy ones).
> > Folks, with your permission, let me share the list first and next we can
> > talk out each item in detail separately:
> >
> > 1. Add ability to enable\disable subset of the metrics for collection:
> > https://issues.apache.org/jira/browse/IGNITE-11927
> > Am I right that the task will let us perform a desired configuration
> > MetricExporterSPI and its specific implementation? Frankly, I couldn't
> > figure out how to do that
> > with "setExportFilter(Predicate<ReadOnlyMetricRegistry> filter);" method
> --
> > probably, that's just the matter of documentation.
> >
> > 2. Extension of IgniteSpiContext in order to allow Ignite developers to
> > implement their own exporters (with no usage of the internal API).
> >
> > 3. Adding/extending/removing public metrics related to the API
> > facade. Andrey, please elaborate on this item if needed.
> >
> > Also, I had a chance to experience the API in wearing a hat of an
> > application developer while preparing for the Ignite 2.8 webinar. Please
> > take into account my feedback below. Overall, I was impressed with the
> way
> > the new system is designed and operates. All I had to do is just to
> > register exporters and connect my favorite tool to the cluster. Kudos to
> > Nikolay, Andrey and everyone else involved. So, that's my list and I
> ready
> > to create JIRA tickets whenever is relevant:
> >
> > 4. After registering an exporter with a node only a subset of metrics
> gets
> > updated. For instance, "GetTime_X" set of metrics belonging to a specific
> > cache get updated all the times while CacheGets/Puts/Hits require me to
> > enable "setStatisticsEnable(true)" for the cache. This is extremely
> > confusing. Any way we can remove this ambiguity? Guess the same applies
> for
> > memory and persistence metrics - there are old ones that might require to
> > turn on "metricsEnabled" flag for data regions.
> >
> > 5. The system view that should track compute tasks (views.tasks) is
> always
> > empty and I can see a number of tasks running via the compute.jobs
> > registry
> > or in "views.views" mxbean. Again, confusing from the application
> developer
> > standpoint.
> >
> > 6. Ignite MBeans are listed under "org.apache.{node_id}" but should be
> > placed under "org.apache.ignite.{node_id}". Guess this is left as is for
> > now to preserve backward compatibility.
> >
> > 7. You see hundreds of metrics once connect through JMX and it's hard to
> > sort out the valuable ones. The documentation can do a great job
> outlining
> > the right ones but we can also add descriptions to MXBeans explaining
> what
> > a metric is used for.
> >
> > 8. Failed to use LogExporterSpi:
> > https://issues.apache.org/jira/browse/IGNITE-12880
> >
> > -
> > Denis
> >
>

dmagda

Re: New Monitoring System: items to complete before GA

In reply to this post by Nikolay Izhikov-2

Nikolay, thanks for a detailed response.

(I've created a handful of tickets linking to IEP-35 with the label named
similarly. I didn't assign any tickets on you letting you decide what you'd
like to take over personally)

With filter user can exclude arbitrary registry from arbitrary exporter.
> Example of filtering out all cache metrics from JMX:

Gotcha, now it's clear. I would add this to the documentation if you don't
mind:
https://issues.apache.org/jira/browse/IGNITE-12943

> 2. Extension of IgniteSpiContext in order to allow Ignite developers to
> implement their own exporters
> Please, clarify, your idea.
> Exporters can be implemented without usage of the internal API.
> You can look into JmxMetricExporterSpi as an example.

@Andrey Gura <[hidden email]>, could you please join and clarify this
requirement?

> 4. After registering an exporter with a node only a subset of metrics
> gets updated. This is extremely confusing. Any way we can remove this
> ambiguity?
> Yes, it a bad thing.
> And it’s known restriction of the current implementation that was
> introduced to preserve backward compatibility.
> It seems we can remove this restriction after IGNITE-11927 would be
> implemented.

Agree, created a ticket we can jump on after IGNITE-11927 is ready:
https://issues.apache.org/jira/browse/IGNITE-12944

In the meantime, I've added callout "Enabling Collection of Legacy Metrics"
to the first section of the documentation page:
https://apacheignite.readme.io/docs/new-metrics

> 5. The system view that should track compute tasks (views.tasks) is
> always empty
> TASKS reflects the running task that was started on the local node.
> You can find an example of using this view in tests [2]
> It seems we should add examples for the views.

I've updated the documentation putting an example. Please check if it
sounds correct:
https://apacheignite.readme.io/docs/tasks

> > 6. Ignite MBeans are listed under "org.apache.{node_id}" but should be
> placed under "org.apache.ignite.{node_id}".
> Please, create a ticket and I will try to fix it.

https://issues.apache.org/jira/browse/IGNITE-12945

> we can also add descriptions to MXBeans explaining what a metric is used
> for.
> Great idea! Let’s do it, for sure!
> Please, create a ticket for it

Cool! https://issues.apache.org/jira/browse/IGNITE-12946
-
Denis

On Sat, Apr 18, 2020 at 6:10 AM Nikolay Izhikov <[hidden email]> wrote: