Tool for performance statistics reports

[Nikita Amelchev]

Hi, Igniters.

I'm working on cluster profiling and the tool for creating a
performance report. [1] I have prepared PoC based on performance
logging to a separate category of Ignite log. The report contains:

- Cache operations and its distribution by types [2]
- Transactions and histogram of durations [3]
- SQL and Scan query statistics, top of slowest queries, logical and
physical reads by query [4]
- Compute statistics, top of slowest tasks and their jobs [5]
Soon I will add:
- Topology and Ignite versions info
- Client ID in case of operations from clients

For now, I'm developing a binary logging format to reduce the effect
on performance. I'll try to reuse Ignite mechanisms.

I would like to hear suggestions for the profiling and the performance report.

[1] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
[2] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
[3] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
[4] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
[5] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png

--
Best wishes,
Amelchev Nikita

[Maxim Muzafarov]

Hello, Nikita

Nice to have such a tool in Apache Ignite.

Can you provide more information about the binary logging format?
Do you have any examples?
Is there are any problems you faced?

>The Ignite will log some additional internal statistics using separate log category of IgniteLogger.
What the additional log category are you talking about?

On Fri, 24 Apr 2020 at 14:58, Nikita Amelchev <[hidden email]> wrote:

>
> Hi, Igniters.
>
> I'm working on cluster profiling and the tool for creating a
> performance report. [1] I have prepared PoC based on performance
> logging to a separate category of Ignite log. The report contains:
>
> - Cache operations and its distribution by types [2]
> - Transactions and histogram of durations [3]
> - SQL and Scan query statistics, top of slowest queries, logical and
> physical reads by query [4]
> - Compute statistics, top of slowest tasks and their jobs [5]
> Soon I will add:
> - Topology and Ignite versions info
> - Client ID in case of operations from clients
>
> For now, I'm developing a binary logging format to reduce the effect
> on performance. I'll try to reuse Ignite mechanisms.
>
> I would like to hear suggestions for the profiling and the performance report.
>
> [1] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> [2] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> [3] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> [4] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> [5] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
>
> --
> Best wishes,
> Amelchev Nikita

[slava.koptilin]

Hello Nikita,

Perhaps, I am missing something...
Apache Ignite already has a web-console tool. Do we want to improve the
existing tool instead of creating a new one?
It seems to me, this can be confusing for users.
 - visor (which is deprecated)
 - web-console (to be honest, I don't quite understand the status of this
tool)
 - new ignite-profiling (which is a monitoring tool as well, judging by the
provided link [1] )

I am not against the new tool, I just want to understand the motivation to
not improve the existing sub-projects.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool

Thanks,
S.

пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]>:

> Hi, Igniters.
>
> I'm working on cluster profiling and the tool for creating a
> performance report. [1] I have prepared PoC based on performance
> logging to a separate category of Ignite log. The report contains:
>
> - Cache operations and its distribution by types [2]
> - Transactions and histogram of durations [3]
> - SQL and Scan query statistics, top of slowest queries, logical and
> physical reads by query [4]
> - Compute statistics, top of slowest tasks and their jobs [5]
> Soon I will add:
> - Topology and Ignite versions info
> - Client ID in case of operations from clients
>
> For now, I'm developing a binary logging format to reduce the effect
> on performance. I'll try to reuse Ignite mechanisms.
>
> I would like to hear suggestions for the profiling and the performance
> report.
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> [2]
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> [3]
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> [4]
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> [5]
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
>
> --
> Best wishes,
> Amelchev Nikita
>

[Ilya Kasnacheev]

Hello!

You need to throw in control.sh also, which does some kind of statistics
too, such as idle_verify.

Regards,
--
Ilya Kasnacheev


пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <[hidden email]>:

> Hello Nikita,
>
> Perhaps, I am missing something...
> Apache Ignite already has a web-console tool. Do we want to improve the
> existing tool instead of creating a new one?
> It seems to me, this can be confusing for users.
>  - visor (which is deprecated)
>  - web-console (to be honest, I don't quite understand the status of this
> tool)
>  - new ignite-profiling (which is a monitoring tool as well, judging by the
> provided link [1] )
>
> I am not against the new tool, I just want to understand the motivation to
> not improve the existing sub-projects.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>
> Thanks,
> S.
>
> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]>:
>
> > Hi, Igniters.
> >
> > I'm working on cluster profiling and the tool for creating a
> > performance report. [1] I have prepared PoC based on performance
> > logging to a separate category of Ignite log. The report contains:
> >
> > - Cache operations and its distribution by types [2]
> > - Transactions and histogram of durations [3]
> > - SQL and Scan query statistics, top of slowest queries, logical and
> > physical reads by query [4]
> > - Compute statistics, top of slowest tasks and their jobs [5]
> > Soon I will add:
> > - Topology and Ignite versions info
> > - Client ID in case of operations from clients
> >
> > For now, I'm developing a binary logging format to reduce the effect
> > on performance. I'll try to reuse Ignite mechanisms.
> >
> > I would like to hear suggestions for the profiling and the performance
> > report.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > [2]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > [3]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > [4]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > [5]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> >
> > --
> > Best wishes,
> > Amelchev Nikita
> >
>

[Nikita Amelchev]

Maxim,

> Can you provide more information about the binary logging format?
I started to investigate. It should be with minimal effect on
performance. I haven't any details for now.

> What the additional log category are you talking about?
PoC is based on logging performance statistics to the "profiling"
category to split with other logging and be able to log to a separate
file. See IgniteLogger#getLogger.

пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <[hidden email]>:

>
> Hello!
>
> You need to throw in control.sh also, which does some kind of statistics
> too, such as idle_verify.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <[hidden email]>:
>
> > Hello Nikita,
> >
> > Perhaps, I am missing something...
> > Apache Ignite already has a web-console tool. Do we want to improve the
> > existing tool instead of creating a new one?
> > It seems to me, this can be confusing for users.
> >  - visor (which is deprecated)
> >  - web-console (to be honest, I don't quite understand the status of this
> > tool)
> >  - new ignite-profiling (which is a monitoring tool as well, judging by the
> > provided link [1] )
> >
> > I am not against the new tool, I just want to understand the motivation to
> > not improve the existing sub-projects.
> >
> > [1]
> >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >
> > Thanks,
> > S.
> >
> > пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]>:
> >
> > > Hi, Igniters.
> > >
> > > I'm working on cluster profiling and the tool for creating a
> > > performance report. [1] I have prepared PoC based on performance
> > > logging to a separate category of Ignite log. The report contains:
> > >
> > > - Cache operations and its distribution by types [2]
> > > - Transactions and histogram of durations [3]
> > > - SQL and Scan query statistics, top of slowest queries, logical and
> > > physical reads by query [4]
> > > - Compute statistics, top of slowest tasks and their jobs [5]
> > > Soon I will add:
> > > - Topology and Ignite versions info
> > > - Client ID in case of operations from clients
> > >
> > > For now, I'm developing a binary logging format to reduce the effect
> > > on performance. I'll try to reuse Ignite mechanisms.
> > >
> > > I would like to hear suggestions for the profiling and the performance
> > > report.
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > [2]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > [3]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > [4]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > [5]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > >
> > > --
> > > Best wishes,
> > > Amelchev Nikita
> > >
> >

--
Best wishes,
Amelchev Nikita

[Nikita Amelchev]

Slava,

The web-console is used to interactive configuration wizard,
management and monitoring tool [1].

I want to implement a profiling tool. It's one of the monitoring parts
and helps to answer the question: What will be running slow? See
IEP-35 [2].

[1] https://apacheignite-tools.readme.io/docs/ignite-web-console
[2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392

пт, 24 апр. 2020 г. в 18:46, Nikita Amelchev <[hidden email]>:

>
> Maxim,
>
> > Can you provide more information about the binary logging format?
> I started to investigate. It should be with minimal effect on
> performance. I haven't any details for now.
>
> > What the additional log category are you talking about?
> PoC is based on logging performance statistics to the "profiling"
> category to split with other logging and be able to log to a separate
> file. See IgniteLogger#getLogger.
>
> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <[hidden email]>:
> >
> > Hello!
> >
> > You need to throw in control.sh also, which does some kind of statistics
> > too, such as idle_verify.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <[hidden email]>:
> >
> > > Hello Nikita,
> > >
> > > Perhaps, I am missing something...
> > > Apache Ignite already has a web-console tool. Do we want to improve the
> > > existing tool instead of creating a new one?
> > > It seems to me, this can be confusing for users.
> > >  - visor (which is deprecated)
> > >  - web-console (to be honest, I don't quite understand the status of this
> > > tool)
> > >  - new ignite-profiling (which is a monitoring tool as well, judging by the
> > > provided link [1] )
> > >
> > > I am not against the new tool, I just want to understand the motivation to
> > > not improve the existing sub-projects.
> > >
> > > [1]
> > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >
> > > Thanks,
> > > S.
> > >
> > > пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]>:
> > >
> > > > Hi, Igniters.
> > > >
> > > > I'm working on cluster profiling and the tool for creating a
> > > > performance report. [1] I have prepared PoC based on performance
> > > > logging to a separate category of Ignite log. The report contains:
> > > >
> > > > - Cache operations and its distribution by types [2]
> > > > - Transactions and histogram of durations [3]
> > > > - SQL and Scan query statistics, top of slowest queries, logical and
> > > > physical reads by query [4]
> > > > - Compute statistics, top of slowest tasks and their jobs [5]
> > > > Soon I will add:
> > > > - Topology and Ignite versions info
> > > > - Client ID in case of operations from clients
> > > >
> > > > For now, I'm developing a binary logging format to reduce the effect
> > > > on performance. I'll try to reuse Ignite mechanisms.
> > > >
> > > > I would like to hear suggestions for the profiling and the performance
> > > > report.
> > > >
> > > > [1]
> > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > > [2]
> > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > > [3]
> > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > > [4]
> > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > > [5]
> > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > > >
> > > > --
> > > > Best wishes,
> > > > Amelchev Nikita
> > > >
> > >
>
>
>
> --
> Best wishes,
> Amelchev Nikita

--
Best wishes,
Amelchev Nikita

[Nikita Amelchev]

Ilya,

You suggest using control.sh to build the report?

пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <[hidden email]>:

>
> Hello!
>
> You need to throw in control.sh also, which does some kind of statistics
> too, such as idle_verify.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <[hidden email]>:
>
> > Hello Nikita,
> >
> > Perhaps, I am missing something...
> > Apache Ignite already has a web-console tool. Do we want to improve the
> > existing tool instead of creating a new one?
> > It seems to me, this can be confusing for users.
> >  - visor (which is deprecated)
> >  - web-console (to be honest, I don't quite understand the status of this
> > tool)
> >  - new ignite-profiling (which is a monitoring tool as well, judging by the
> > provided link [1] )
> >
> > I am not against the new tool, I just want to understand the motivation to
> > not improve the existing sub-projects.
> >
> > [1]
> >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >
> > Thanks,
> > S.
> >
> > пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]>:
> >
> > > Hi, Igniters.
> > >
> > > I'm working on cluster profiling and the tool for creating a
> > > performance report. [1] I have prepared PoC based on performance
> > > logging to a separate category of Ignite log. The report contains:
> > >
> > > - Cache operations and its distribution by types [2]
> > > - Transactions and histogram of durations [3]
> > > - SQL and Scan query statistics, top of slowest queries, logical and
> > > physical reads by query [4]
> > > - Compute statistics, top of slowest tasks and their jobs [5]
> > > Soon I will add:
> > > - Topology and Ignite versions info
> > > - Client ID in case of operations from clients
> > >
> > > For now, I'm developing a binary logging format to reduce the effect
> > > on performance. I'll try to reuse Ignite mechanisms.
> > >
> > > I would like to hear suggestions for the profiling and the performance
> > > report.
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > [2]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > [3]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > [4]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > [5]
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > >
> > > --
> > > Best wishes,
> > > Amelchev Nikita
> > >
> >

--
Best wishes,
Amelchev Nikita

[dmagda]

Nikita,

Thanks for preparing the IEP and breaking the initiative down into pieces.
I studied the IEP and it feels like the goal is to built a tool similar to
Oracle AWR. If it's so then shouldn't we add this as an extra function to
either visorcmd or control.sh. Hopefully, we can start moving in the
direction of the tooling unification.

Speaking of the probes/statistics collection approach, is it supposed to
reuse tracing capabilities that are to be added as part of IEP-35? [1].
According to the draft of our roadmap [2], @Ivan Rakov <[hidden email]> is
planning to contribute to the feature this year.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
[2] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Roadmap

-
Denis


On Fri, Apr 24, 2020 at 8:48 AM Nikita Amelchev <[hidden email]>
wrote:

> Slava,
>
> The web-console is used to interactive configuration wizard,
> management and monitoring tool [1].
>
> I want to implement a profiling tool. It's one of the monitoring parts
> and helps to answer the question: What will be running slow? See
> IEP-35 [2].
>
> [1] https://apacheignite-tools.readme.io/docs/ignite-web-console
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
>
> пт, 24 апр. 2020 г. в 18:46, Nikita Amelchev <[hidden email]>:
> >
> > Maxim,
> >
> > > Can you provide more information about the binary logging format?
> > I started to investigate. It should be with minimal effect on
> > performance. I haven't any details for now.
> >
> > > What the additional log category are you talking about?
> > PoC is based on logging performance statistics to the "profiling"
> > category to split with other logging and be able to log to a separate
> > file. See IgniteLogger#getLogger.
> >
> > пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <[hidden email]
> >:
> > >
> > > Hello!
> > >
> > > You need to throw in control.sh also, which does some kind of
> statistics
> > > too, such as idle_verify.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> [hidden email]>:
> > >
> > > > Hello Nikita,
> > > >
> > > > Perhaps, I am missing something...
> > > > Apache Ignite already has a web-console tool. Do we want to improve
> the
> > > > existing tool instead of creating a new one?
> > > > It seems to me, this can be confusing for users.
> > > >  - visor (which is deprecated)
> > > >  - web-console (to be honest, I don't quite understand the status of
> this
> > > > tool)
> > > >  - new ignite-profiling (which is a monitoring tool as well, judging
> by the
> > > > provided link [1] )
> > > >
> > > > I am not against the new tool, I just want to understand the
> motivation to
> > > > not improve the existing sub-projects.
> > > >
> > > > [1]
> > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >
> > > > Thanks,
> > > > S.
> > > >
> > > > пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]>:
> > > >
> > > > > Hi, Igniters.
> > > > >
> > > > > I'm working on cluster profiling and the tool for creating a
> > > > > performance report. [1] I have prepared PoC based on performance
> > > > > logging to a separate category of Ignite log. The report contains:
> > > > >
> > > > > - Cache operations and its distribution by types [2]
> > > > > - Transactions and histogram of durations [3]
> > > > > - SQL and Scan query statistics, top of slowest queries, logical
> and
> > > > > physical reads by query [4]
> > > > > - Compute statistics, top of slowest tasks and their jobs [5]
> > > > > Soon I will add:
> > > > > - Topology and Ignite versions info
> > > > > - Client ID in case of operations from clients
> > > > >
> > > > > For now, I'm developing a binary logging format to reduce the
> effect
> > > > > on performance. I'll try to reuse Ignite mechanisms.
> > > > >
> > > > > I would like to hear suggestions for the profiling and the
> performance
> > > > > report.
> > > > >
> > > > > [1]
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > > > [2]
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > > > [3]
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > > > [4]
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > > > [5]
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > > > >
> > > > > --
> > > > > Best wishes,
> > > > > Amelchev Nikita
> > > > >
> > > >
> >
> >
> >
> > --
> > Best wishes,
> > Amelchev Nikita
>
>
>
> --
> Best wishes,
> Amelchev Nikita
>

[Ilya Kasnacheev]

Hello!

I suggest that it's one of the places where it could be put instead of
adding a new tool.

Regards,
--
Ilya Kasnacheev


пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]>:

> Ilya,
>
> You suggest using control.sh to build the report?
>
> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <[hidden email]>:
> >
> > Hello!
> >
> > You need to throw in control.sh also, which does some kind of statistics
> > too, such as idle_verify.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <[hidden email]
> >:
> >
> > > Hello Nikita,
> > >
> > > Perhaps, I am missing something...
> > > Apache Ignite already has a web-console tool. Do we want to improve the
> > > existing tool instead of creating a new one?
> > > It seems to me, this can be confusing for users.
> > >  - visor (which is deprecated)
> > >  - web-console (to be honest, I don't quite understand the status of
> this
> > > tool)
> > >  - new ignite-profiling (which is a monitoring tool as well, judging
> by the
> > > provided link [1] )
> > >
> > > I am not against the new tool, I just want to understand the
> motivation to
> > > not improve the existing sub-projects.
> > >
> > > [1]
> > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >
> > > Thanks,
> > > S.
> > >
> > > пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]>:
> > >
> > > > Hi, Igniters.
> > > >
> > > > I'm working on cluster profiling and the tool for creating a
> > > > performance report. [1] I have prepared PoC based on performance
> > > > logging to a separate category of Ignite log. The report contains:
> > > >
> > > > - Cache operations and its distribution by types [2]
> > > > - Transactions and histogram of durations [3]
> > > > - SQL and Scan query statistics, top of slowest queries, logical and
> > > > physical reads by query [4]
> > > > - Compute statistics, top of slowest tasks and their jobs [5]
> > > > Soon I will add:
> > > > - Topology and Ignite versions info
> > > > - Client ID in case of operations from clients
> > > >
> > > > For now, I'm developing a binary logging format to reduce the effect
> > > > on performance. I'll try to reuse Ignite mechanisms.
> > > >
> > > > I would like to hear suggestions for the profiling and the
> performance
> > > > report.
> > > >
> > > > [1]
> > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > > [2]
> > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > > [3]
> > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > > [4]
> > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > > [5]
> > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > > >
> > > > --
> > > > Best wishes,
> > > > Amelchev Nikita
> > > >
> > >
>
>
>
> --
> Best wishes,
> Amelchev Nikita
>

[Nikita Amelchev]

Denis, Ilya,

I will try to integrate profiling functionality into control.sh utility.

> Speaking of the probes/statistics collection approach, is it supposed to
> reuse tracing capabilities that are to be added as part of IEP-35?
Tracing is more deeply takes statistics. If it will be possible, I'm for reuse.

пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <[hidden email]>:

>
> Hello!
>
> I suggest that it's one of the places where it could be put instead of
> adding a new tool.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]>:
>
> > Ilya,
> >
> > You suggest using control.sh to build the report?
> >
> > пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <[hidden email]>:
> > >
> > > Hello!
> > >
> > > You need to throw in control.sh also, which does some kind of statistics
> > > too, such as idle_verify.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <[hidden email]
> > >:
> > >
> > > > Hello Nikita,
> > > >
> > > > Perhaps, I am missing something...
> > > > Apache Ignite already has a web-console tool. Do we want to improve the
> > > > existing tool instead of creating a new one?
> > > > It seems to me, this can be confusing for users.
> > > >  - visor (which is deprecated)
> > > >  - web-console (to be honest, I don't quite understand the status of
> > this
> > > > tool)
> > > >  - new ignite-profiling (which is a monitoring tool as well, judging
> > by the
> > > > provided link [1] )
> > > >
> > > > I am not against the new tool, I just want to understand the
> > motivation to
> > > > not improve the existing sub-projects.
> > > >
> > > > [1]
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >
> > > > Thanks,
> > > > S.
> > > >
> > > > пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]>:
> > > >
> > > > > Hi, Igniters.
> > > > >
> > > > > I'm working on cluster profiling and the tool for creating a
> > > > > performance report. [1] I have prepared PoC based on performance
> > > > > logging to a separate category of Ignite log. The report contains:
> > > > >
> > > > > - Cache operations and its distribution by types [2]
> > > > > - Transactions and histogram of durations [3]
> > > > > - SQL and Scan query statistics, top of slowest queries, logical and
> > > > > physical reads by query [4]
> > > > > - Compute statistics, top of slowest tasks and their jobs [5]
> > > > > Soon I will add:
> > > > > - Topology and Ignite versions info
> > > > > - Client ID in case of operations from clients
> > > > >
> > > > > For now, I'm developing a binary logging format to reduce the effect
> > > > > on performance. I'll try to reuse Ignite mechanisms.
> > > > >
> > > > > I would like to hear suggestions for the profiling and the
> > performance
> > > > > report.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > > > [2]
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > > > [3]
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > > > [4]
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > > > [5]
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > > > >
> > > > > --
> > > > > Best wishes,
> > > > > Amelchev Nikita
> > > > >
> > > >
> >
> >
> >
> > --
> > Best wishes,
> > Amelchev Nikita
> >

--
Best wishes,
Amelchev Nikita

[dmagda]

>
> Tracing is more deeply takes statistics. If it will be possible, I'm for
> reuse.


Looks like we need to sync up on these activities/initiatives to ensure we
don't do a duplicate job. If you think a separate discussion is necessary
let's kick it off.

-
Denis


On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <[hidden email]>
wrote:

> Denis, Ilya,
>
> I will try to integrate profiling functionality into control.sh utility.
>
> > Speaking of the probes/statistics collection approach, is it supposed to
> > reuse tracing capabilities that are to be added as part of IEP-35?
> Tracing is more deeply takes statistics. If it will be possible, I'm for
> reuse.
>
> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <[hidden email]>:
> >
> > Hello!
> >
> > I suggest that it's one of the places where it could be put instead of
> > adding a new tool.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]>:
> >
> > > Ilya,
> > >
> > > You suggest using control.sh to build the report?
> > >
> > > пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
> [hidden email]>:
> > > >
> > > > Hello!
> > > >
> > > > You need to throw in control.sh also, which does some kind of
> statistics
> > > > too, such as idle_verify.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> [hidden email]
> > > >:
> > > >
> > > > > Hello Nikita,
> > > > >
> > > > > Perhaps, I am missing something...
> > > > > Apache Ignite already has a web-console tool. Do we want to
> improve the
> > > > > existing tool instead of creating a new one?
> > > > > It seems to me, this can be confusing for users.
> > > > >  - visor (which is deprecated)
> > > > >  - web-console (to be honest, I don't quite understand the status
> of
> > > this
> > > > > tool)
> > > > >  - new ignite-profiling (which is a monitoring tool as well,
> judging
> > > by the
> > > > > provided link [1] )
> > > > >
> > > > > I am not against the new tool, I just want to understand the
> > > motivation to
> > > > > not improve the existing sub-projects.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > > >
> > > > > Thanks,
> > > > > S.
> > > > >
> > > > > пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]
> >:
> > > > >
> > > > > > Hi, Igniters.
> > > > > >
> > > > > > I'm working on cluster profiling and the tool for creating a
> > > > > > performance report. [1] I have prepared PoC based on performance
> > > > > > logging to a separate category of Ignite log. The report
> contains:
> > > > > >
> > > > > > - Cache operations and its distribution by types [2]
> > > > > > - Transactions and histogram of durations [3]
> > > > > > - SQL and Scan query statistics, top of slowest queries, logical
> and
> > > > > > physical reads by query [4]
> > > > > > - Compute statistics, top of slowest tasks and their jobs [5]
> > > > > > Soon I will add:
> > > > > > - Topology and Ignite versions info
> > > > > > - Client ID in case of operations from clients
> > > > > >
> > > > > > For now, I'm developing a binary logging format to reduce the
> effect
> > > > > > on performance. I'll try to reuse Ignite mechanisms.
> > > > > >
> > > > > > I would like to hear suggestions for the profiling and the
> > > performance
> > > > > > report.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > > > > [2]
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > > > > [3]
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > > > > [4]
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > > > > [5]
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > > > > >
> > > > > > --
> > > > > > Best wishes,
> > > > > > Amelchev Nikita
> > > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > > Best wishes,
> > > Amelchev Nikita
> > >
>
>
>
> --
> Best wishes,
> Amelchev Nikita
>

[Nikolay Izhikov-2]

Hello, Slava, Ilya, Denis.

Thanks for joining this discussion!

> - visor (which is deprecated)

Who deprecated visor and when?
Maybe I miss something?

> - web-console (to be honest, I don't quite understand the status of this tool)

+1.

> I am not against the new tool, I just want to understand the motivation to not improve the existing sub-projects.

It's true that, for now, Ignite doesn't have "tool strategy"
I think it's a big issue from the user's point of view.
We should solve it in the nearest time.
Feel free to start this activity.

> - new ignite-profiling (which is a monitoring tool as well, judging by the provided link [1] )

The general idea is the following:

1. We should have some profiling mechanism that will generate a node-local event log
2. We should have a tool that can export events to some third-party system. This system can be an Elastic Search(Kibana) or Ignite performance report or Kafka log, whatever.
3. Ignite performance report, in the first release, should be a "static" tool.
    This means we take static logs(that is not rewritten in the analysis time) and feed them in the script(or tool or control.sh, whatever)
    The script produces static report that can be used for overall performance analysis.

The primary users of this report is a developer of Ignite based applications and performance engineers.

Ilya,

> You need to throw in control.sh also, which does some kind of statistics too, such as idle_verify.

Please, clarify your idea:
    We should use some info from control.sh to the report?
    The report should be generated by some control.sh subcommand?


Denis,

> Speaking of the probes/statistics collection approach, is it supposed to reuse tracing capabilities that are to be added as part of IEP-35?

For now, we don't have any results of tracing development available in Apache Ignite.
Hopefully, we got some in a couple of weeks.
After it, we can start a discussion of how to merge two improvements.

> 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]> написал(а):
>
>>
>> Tracing is more deeply takes statistics. If it will be possible, I'm for
>> reuse.
>
>
> Looks like we need to sync up on these activities/initiatives to ensure we
> don't do a duplicate job. If you think a separate discussion is necessary
> let's kick it off.
>
> -
> Denis
>
>
> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <[hidden email]>
> wrote:
>
>> Denis, Ilya,
>>
>> I will try to integrate profiling functionality into control.sh utility.
>>
>>> Speaking of the probes/statistics collection approach, is it supposed to
>>> reuse tracing capabilities that are to be added as part of IEP-35?
>> Tracing is more deeply takes statistics. If it will be possible, I'm for
>> reuse.
>>
>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <[hidden email]>:
>>>
>>> Hello!
>>>
>>> I suggest that it's one of the places where it could be put instead of
>>> adding a new tool.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]>:
>>>
>>>> Ilya,
>>>>
>>>> You suggest using control.sh to build the report?
>>>>
>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
>> [hidden email]>:
>>>>>
>>>>> Hello!
>>>>>
>>>>> You need to throw in control.sh also, which does some kind of
>> statistics
>>>>> too, such as idle_verify.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
>> [hidden email]
>>>>> :
>>>>>
>>>>>> Hello Nikita,
>>>>>>
>>>>>> Perhaps, I am missing something...
>>>>>> Apache Ignite already has a web-console tool. Do we want to
>> improve the
>>>>>> existing tool instead of creating a new one?
>>>>>> It seems to me, this can be confusing for users.
>>>>>> - visor (which is deprecated)
>>>>>> - web-console (to be honest, I don't quite understand the status
>> of
>>>> this
>>>>>> tool)
>>>>>> - new ignite-profiling (which is a monitoring tool as well,
>> judging
>>>> by the
>>>>>> provided link [1] )
>>>>>>
>>>>>> I am not against the new tool, I just want to understand the
>>>> motivation to
>>>>>> not improve the existing sub-projects.
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>>>>>>
>>>>>> Thanks,
>>>>>> S.
>>>>>>
>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]
>>> :
>>>>>>
>>>>>>> Hi, Igniters.
>>>>>>>
>>>>>>> I'm working on cluster profiling and the tool for creating a
>>>>>>> performance report. [1] I have prepared PoC based on performance
>>>>>>> logging to a separate category of Ignite log. The report
>> contains:
>>>>>>>
>>>>>>> - Cache operations and its distribution by types [2]
>>>>>>> - Transactions and histogram of durations [3]
>>>>>>> - SQL and Scan query statistics, top of slowest queries, logical
>> and
>>>>>>> physical reads by query [4]
>>>>>>> - Compute statistics, top of slowest tasks and their jobs [5]
>>>>>>> Soon I will add:
>>>>>>> - Topology and Ignite versions info
>>>>>>> - Client ID in case of operations from clients
>>>>>>>
>>>>>>> For now, I'm developing a binary logging format to reduce the
>> effect
>>>>>>> on performance. I'll try to reuse Ignite mechanisms.
>>>>>>>
>>>>>>> I would like to hear suggestions for the profiling and the
>>>> performance
>>>>>>> report.
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>>>>>>> [2]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
>>>>>>> [3]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
>>>>>>> [4]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
>>>>>>> [5]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
>>>>>>>
>>>>>>> --
>>>>>>> Best wishes,
>>>>>>> Amelchev Nikita
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best wishes,
>>>> Amelchev Nikita
>>>>
>>
>>
>>
>> --
>> Best wishes,
>> Amelchev Nikita
>>

[slava.koptilin]

Hello Nikolay,

> Who deprecated visor and when? Maybe I miss something?
On the one hand, there was technically no community consensus that this
tool should be obsolete.
On the other hand, my opinion based on the following topic:
http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
Moreover, it seems to me, currently, the control utility is widely used and
actively developed, instead of the visor.

> It's true that, for now, Ignite doesn't have "tool strategy" I think it's
a big issue from the user's point of view.
I absolutely agree with that.

> We should solve it in the nearest time. Feel free to start this activity
I have no plan at the moment. However, at the first stage, we could
understand the difference between visor and control utility.
All useful features from visor should be moved/implemented in control
utility and after that visor tool and should be marked as
deprecated/obsoleted.

> You need to throw in control.sh also, which does some kind of statistics
too, such as idle_verify.
> Please, clarify your idea:
>    We should use some info from control.sh to the report?
>    The report should be generated by some control.sh subcommand?
If I am not mistaken, the oracle database has AWR tool (mentioned on the
IEP page) which is a command-line utility that generates HTML reports.
I like this idea and I think this is a good approach that can be realized
in the control utility.
If we have a case that cannot be implemented in this way, we have to
clearly states the difference between these tools so as not to confuse our
users.
What do you think?

Thanks,
Slava.


сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <[hidden email]>:

> Hello, Slava, Ilya, Denis.
>
> Thanks for joining this discussion!
>
> > - visor (which is deprecated)
>
> Who deprecated visor and when?
> Maybe I miss something?
>
> > - web-console (to be honest, I don't quite understand the status of this
> tool)
>
> +1.
>
> > I am not against the new tool, I just want to understand the motivation
> to not improve the existing sub-projects.
>
> It's true that, for now, Ignite doesn't have "tool strategy"
> I think it's a big issue from the user's point of view.
> We should solve it in the nearest time.
> Feel free to start this activity.
>
> > - new ignite-profiling (which is a monitoring tool as well, judging by
> the provided link [1] )
>
> The general idea is the following:
>
> 1. We should have some profiling mechanism that will generate a node-local
> event log
> 2. We should have a tool that can export events to some third-party
> system. This system can be an Elastic Search(Kibana) or Ignite performance
> report or Kafka log, whatever.
> 3. Ignite performance report, in the first release, should be a "static"
> tool.
>     This means we take static logs(that is not rewritten in the analysis
> time) and feed them in the script(or tool or control.sh, whatever)
>     The script produces static report that can be used for overall
> performance analysis.
>
> The primary users of this report is a developer of Ignite based
> applications and performance engineers.
>
> Ilya,
>
> > You need to throw in control.sh also, which does some kind of statistics
> too, such as idle_verify.
>
> Please, clarify your idea:
>     We should use some info from control.sh to the report?
>     The report should be generated by some control.sh subcommand?
>
>
> Denis,
>
> > Speaking of the probes/statistics collection approach, is it supposed to
> reuse tracing capabilities that are to be added as part of IEP-35?
>
> For now, we don't have any results of tracing development available in
> Apache Ignite.
> Hopefully, we got some in a couple of weeks.
> After it, we can start a discussion of how to merge two improvements.
>
>
>
> > 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]> написал(а):
> >
> >>
> >> Tracing is more deeply takes statistics. If it will be possible, I'm for
> >> reuse.
> >
> >
> > Looks like we need to sync up on these activities/initiatives to ensure
> we
> > don't do a duplicate job. If you think a separate discussion is necessary
> > let's kick it off.
> >
> > -
> > Denis
> >
> >
> > On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <[hidden email]>
> > wrote:
> >
> >> Denis, Ilya,
> >>
> >> I will try to integrate profiling functionality into control.sh utility.
> >>
> >>> Speaking of the probes/statistics collection approach, is it supposed
> to
> >>> reuse tracing capabilities that are to be added as part of IEP-35?
> >> Tracing is more deeply takes statistics. If it will be possible, I'm for
> >> reuse.
> >>
> >> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <[hidden email]
> >:
> >>>
> >>> Hello!
> >>>
> >>> I suggest that it's one of the places where it could be put instead of
> >>> adding a new tool.
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]>:
> >>>
> >>>> Ilya,
> >>>>
> >>>> You suggest using control.sh to build the report?
> >>>>
> >>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
> >> [hidden email]>:
> >>>>>
> >>>>> Hello!
> >>>>>
> >>>>> You need to throw in control.sh also, which does some kind of
> >> statistics
> >>>>> too, such as idle_verify.
> >>>>>
> >>>>> Regards,
> >>>>> --
> >>>>> Ilya Kasnacheev
> >>>>>
> >>>>>
> >>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> >> [hidden email]
> >>>>> :
> >>>>>
> >>>>>> Hello Nikita,
> >>>>>>
> >>>>>> Perhaps, I am missing something...
> >>>>>> Apache Ignite already has a web-console tool. Do we want to
> >> improve the
> >>>>>> existing tool instead of creating a new one?
> >>>>>> It seems to me, this can be confusing for users.
> >>>>>> - visor (which is deprecated)
> >>>>>> - web-console (to be honest, I don't quite understand the status
> >> of
> >>>> this
> >>>>>> tool)
> >>>>>> - new ignite-profiling (which is a monitoring tool as well,
> >> judging
> >>>> by the
> >>>>>> provided link [1] )
> >>>>>>
> >>>>>> I am not against the new tool, I just want to understand the
> >>>> motivation to
> >>>>>> not improve the existing sub-projects.
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>>>>>
> >>>>>> Thanks,
> >>>>>> S.
> >>>>>>
> >>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]
> >>> :
> >>>>>>
> >>>>>>> Hi, Igniters.
> >>>>>>>
> >>>>>>> I'm working on cluster profiling and the tool for creating a
> >>>>>>> performance report. [1] I have prepared PoC based on performance
> >>>>>>> logging to a separate category of Ignite log. The report
> >> contains:
> >>>>>>>
> >>>>>>> - Cache operations and its distribution by types [2]
> >>>>>>> - Transactions and histogram of durations [3]
> >>>>>>> - SQL and Scan query statistics, top of slowest queries, logical
> >> and
> >>>>>>> physical reads by query [4]
> >>>>>>> - Compute statistics, top of slowest tasks and their jobs [5]
> >>>>>>> Soon I will add:
> >>>>>>> - Topology and Ignite versions info
> >>>>>>> - Client ID in case of operations from clients
> >>>>>>>
> >>>>>>> For now, I'm developing a binary logging format to reduce the
> >> effect
> >>>>>>> on performance. I'll try to reuse Ignite mechanisms.
> >>>>>>>
> >>>>>>> I would like to hear suggestions for the profiling and the
> >>>> performance
> >>>>>>> report.
> >>>>>>>
> >>>>>>> [1]
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>>>>>> [2]
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> >>>>>>> [3]
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> >>>>>>> [4]
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> >>>>>>> [5]
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> >>>>>>>
> >>>>>>> --
> >>>>>>> Best wishes,
> >>>>>>> Amelchev Nikita
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best wishes,
> >>>> Amelchev Nikita
> >>>>
> >>
> >>
> >>
> >> --
> >> Best wishes,
> >> Amelchev Nikita
> >>
>
>

[Nikita Amelchev]

Hi, Igniters.

I have implemented cluster profiling and tool to build the performance
report. It's ready to be reviewed. [1, 2]

Profiling can be managed by JMX bean. I have plans to implement it to
control.sh also.

Nodes write statistics to the temporary off heap buffer and then one
thread flushes to the profiling files. The write mechanics and format
is like WAL logging.
The report contains the following statistics:
- nodes and caches info
- cache operations and transaction statistics
- SQL and scan queries statistics (include logical and physical reads per query)
- tasks and jobs statistics.

More details in the IEP [3].

[1] https://github.com/apache/ignite/pull/7693
[2] https://issues.apache.org/jira/browse/IGNITE-12666
[3] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool

вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <[hidden email]>:

>
> Hello Nikolay,
>
> > Who deprecated visor and when? Maybe I miss something?
> On the one hand, there was technically no community consensus that this
> tool should be obsolete.
> On the other hand, my opinion based on the following topic:
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> Moreover, it seems to me, currently, the control utility is widely used and
> actively developed, instead of the visor.
>
> > It's true that, for now, Ignite doesn't have "tool strategy" I think it's
> a big issue from the user's point of view.
> I absolutely agree with that.
>
> > We should solve it in the nearest time. Feel free to start this activity
> I have no plan at the moment. However, at the first stage, we could
> understand the difference between visor and control utility.
> All useful features from visor should be moved/implemented in control
> utility and after that visor tool and should be marked as
> deprecated/obsoleted.
>
> > You need to throw in control.sh also, which does some kind of statistics
> too, such as idle_verify.
> > Please, clarify your idea:
> >    We should use some info from control.sh to the report?
> >    The report should be generated by some control.sh subcommand?
> If I am not mistaken, the oracle database has AWR tool (mentioned on the
> IEP page) which is a command-line utility that generates HTML reports.
> I like this idea and I think this is a good approach that can be realized
> in the control utility.
> If we have a case that cannot be implemented in this way, we have to
> clearly states the difference between these tools so as not to confuse our
> users.
> What do you think?
>
> Thanks,
> Slava.
>
>
> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <[hidden email]>:
>
> > Hello, Slava, Ilya, Denis.
> >
> > Thanks for joining this discussion!
> >
> > > - visor (which is deprecated)
> >
> > Who deprecated visor and when?
> > Maybe I miss something?
> >
> > > - web-console (to be honest, I don't quite understand the status of this
> > tool)
> >
> > +1.
> >
> > > I am not against the new tool, I just want to understand the motivation
> > to not improve the existing sub-projects.
> >
> > It's true that, for now, Ignite doesn't have "tool strategy"
> > I think it's a big issue from the user's point of view.
> > We should solve it in the nearest time.
> > Feel free to start this activity.
> >
> > > - new ignite-profiling (which is a monitoring tool as well, judging by
> > the provided link [1] )
> >
> > The general idea is the following:
> >
> > 1. We should have some profiling mechanism that will generate a node-local
> > event log
> > 2. We should have a tool that can export events to some third-party
> > system. This system can be an Elastic Search(Kibana) or Ignite performance
> > report or Kafka log, whatever.
> > 3. Ignite performance report, in the first release, should be a "static"
> > tool.
> >     This means we take static logs(that is not rewritten in the analysis
> > time) and feed them in the script(or tool or control.sh, whatever)
> >     The script produces static report that can be used for overall
> > performance analysis.
> >
> > The primary users of this report is a developer of Ignite based
> > applications and performance engineers.
> >
> > Ilya,
> >
> > > You need to throw in control.sh also, which does some kind of statistics
> > too, such as idle_verify.
> >
> > Please, clarify your idea:
> >     We should use some info from control.sh to the report?
> >     The report should be generated by some control.sh subcommand?
> >
> >
> > Denis,
> >
> > > Speaking of the probes/statistics collection approach, is it supposed to
> > reuse tracing capabilities that are to be added as part of IEP-35?
> >
> > For now, we don't have any results of tracing development available in
> > Apache Ignite.
> > Hopefully, we got some in a couple of weeks.
> > After it, we can start a discussion of how to merge two improvements.
> >
> >
> >
> > > 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]> написал(а):
> > >
> > >>
> > >> Tracing is more deeply takes statistics. If it will be possible, I'm for
> > >> reuse.
> > >
> > >
> > > Looks like we need to sync up on these activities/initiatives to ensure
> > we
> > > don't do a duplicate job. If you think a separate discussion is necessary
> > > let's kick it off.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <[hidden email]>
> > > wrote:
> > >
> > >> Denis, Ilya,
> > >>
> > >> I will try to integrate profiling functionality into control.sh utility.
> > >>
> > >>> Speaking of the probes/statistics collection approach, is it supposed
> > to
> > >>> reuse tracing capabilities that are to be added as part of IEP-35?
> > >> Tracing is more deeply takes statistics. If it will be possible, I'm for
> > >> reuse.
> > >>
> > >> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <[hidden email]
> > >:
> > >>>
> > >>> Hello!
> > >>>
> > >>> I suggest that it's one of the places where it could be put instead of
> > >>> adding a new tool.
> > >>>
> > >>> Regards,
> > >>> --
> > >>> Ilya Kasnacheev
> > >>>
> > >>>
> > >>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]>:
> > >>>
> > >>>> Ilya,
> > >>>>
> > >>>> You suggest using control.sh to build the report?
> > >>>>
> > >>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
> > >> [hidden email]>:
> > >>>>>
> > >>>>> Hello!
> > >>>>>
> > >>>>> You need to throw in control.sh also, which does some kind of
> > >> statistics
> > >>>>> too, such as idle_verify.
> > >>>>>
> > >>>>> Regards,
> > >>>>> --
> > >>>>> Ilya Kasnacheev
> > >>>>>
> > >>>>>
> > >>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> > >> [hidden email]
> > >>>>> :
> > >>>>>
> > >>>>>> Hello Nikita,
> > >>>>>>
> > >>>>>> Perhaps, I am missing something...
> > >>>>>> Apache Ignite already has a web-console tool. Do we want to
> > >> improve the
> > >>>>>> existing tool instead of creating a new one?
> > >>>>>> It seems to me, this can be confusing for users.
> > >>>>>> - visor (which is deprecated)
> > >>>>>> - web-console (to be honest, I don't quite understand the status
> > >> of
> > >>>> this
> > >>>>>> tool)
> > >>>>>> - new ignite-profiling (which is a monitoring tool as well,
> > >> judging
> > >>>> by the
> > >>>>>> provided link [1] )
> > >>>>>>
> > >>>>>> I am not against the new tool, I just want to understand the
> > >>>> motivation to
> > >>>>>> not improve the existing sub-projects.
> > >>>>>>
> > >>>>>> [1]
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> S.
> > >>>>>>
> > >>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <[hidden email]
> > >>> :
> > >>>>>>
> > >>>>>>> Hi, Igniters.
> > >>>>>>>
> > >>>>>>> I'm working on cluster profiling and the tool for creating a
> > >>>>>>> performance report. [1] I have prepared PoC based on performance
> > >>>>>>> logging to a separate category of Ignite log. The report
> > >> contains:
> > >>>>>>>
> > >>>>>>> - Cache operations and its distribution by types [2]
> > >>>>>>> - Transactions and histogram of durations [3]
> > >>>>>>> - SQL and Scan query statistics, top of slowest queries, logical
> > >> and
> > >>>>>>> physical reads by query [4]
> > >>>>>>> - Compute statistics, top of slowest tasks and their jobs [5]
> > >>>>>>> Soon I will add:
> > >>>>>>> - Topology and Ignite versions info
> > >>>>>>> - Client ID in case of operations from clients
> > >>>>>>>
> > >>>>>>> For now, I'm developing a binary logging format to reduce the
> > >> effect
> > >>>>>>> on performance. I'll try to reuse Ignite mechanisms.
> > >>>>>>>
> > >>>>>>> I would like to hear suggestions for the profiling and the
> > >>>> performance
> > >>>>>>> report.
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>>>>>> [2]
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > >>>>>>> [3]
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > >>>>>>> [4]
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > >>>>>>> [5]
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Best wishes,
> > >>>>>>> Amelchev Nikita
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Best wishes,
> > >>>> Amelchev Nikita
> > >>>>
> > >>
> > >>
> > >>
> > >> --
> > >> Best wishes,
> > >> Amelchev Nikita
> > >>
> >
> >

--
Best wishes,
Amelchev Nikita

[Alexey Goncharuk]

Nikita, Igniters,

I left a few comments on the tool itself in the PR.

However, I would like to reiterate and discuss why a user would prefer to
use the profiling tool over tracing? Profiling tool only captures very
high-level details of the operations (a single cache operation, for
example), and does not interconnect operations happened on different nodes.
For example, a user would not be able to match a long task with a long job
in that task. In other words, profiling data is always a subset of data
collected by tracing.

Maybe it makes sense to adopt local log file approach to write spans so we
can process those spans later to build a report?

чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <[hidden email]>:

> Hi, Igniters.
>
> I have implemented cluster profiling and tool to build the performance
> report. It's ready to be reviewed. [1, 2]
>
> Profiling can be managed by JMX bean. I have plans to implement it to
> control.sh also.
>
> Nodes write statistics to the temporary off heap buffer and then one
> thread flushes to the profiling files. The write mechanics and format
> is like WAL logging.
> The report contains the following statistics:
> - nodes and caches info
> - cache operations and transaction statistics
> - SQL and scan queries statistics (include logical and physical reads per
> query)
> - tasks and jobs statistics.
>
> More details in the IEP [3].
>
> [1] https://github.com/apache/ignite/pull/7693
> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> [3]
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>
> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <[hidden email]>:
> >
> > Hello Nikolay,
> >
> > > Who deprecated visor and when? Maybe I miss something?
> > On the one hand, there was technically no community consensus that this
> > tool should be obsolete.
> > On the other hand, my opinion based on the following topic:
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> > Moreover, it seems to me, currently, the control utility is widely used
> and
> > actively developed, instead of the visor.
> >
> > > It's true that, for now, Ignite doesn't have "tool strategy" I think
> it's
> > a big issue from the user's point of view.
> > I absolutely agree with that.
> >
> > > We should solve it in the nearest time. Feel free to start this
> activity
> > I have no plan at the moment. However, at the first stage, we could
> > understand the difference between visor and control utility.
> > All useful features from visor should be moved/implemented in control
> > utility and after that visor tool and should be marked as
> > deprecated/obsoleted.
> >
> > > You need to throw in control.sh also, which does some kind of
> statistics
> > too, such as idle_verify.
> > > Please, clarify your idea:
> > >    We should use some info from control.sh to the report?
> > >    The report should be generated by some control.sh subcommand?
> > If I am not mistaken, the oracle database has AWR tool (mentioned on the
> > IEP page) which is a command-line utility that generates HTML reports.
> > I like this idea and I think this is a good approach that can be realized
> > in the control utility.
> > If we have a case that cannot be implemented in this way, we have to
> > clearly states the difference between these tools so as not to confuse
> our
> > users.
> > What do you think?
> >
> > Thanks,
> > Slava.
> >
> >
> > сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <[hidden email]>:
> >
> > > Hello, Slava, Ilya, Denis.
> > >
> > > Thanks for joining this discussion!
> > >
> > > > - visor (which is deprecated)
> > >
> > > Who deprecated visor and when?
> > > Maybe I miss something?
> > >
> > > > - web-console (to be honest, I don't quite understand the status of
> this
> > > tool)
> > >
> > > +1.
> > >
> > > > I am not against the new tool, I just want to understand the
> motivation
> > > to not improve the existing sub-projects.
> > >
> > > It's true that, for now, Ignite doesn't have "tool strategy"
> > > I think it's a big issue from the user's point of view.
> > > We should solve it in the nearest time.
> > > Feel free to start this activity.
> > >
> > > > - new ignite-profiling (which is a monitoring tool as well, judging
> by
> > > the provided link [1] )
> > >
> > > The general idea is the following:
> > >
> > > 1. We should have some profiling mechanism that will generate a
> node-local
> > > event log
> > > 2. We should have a tool that can export events to some third-party
> > > system. This system can be an Elastic Search(Kibana) or Ignite
> performance
> > > report or Kafka log, whatever.
> > > 3. Ignite performance report, in the first release, should be a
> "static"
> > > tool.
> > >     This means we take static logs(that is not rewritten in the
> analysis
> > > time) and feed them in the script(or tool or control.sh, whatever)
> > >     The script produces static report that can be used for overall
> > > performance analysis.
> > >
> > > The primary users of this report is a developer of Ignite based
> > > applications and performance engineers.
> > >
> > > Ilya,
> > >
> > > > You need to throw in control.sh also, which does some kind of
> statistics
> > > too, such as idle_verify.
> > >
> > > Please, clarify your idea:
> > >     We should use some info from control.sh to the report?
> > >     The report should be generated by some control.sh subcommand?
> > >
> > >
> > > Denis,
> > >
> > > > Speaking of the probes/statistics collection approach, is it
> supposed to
> > > reuse tracing capabilities that are to be added as part of IEP-35?
> > >
> > > For now, we don't have any results of tracing development available in
> > > Apache Ignite.
> > > Hopefully, we got some in a couple of weeks.
> > > After it, we can start a discussion of how to merge two improvements.
> > >
> > >
> > >
> > > > 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]>
> написал(а):
> > > >
> > > >>
> > > >> Tracing is more deeply takes statistics. If it will be possible,
> I'm for
> > > >> reuse.
> > > >
> > > >
> > > > Looks like we need to sync up on these activities/initiatives to
> ensure
> > > we
> > > > don't do a duplicate job. If you think a separate discussion is
> necessary
> > > > let's kick it off.
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
> [hidden email]>
> > > > wrote:
> > > >
> > > >> Denis, Ilya,
> > > >>
> > > >> I will try to integrate profiling functionality into control.sh
> utility.
> > > >>
> > > >>> Speaking of the probes/statistics collection approach, is it
> supposed
> > > to
> > > >>> reuse tracing capabilities that are to be added as part of IEP-35?
> > > >> Tracing is more deeply takes statistics. If it will be possible,
> I'm for
> > > >> reuse.
> > > >>
> > > >> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
> [hidden email]
> > > >:
> > > >>>
> > > >>> Hello!
> > > >>>
> > > >>> I suggest that it's one of the places where it could be put
> instead of
> > > >>> adding a new tool.
> > > >>>
> > > >>> Regards,
> > > >>> --
> > > >>> Ilya Kasnacheev
> > > >>>
> > > >>>
> > > >>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]
> >:
> > > >>>
> > > >>>> Ilya,
> > > >>>>
> > > >>>> You suggest using control.sh to build the report?
> > > >>>>
> > > >>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
> > > >> [hidden email]>:
> > > >>>>>
> > > >>>>> Hello!
> > > >>>>>
> > > >>>>> You need to throw in control.sh also, which does some kind of
> > > >> statistics
> > > >>>>> too, such as idle_verify.
> > > >>>>>
> > > >>>>> Regards,
> > > >>>>> --
> > > >>>>> Ilya Kasnacheev
> > > >>>>>
> > > >>>>>
> > > >>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> > > >> [hidden email]
> > > >>>>> :
> > > >>>>>
> > > >>>>>> Hello Nikita,
> > > >>>>>>
> > > >>>>>> Perhaps, I am missing something...
> > > >>>>>> Apache Ignite already has a web-console tool. Do we want to
> > > >> improve the
> > > >>>>>> existing tool instead of creating a new one?
> > > >>>>>> It seems to me, this can be confusing for users.
> > > >>>>>> - visor (which is deprecated)
> > > >>>>>> - web-console (to be honest, I don't quite understand the status
> > > >> of
> > > >>>> this
> > > >>>>>> tool)
> > > >>>>>> - new ignite-profiling (which is a monitoring tool as well,
> > > >> judging
> > > >>>> by the
> > > >>>>>> provided link [1] )
> > > >>>>>>
> > > >>>>>> I am not against the new tool, I just want to understand the
> > > >>>> motivation to
> > > >>>>>> not improve the existing sub-projects.
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> S.
> > > >>>>>>
> > > >>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <
> [hidden email]
> > > >>> :
> > > >>>>>>
> > > >>>>>>> Hi, Igniters.
> > > >>>>>>>
> > > >>>>>>> I'm working on cluster profiling and the tool for creating a
> > > >>>>>>> performance report. [1] I have prepared PoC based on
> performance
> > > >>>>>>> logging to a separate category of Ignite log. The report
> > > >> contains:
> > > >>>>>>>
> > > >>>>>>> - Cache operations and its distribution by types [2]
> > > >>>>>>> - Transactions and histogram of durations [3]
> > > >>>>>>> - SQL and Scan query statistics, top of slowest queries,
> logical
> > > >> and
> > > >>>>>>> physical reads by query [4]
> > > >>>>>>> - Compute statistics, top of slowest tasks and their jobs [5]
> > > >>>>>>> Soon I will add:
> > > >>>>>>> - Topology and Ignite versions info
> > > >>>>>>> - Client ID in case of operations from clients
> > > >>>>>>>
> > > >>>>>>> For now, I'm developing a binary logging format to reduce the
> > > >> effect
> > > >>>>>>> on performance. I'll try to reuse Ignite mechanisms.
> > > >>>>>>>
> > > >>>>>>> I would like to hear suggestions for the profiling and the
> > > >>>> performance
> > > >>>>>>> report.
> > > >>>>>>>
> > > >>>>>>> [1]
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >>>>>>> [2]
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > >>>>>>> [3]
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > >>>>>>> [4]
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > >>>>>>> [5]
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Best wishes,
> > > >>>>>>> Amelchev Nikita
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Best wishes,
> > > >>>> Amelchev Nikita
> > > >>>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best wishes,
> > > >> Amelchev Nikita
> > > >>
> > >
> > >
>
>
>
> --
> Best wishes,
> Amelchev Nikita
>

[Nikolay Izhikov-2]

Hello, Alexey.

Thanks for the review.

My understanding if the following:

We will have 3 in-depth tool to find issues in cluster:

1. Metrics + System views - data that describe Ignite entities very high-level.

2. Profiling - tool to know what specific query of transactions are slow.
In many cases, this knowledge is enough to fix the issue(rewrite query, redesign transactions flow, etc)

3. Tracing - tool to know why one of 1000 of the same queries was slow.
The most detailed view of the Ignite internal processes.

> For example, a user would not be able to match a long task with a long job in that task.

This is not true.
Profiling report will aggregate data from all nodes.
So there will be both

 * summary time of the task
 * time of the each job in the task.

> 8 июня 2020 г., в 12:52, Alexey Goncharuk <[hidden email]> написал(а):
>
> Nikita, Igniters,
>
> I left a few comments on the tool itself in the PR.
>
> However, I would like to reiterate and discuss why a user would prefer to
> use the profiling tool over tracing? Profiling tool only captures very
> high-level details of the operations (a single cache operation, for
> example), and does not interconnect operations happened on different nodes.
> For example, a user would not be able to match a long task with a long job
> in that task. In other words, profiling data is always a subset of data
> collected by tracing.
>
> Maybe it makes sense to adopt local log file approach to write spans so we
> can process those spans later to build a report?
>
> чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <[hidden email]>:
>
>> Hi, Igniters.
>>
>> I have implemented cluster profiling and tool to build the performance
>> report. It's ready to be reviewed. [1, 2]
>>
>> Profiling can be managed by JMX bean. I have plans to implement it to
>> control.sh also.
>>
>> Nodes write statistics to the temporary off heap buffer and then one
>> thread flushes to the profiling files. The write mechanics and format
>> is like WAL logging.
>> The report contains the following statistics:
>> - nodes and caches info
>> - cache operations and transaction statistics
>> - SQL and scan queries statistics (include logical and physical reads per
>> query)
>> - tasks and jobs statistics.
>>
>> More details in the IEP [3].
>>
>> [1] https://github.com/apache/ignite/pull/7693
>> [2] https://issues.apache.org/jira/browse/IGNITE-12666
>> [3]
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>>
>> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <[hidden email]>:
>>>
>>> Hello Nikolay,
>>>
>>>> Who deprecated visor and when? Maybe I miss something?
>>> On the one hand, there was technically no community consensus that this
>>> tool should be obsolete.
>>> On the other hand, my opinion based on the following topic:
>>>
>> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
>>> Moreover, it seems to me, currently, the control utility is widely used
>> and
>>> actively developed, instead of the visor.
>>>
>>>> It's true that, for now, Ignite doesn't have "tool strategy" I think
>> it's
>>> a big issue from the user's point of view.
>>> I absolutely agree with that.
>>>
>>>> We should solve it in the nearest time. Feel free to start this
>> activity
>>> I have no plan at the moment. However, at the first stage, we could
>>> understand the difference between visor and control utility.
>>> All useful features from visor should be moved/implemented in control
>>> utility and after that visor tool and should be marked as
>>> deprecated/obsoleted.
>>>
>>>> You need to throw in control.sh also, which does some kind of
>> statistics
>>> too, such as idle_verify.
>>>> Please, clarify your idea:
>>>>   We should use some info from control.sh to the report?
>>>>   The report should be generated by some control.sh subcommand?
>>> If I am not mistaken, the oracle database has AWR tool (mentioned on the
>>> IEP page) which is a command-line utility that generates HTML reports.
>>> I like this idea and I think this is a good approach that can be realized
>>> in the control utility.
>>> If we have a case that cannot be implemented in this way, we have to
>>> clearly states the difference between these tools so as not to confuse
>> our
>>> users.
>>> What do you think?
>>>
>>> Thanks,
>>> Slava.
>>>
>>>
>>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <[hidden email]>:
>>>
>>>> Hello, Slava, Ilya, Denis.
>>>>
>>>> Thanks for joining this discussion!
>>>>
>>>>> - visor (which is deprecated)
>>>>
>>>> Who deprecated visor and when?
>>>> Maybe I miss something?
>>>>
>>>>> - web-console (to be honest, I don't quite understand the status of
>> this
>>>> tool)
>>>>
>>>> +1.
>>>>
>>>>> I am not against the new tool, I just want to understand the
>> motivation
>>>> to not improve the existing sub-projects.
>>>>
>>>> It's true that, for now, Ignite doesn't have "tool strategy"
>>>> I think it's a big issue from the user's point of view.
>>>> We should solve it in the nearest time.
>>>> Feel free to start this activity.
>>>>
>>>>> - new ignite-profiling (which is a monitoring tool as well, judging
>> by
>>>> the provided link [1] )
>>>>
>>>> The general idea is the following:
>>>>
>>>> 1. We should have some profiling mechanism that will generate a
>> node-local
>>>> event log
>>>> 2. We should have a tool that can export events to some third-party
>>>> system. This system can be an Elastic Search(Kibana) or Ignite
>> performance
>>>> report or Kafka log, whatever.
>>>> 3. Ignite performance report, in the first release, should be a
>> "static"
>>>> tool.
>>>>    This means we take static logs(that is not rewritten in the
>> analysis
>>>> time) and feed them in the script(or tool or control.sh, whatever)
>>>>    The script produces static report that can be used for overall
>>>> performance analysis.
>>>>
>>>> The primary users of this report is a developer of Ignite based
>>>> applications and performance engineers.
>>>>
>>>> Ilya,
>>>>
>>>>> You need to throw in control.sh also, which does some kind of
>> statistics
>>>> too, such as idle_verify.
>>>>
>>>> Please, clarify your idea:
>>>>    We should use some info from control.sh to the report?
>>>>    The report should be generated by some control.sh subcommand?
>>>>
>>>>
>>>> Denis,
>>>>
>>>>> Speaking of the probes/statistics collection approach, is it
>> supposed to
>>>> reuse tracing capabilities that are to be added as part of IEP-35?
>>>>
>>>> For now, we don't have any results of tracing development available in
>>>> Apache Ignite.
>>>> Hopefully, we got some in a couple of weeks.
>>>> After it, we can start a discussion of how to merge two improvements.
>>>>
>>>>
>>>>
>>>>> 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]>
>> написал(а):
>>>>>
>>>>>>
>>>>>> Tracing is more deeply takes statistics. If it will be possible,
>> I'm for
>>>>>> reuse.
>>>>>
>>>>>
>>>>> Looks like we need to sync up on these activities/initiatives to
>> ensure
>>>> we
>>>>> don't do a duplicate job. If you think a separate discussion is
>> necessary
>>>>> let's kick it off.
>>>>>
>>>>> -
>>>>> Denis
>>>>>
>>>>>
>>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
>> [hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Denis, Ilya,
>>>>>>
>>>>>> I will try to integrate profiling functionality into control.sh
>> utility.
>>>>>>
>>>>>>> Speaking of the probes/statistics collection approach, is it
>> supposed
>>>> to
>>>>>>> reuse tracing capabilities that are to be added as part of IEP-35?
>>>>>> Tracing is more deeply takes statistics. If it will be possible,
>> I'm for
>>>>>> reuse.
>>>>>>
>>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
>> [hidden email]
>>>>> :
>>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> I suggest that it's one of the places where it could be put
>> instead of
>>>>>>> adding a new tool.
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>>
>>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]
>>> :
>>>>>>>
>>>>>>>> Ilya,
>>>>>>>>
>>>>>>>> You suggest using control.sh to build the report?
>>>>>>>>
>>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
>>>>>> [hidden email]>:
>>>>>>>>>
>>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> You need to throw in control.sh also, which does some kind of
>>>>>> statistics
>>>>>>>>> too, such as idle_verify.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> --
>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
>>>>>> [hidden email]
>>>>>>>>> :
>>>>>>>>>
>>>>>>>>>> Hello Nikita,
>>>>>>>>>>
>>>>>>>>>> Perhaps, I am missing something...
>>>>>>>>>> Apache Ignite already has a web-console tool. Do we want to
>>>>>> improve the
>>>>>>>>>> existing tool instead of creating a new one?
>>>>>>>>>> It seems to me, this can be confusing for users.
>>>>>>>>>> - visor (which is deprecated)
>>>>>>>>>> - web-console (to be honest, I don't quite understand the status
>>>>>> of
>>>>>>>> this
>>>>>>>>>> tool)
>>>>>>>>>> - new ignite-profiling (which is a monitoring tool as well,
>>>>>> judging
>>>>>>>> by the
>>>>>>>>>> provided link [1] )
>>>>>>>>>>
>>>>>>>>>> I am not against the new tool, I just want to understand the
>>>>>>>> motivation to
>>>>>>>>>> not improve the existing sub-projects.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> S.
>>>>>>>>>>
>>>>>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <
>> [hidden email]
>>>>>>> :
>>>>>>>>>>
>>>>>>>>>>> Hi, Igniters.
>>>>>>>>>>>
>>>>>>>>>>> I'm working on cluster profiling and the tool for creating a
>>>>>>>>>>> performance report. [1] I have prepared PoC based on
>> performance
>>>>>>>>>>> logging to a separate category of Ignite log. The report
>>>>>> contains:
>>>>>>>>>>>
>>>>>>>>>>> - Cache operations and its distribution by types [2]
>>>>>>>>>>> - Transactions and histogram of durations [3]
>>>>>>>>>>> - SQL and Scan query statistics, top of slowest queries,
>> logical
>>>>>> and
>>>>>>>>>>> physical reads by query [4]
>>>>>>>>>>> - Compute statistics, top of slowest tasks and their jobs [5]
>>>>>>>>>>> Soon I will add:
>>>>>>>>>>> - Topology and Ignite versions info
>>>>>>>>>>> - Client ID in case of operations from clients
>>>>>>>>>>>
>>>>>>>>>>> For now, I'm developing a binary logging format to reduce the
>>>>>> effect
>>>>>>>>>>> on performance. I'll try to reuse Ignite mechanisms.
>>>>>>>>>>>
>>>>>>>>>>> I would like to hear suggestions for the profiling and the
>>>>>>>> performance
>>>>>>>>>>> report.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>>>>>>>>>>> [2]
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
>>>>>>>>>>> [3]
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
>>>>>>>>>>> [4]
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
>>>>>>>>>>> [5]
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best wishes,
>>>>>>>>>>> Amelchev Nikita
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best wishes,
>>>>>>>> Amelchev Nikita
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best wishes,
>>>>>> Amelchev Nikita
>>>>>>
>>>>
>>>>
>>
>>
>>
>> --
>> Best wishes,
>> Amelchev Nikita
>>

[Saikat Maitra]

Hello Nikita,

I observed we have an open PR in ignite repo for this feature with
different set of changes compared to ignite extensions repo.


apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>

https://github.com/apache/ignite-extensions/pull/16

Can you please share more info on how we can use the profiling tool with
ignite-extensions modules?


Regards,

Saikat

On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <[hidden email]> wrote:

> Hello, Alexey.
>
> Thanks for the review.
>
> My understanding if the following:
>
> We will have 3 in-depth tool to find issues in cluster:
>
> 1. Metrics + System views - data that describe Ignite entities very
> high-level.
>
> 2. Profiling - tool to know what specific query of transactions are slow.
> In many cases, this knowledge is enough to fix the issue(rewrite query,
> redesign transactions flow, etc)
>
> 3. Tracing - tool to know why one of 1000 of the same queries was slow.
> The most detailed view of the Ignite internal processes.
>
> > For example, a user would not be able to match a long task with a long
> job in that task.
>
> This is not true.
> Profiling report will aggregate data from all nodes.
> So there will be both
>
>  * summary time of the task
>  * time of the each job in the task.
>
>
> > 8 июня 2020 г., в 12:52, Alexey Goncharuk <[hidden email]>
> написал(а):
> >
> > Nikita, Igniters,
> >
> > I left a few comments on the tool itself in the PR.
> >
> > However, I would like to reiterate and discuss why a user would prefer to
> > use the profiling tool over tracing? Profiling tool only captures very
> > high-level details of the operations (a single cache operation, for
> > example), and does not interconnect operations happened on different
> nodes.
> > For example, a user would not be able to match a long task with a long
> job
> > in that task. In other words, profiling data is always a subset of data
> > collected by tracing.
> >
> > Maybe it makes sense to adopt local log file approach to write spans so
> we
> > can process those spans later to build a report?
> >
> > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <[hidden email]>:
> >
> >> Hi, Igniters.
> >>
> >> I have implemented cluster profiling and tool to build the performance
> >> report. It's ready to be reviewed. [1, 2]
> >>
> >> Profiling can be managed by JMX bean. I have plans to implement it to
> >> control.sh also.
> >>
> >> Nodes write statistics to the temporary off heap buffer and then one
> >> thread flushes to the profiling files. The write mechanics and format
> >> is like WAL logging.
> >> The report contains the following statistics:
> >> - nodes and caches info
> >> - cache operations and transaction statistics
> >> - SQL and scan queries statistics (include logical and physical reads
> per
> >> query)
> >> - tasks and jobs statistics.
> >>
> >> More details in the IEP [3].
> >>
> >> [1] https://github.com/apache/ignite/pull/7693
> >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> >> [3]
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>
> >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <
> [hidden email]>:
> >>>
> >>> Hello Nikolay,
> >>>
> >>>> Who deprecated visor and when? Maybe I miss something?
> >>> On the one hand, there was technically no community consensus that this
> >>> tool should be obsolete.
> >>> On the other hand, my opinion based on the following topic:
> >>>
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> >>> Moreover, it seems to me, currently, the control utility is widely used
> >> and
> >>> actively developed, instead of the visor.
> >>>
> >>>> It's true that, for now, Ignite doesn't have "tool strategy" I think
> >> it's
> >>> a big issue from the user's point of view.
> >>> I absolutely agree with that.
> >>>
> >>>> We should solve it in the nearest time. Feel free to start this
> >> activity
> >>> I have no plan at the moment. However, at the first stage, we could
> >>> understand the difference between visor and control utility.
> >>> All useful features from visor should be moved/implemented in control
> >>> utility and after that visor tool and should be marked as
> >>> deprecated/obsoleted.
> >>>
> >>>> You need to throw in control.sh also, which does some kind of
> >> statistics
> >>> too, such as idle_verify.
> >>>> Please, clarify your idea:
> >>>>   We should use some info from control.sh to the report?
> >>>>   The report should be generated by some control.sh subcommand?
> >>> If I am not mistaken, the oracle database has AWR tool (mentioned on
> the
> >>> IEP page) which is a command-line utility that generates HTML reports.
> >>> I like this idea and I think this is a good approach that can be
> realized
> >>> in the control utility.
> >>> If we have a case that cannot be implemented in this way, we have to
> >>> clearly states the difference between these tools so as not to confuse
> >> our
> >>> users.
> >>> What do you think?
> >>>
> >>> Thanks,
> >>> Slava.
> >>>
> >>>
> >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <[hidden email]>:
> >>>
> >>>> Hello, Slava, Ilya, Denis.
> >>>>
> >>>> Thanks for joining this discussion!
> >>>>
> >>>>> - visor (which is deprecated)
> >>>>
> >>>> Who deprecated visor and when?
> >>>> Maybe I miss something?
> >>>>
> >>>>> - web-console (to be honest, I don't quite understand the status of
> >> this
> >>>> tool)
> >>>>
> >>>> +1.
> >>>>
> >>>>> I am not against the new tool, I just want to understand the
> >> motivation
> >>>> to not improve the existing sub-projects.
> >>>>
> >>>> It's true that, for now, Ignite doesn't have "tool strategy"
> >>>> I think it's a big issue from the user's point of view.
> >>>> We should solve it in the nearest time.
> >>>> Feel free to start this activity.
> >>>>
> >>>>> - new ignite-profiling (which is a monitoring tool as well, judging
> >> by
> >>>> the provided link [1] )
> >>>>
> >>>> The general idea is the following:
> >>>>
> >>>> 1. We should have some profiling mechanism that will generate a
> >> node-local
> >>>> event log
> >>>> 2. We should have a tool that can export events to some third-party
> >>>> system. This system can be an Elastic Search(Kibana) or Ignite
> >> performance
> >>>> report or Kafka log, whatever.
> >>>> 3. Ignite performance report, in the first release, should be a
> >> "static"
> >>>> tool.
> >>>>    This means we take static logs(that is not rewritten in the
> >> analysis
> >>>> time) and feed them in the script(or tool or control.sh, whatever)
> >>>>    The script produces static report that can be used for overall
> >>>> performance analysis.
> >>>>
> >>>> The primary users of this report is a developer of Ignite based
> >>>> applications and performance engineers.
> >>>>
> >>>> Ilya,
> >>>>
> >>>>> You need to throw in control.sh also, which does some kind of
> >> statistics
> >>>> too, such as idle_verify.
> >>>>
> >>>> Please, clarify your idea:
> >>>>    We should use some info from control.sh to the report?
> >>>>    The report should be generated by some control.sh subcommand?
> >>>>
> >>>>
> >>>> Denis,
> >>>>
> >>>>> Speaking of the probes/statistics collection approach, is it
> >> supposed to
> >>>> reuse tracing capabilities that are to be added as part of IEP-35?
> >>>>
> >>>> For now, we don't have any results of tracing development available in
> >>>> Apache Ignite.
> >>>> Hopefully, we got some in a couple of weeks.
> >>>> After it, we can start a discussion of how to merge two improvements.
> >>>>
> >>>>
> >>>>
> >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]>
> >> написал(а):
> >>>>>
> >>>>>>
> >>>>>> Tracing is more deeply takes statistics. If it will be possible,
> >> I'm for
> >>>>>> reuse.
> >>>>>
> >>>>>
> >>>>> Looks like we need to sync up on these activities/initiatives to
> >> ensure
> >>>> we
> >>>>> don't do a duplicate job. If you think a separate discussion is
> >> necessary
> >>>>> let's kick it off.
> >>>>>
> >>>>> -
> >>>>> Denis
> >>>>>
> >>>>>
> >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
> >> [hidden email]>
> >>>>> wrote:
> >>>>>
> >>>>>> Denis, Ilya,
> >>>>>>
> >>>>>> I will try to integrate profiling functionality into control.sh
> >> utility.
> >>>>>>
> >>>>>>> Speaking of the probes/statistics collection approach, is it
> >> supposed
> >>>> to
> >>>>>>> reuse tracing capabilities that are to be added as part of IEP-35?
> >>>>>> Tracing is more deeply takes statistics. If it will be possible,
> >> I'm for
> >>>>>> reuse.
> >>>>>>
> >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
> >> [hidden email]
> >>>>> :
> >>>>>>>
> >>>>>>> Hello!
> >>>>>>>
> >>>>>>> I suggest that it's one of the places where it could be put
> >> instead of
> >>>>>>> adding a new tool.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> --
> >>>>>>> Ilya Kasnacheev
> >>>>>>>
> >>>>>>>
> >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]
> >>> :
> >>>>>>>
> >>>>>>>> Ilya,
> >>>>>>>>
> >>>>>>>> You suggest using control.sh to build the report?
> >>>>>>>>
> >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
> >>>>>> [hidden email]>:
> >>>>>>>>>
> >>>>>>>>> Hello!
> >>>>>>>>>
> >>>>>>>>> You need to throw in control.sh also, which does some kind of
> >>>>>> statistics
> >>>>>>>>> too, such as idle_verify.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> --
> >>>>>>>>> Ilya Kasnacheev
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> >>>>>> [hidden email]
> >>>>>>>>> :
> >>>>>>>>>
> >>>>>>>>>> Hello Nikita,
> >>>>>>>>>>
> >>>>>>>>>> Perhaps, I am missing something...
> >>>>>>>>>> Apache Ignite already has a web-console tool. Do we want to
> >>>>>> improve the
> >>>>>>>>>> existing tool instead of creating a new one?
> >>>>>>>>>> It seems to me, this can be confusing for users.
> >>>>>>>>>> - visor (which is deprecated)
> >>>>>>>>>> - web-console (to be honest, I don't quite understand the status
> >>>>>> of
> >>>>>>>> this
> >>>>>>>>>> tool)
> >>>>>>>>>> - new ignite-profiling (which is a monitoring tool as well,
> >>>>>> judging
> >>>>>>>> by the
> >>>>>>>>>> provided link [1] )
> >>>>>>>>>>
> >>>>>>>>>> I am not against the new tool, I just want to understand the
> >>>>>>>> motivation to
> >>>>>>>>>> not improve the existing sub-projects.
> >>>>>>>>>>
> >>>>>>>>>> [1]
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> S.
> >>>>>>>>>>
> >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <
> >> [hidden email]
> >>>>>>> :
> >>>>>>>>>>
> >>>>>>>>>>> Hi, Igniters.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm working on cluster profiling and the tool for creating a
> >>>>>>>>>>> performance report. [1] I have prepared PoC based on
> >> performance
> >>>>>>>>>>> logging to a separate category of Ignite log. The report
> >>>>>> contains:
> >>>>>>>>>>>
> >>>>>>>>>>> - Cache operations and its distribution by types [2]
> >>>>>>>>>>> - Transactions and histogram of durations [3]
> >>>>>>>>>>> - SQL and Scan query statistics, top of slowest queries,
> >> logical
> >>>>>> and
> >>>>>>>>>>> physical reads by query [4]
> >>>>>>>>>>> - Compute statistics, top of slowest tasks and their jobs [5]
> >>>>>>>>>>> Soon I will add:
> >>>>>>>>>>> - Topology and Ignite versions info
> >>>>>>>>>>> - Client ID in case of operations from clients
> >>>>>>>>>>>
> >>>>>>>>>>> For now, I'm developing a binary logging format to reduce the
> >>>>>> effect
> >>>>>>>>>>> on performance. I'll try to reuse Ignite mechanisms.
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to hear suggestions for the profiling and the
> >>>>>>>> performance
> >>>>>>>>>>> report.
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>>>>>>>>>> [2]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> >>>>>>>>>>> [3]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> >>>>>>>>>>> [4]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> >>>>>>>>>>> [5]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Best wishes,
> >>>>>>>>>>> Amelchev Nikita
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Best wishes,
> >>>>>>>> Amelchev Nikita
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best wishes,
> >>>>>> Amelchev Nikita
> >>>>>>
> >>>>
> >>>>
> >>
> >>
> >>
> >> --
> >> Best wishes,
> >> Amelchev Nikita
> >>
>
>

[Nikita Amelchev]

Hello,

> Can you please share more info on how we can use the profiling tool with
> ignite-extensions modules?

I have updated the module readme file. You can find instructions
there. For now, I am working on review fixes and marked PR as a draft.

чт, 11 июн. 2020 г. в 03:21, Saikat Maitra <[hidden email]>:

>
> Hello Nikita,
>
> I observed we have an open PR in ignite repo for this feature with
> different set of changes compared to ignite extensions repo.
>
>
> apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>
>
> https://github.com/apache/ignite-extensions/pull/16
>
> Can you please share more info on how we can use the profiling tool with
> ignite-extensions modules?
>
>
> Regards,
>
> Saikat
>
> On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <[hidden email]> wrote:
>
> > Hello, Alexey.
> >
> > Thanks for the review.
> >
> > My understanding if the following:
> >
> > We will have 3 in-depth tool to find issues in cluster:
> >
> > 1. Metrics + System views - data that describe Ignite entities very
> > high-level.
> >
> > 2. Profiling - tool to know what specific query of transactions are slow.
> > In many cases, this knowledge is enough to fix the issue(rewrite query,
> > redesign transactions flow, etc)
> >
> > 3. Tracing - tool to know why one of 1000 of the same queries was slow.
> > The most detailed view of the Ignite internal processes.
> >
> > > For example, a user would not be able to match a long task with a long
> > job in that task.
> >
> > This is not true.
> > Profiling report will aggregate data from all nodes.
> > So there will be both
> >
> >  * summary time of the task
> >  * time of the each job in the task.
> >
> >
> > > 8 июня 2020 г., в 12:52, Alexey Goncharuk <[hidden email]>
> > написал(а):
> > >
> > > Nikita, Igniters,
> > >
> > > I left a few comments on the tool itself in the PR.
> > >
> > > However, I would like to reiterate and discuss why a user would prefer to
> > > use the profiling tool over tracing? Profiling tool only captures very
> > > high-level details of the operations (a single cache operation, for
> > > example), and does not interconnect operations happened on different
> > nodes.
> > > For example, a user would not be able to match a long task with a long
> > job
> > > in that task. In other words, profiling data is always a subset of data
> > > collected by tracing.
> > >
> > > Maybe it makes sense to adopt local log file approach to write spans so
> > we
> > > can process those spans later to build a report?
> > >
> > > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <[hidden email]>:
> > >
> > >> Hi, Igniters.
> > >>
> > >> I have implemented cluster profiling and tool to build the performance
> > >> report. It's ready to be reviewed. [1, 2]
> > >>
> > >> Profiling can be managed by JMX bean. I have plans to implement it to
> > >> control.sh also.
> > >>
> > >> Nodes write statistics to the temporary off heap buffer and then one
> > >> thread flushes to the profiling files. The write mechanics and format
> > >> is like WAL logging.
> > >> The report contains the following statistics:
> > >> - nodes and caches info
> > >> - cache operations and transaction statistics
> > >> - SQL and scan queries statistics (include logical and physical reads
> > per
> > >> query)
> > >> - tasks and jobs statistics.
> > >>
> > >> More details in the IEP [3].
> > >>
> > >> [1] https://github.com/apache/ignite/pull/7693
> > >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> > >> [3]
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>
> > >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <
> > [hidden email]>:
> > >>>
> > >>> Hello Nikolay,
> > >>>
> > >>>> Who deprecated visor and when? Maybe I miss something?
> > >>> On the one hand, there was technically no community consensus that this
> > >>> tool should be obsolete.
> > >>> On the other hand, my opinion based on the following topic:
> > >>>
> > >>
> > http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> > >>> Moreover, it seems to me, currently, the control utility is widely used
> > >> and
> > >>> actively developed, instead of the visor.
> > >>>
> > >>>> It's true that, for now, Ignite doesn't have "tool strategy" I think
> > >> it's
> > >>> a big issue from the user's point of view.
> > >>> I absolutely agree with that.
> > >>>
> > >>>> We should solve it in the nearest time. Feel free to start this
> > >> activity
> > >>> I have no plan at the moment. However, at the first stage, we could
> > >>> understand the difference between visor and control utility.
> > >>> All useful features from visor should be moved/implemented in control
> > >>> utility and after that visor tool and should be marked as
> > >>> deprecated/obsoleted.
> > >>>
> > >>>> You need to throw in control.sh also, which does some kind of
> > >> statistics
> > >>> too, such as idle_verify.
> > >>>> Please, clarify your idea:
> > >>>>   We should use some info from control.sh to the report?
> > >>>>   The report should be generated by some control.sh subcommand?
> > >>> If I am not mistaken, the oracle database has AWR tool (mentioned on
> > the
> > >>> IEP page) which is a command-line utility that generates HTML reports.
> > >>> I like this idea and I think this is a good approach that can be
> > realized
> > >>> in the control utility.
> > >>> If we have a case that cannot be implemented in this way, we have to
> > >>> clearly states the difference between these tools so as not to confuse
> > >> our
> > >>> users.
> > >>> What do you think?
> > >>>
> > >>> Thanks,
> > >>> Slava.
> > >>>
> > >>>
> > >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <[hidden email]>:
> > >>>
> > >>>> Hello, Slava, Ilya, Denis.
> > >>>>
> > >>>> Thanks for joining this discussion!
> > >>>>
> > >>>>> - visor (which is deprecated)
> > >>>>
> > >>>> Who deprecated visor and when?
> > >>>> Maybe I miss something?
> > >>>>
> > >>>>> - web-console (to be honest, I don't quite understand the status of
> > >> this
> > >>>> tool)
> > >>>>
> > >>>> +1.
> > >>>>
> > >>>>> I am not against the new tool, I just want to understand the
> > >> motivation
> > >>>> to not improve the existing sub-projects.
> > >>>>
> > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
> > >>>> I think it's a big issue from the user's point of view.
> > >>>> We should solve it in the nearest time.
> > >>>> Feel free to start this activity.
> > >>>>
> > >>>>> - new ignite-profiling (which is a monitoring tool as well, judging
> > >> by
> > >>>> the provided link [1] )
> > >>>>
> > >>>> The general idea is the following:
> > >>>>
> > >>>> 1. We should have some profiling mechanism that will generate a
> > >> node-local
> > >>>> event log
> > >>>> 2. We should have a tool that can export events to some third-party
> > >>>> system. This system can be an Elastic Search(Kibana) or Ignite
> > >> performance
> > >>>> report or Kafka log, whatever.
> > >>>> 3. Ignite performance report, in the first release, should be a
> > >> "static"
> > >>>> tool.
> > >>>>    This means we take static logs(that is not rewritten in the
> > >> analysis
> > >>>> time) and feed them in the script(or tool or control.sh, whatever)
> > >>>>    The script produces static report that can be used for overall
> > >>>> performance analysis.
> > >>>>
> > >>>> The primary users of this report is a developer of Ignite based
> > >>>> applications and performance engineers.
> > >>>>
> > >>>> Ilya,
> > >>>>
> > >>>>> You need to throw in control.sh also, which does some kind of
> > >> statistics
> > >>>> too, such as idle_verify.
> > >>>>
> > >>>> Please, clarify your idea:
> > >>>>    We should use some info from control.sh to the report?
> > >>>>    The report should be generated by some control.sh subcommand?
> > >>>>
> > >>>>
> > >>>> Denis,
> > >>>>
> > >>>>> Speaking of the probes/statistics collection approach, is it
> > >> supposed to
> > >>>> reuse tracing capabilities that are to be added as part of IEP-35?
> > >>>>
> > >>>> For now, we don't have any results of tracing development available in
> > >>>> Apache Ignite.
> > >>>> Hopefully, we got some in a couple of weeks.
> > >>>> After it, we can start a discussion of how to merge two improvements.
> > >>>>
> > >>>>
> > >>>>
> > >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]>
> > >> написал(а):
> > >>>>>
> > >>>>>>
> > >>>>>> Tracing is more deeply takes statistics. If it will be possible,
> > >> I'm for
> > >>>>>> reuse.
> > >>>>>
> > >>>>>
> > >>>>> Looks like we need to sync up on these activities/initiatives to
> > >> ensure
> > >>>> we
> > >>>>> don't do a duplicate job. If you think a separate discussion is
> > >> necessary
> > >>>>> let's kick it off.
> > >>>>>
> > >>>>> -
> > >>>>> Denis
> > >>>>>
> > >>>>>
> > >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
> > >> [hidden email]>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Denis, Ilya,
> > >>>>>>
> > >>>>>> I will try to integrate profiling functionality into control.sh
> > >> utility.
> > >>>>>>
> > >>>>>>> Speaking of the probes/statistics collection approach, is it
> > >> supposed
> > >>>> to
> > >>>>>>> reuse tracing capabilities that are to be added as part of IEP-35?
> > >>>>>> Tracing is more deeply takes statistics. If it will be possible,
> > >> I'm for
> > >>>>>> reuse.
> > >>>>>>
> > >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
> > >> [hidden email]
> > >>>>> :
> > >>>>>>>
> > >>>>>>> Hello!
> > >>>>>>>
> > >>>>>>> I suggest that it's one of the places where it could be put
> > >> instead of
> > >>>>>>> adding a new tool.
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> --
> > >>>>>>> Ilya Kasnacheev
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <[hidden email]
> > >>> :
> > >>>>>>>
> > >>>>>>>> Ilya,
> > >>>>>>>>
> > >>>>>>>> You suggest using control.sh to build the report?
> > >>>>>>>>
> > >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
> > >>>>>> [hidden email]>:
> > >>>>>>>>>
> > >>>>>>>>> Hello!
> > >>>>>>>>>
> > >>>>>>>>> You need to throw in control.sh also, which does some kind of
> > >>>>>> statistics
> > >>>>>>>>> too, such as idle_verify.
> > >>>>>>>>>
> > >>>>>>>>> Regards,
> > >>>>>>>>> --
> > >>>>>>>>> Ilya Kasnacheev
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> > >>>>>> [hidden email]
> > >>>>>>>>> :
> > >>>>>>>>>
> > >>>>>>>>>> Hello Nikita,
> > >>>>>>>>>>
> > >>>>>>>>>> Perhaps, I am missing something...
> > >>>>>>>>>> Apache Ignite already has a web-console tool. Do we want to
> > >>>>>> improve the
> > >>>>>>>>>> existing tool instead of creating a new one?
> > >>>>>>>>>> It seems to me, this can be confusing for users.
> > >>>>>>>>>> - visor (which is deprecated)
> > >>>>>>>>>> - web-console (to be honest, I don't quite understand the status
> > >>>>>> of
> > >>>>>>>> this
> > >>>>>>>>>> tool)
> > >>>>>>>>>> - new ignite-profiling (which is a monitoring tool as well,
> > >>>>>> judging
> > >>>>>>>> by the
> > >>>>>>>>>> provided link [1] )
> > >>>>>>>>>>
> > >>>>>>>>>> I am not against the new tool, I just want to understand the
> > >>>>>>>> motivation to
> > >>>>>>>>>> not improve the existing sub-projects.
> > >>>>>>>>>>
> > >>>>>>>>>> [1]
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> S.
> > >>>>>>>>>>
> > >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <
> > >> [hidden email]
> > >>>>>>> :
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi, Igniters.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm working on cluster profiling and the tool for creating a
> > >>>>>>>>>>> performance report. [1] I have prepared PoC based on
> > >> performance
> > >>>>>>>>>>> logging to a separate category of Ignite log. The report
> > >>>>>> contains:
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Cache operations and its distribution by types [2]
> > >>>>>>>>>>> - Transactions and histogram of durations [3]
> > >>>>>>>>>>> - SQL and Scan query statistics, top of slowest queries,
> > >> logical
> > >>>>>> and
> > >>>>>>>>>>> physical reads by query [4]
> > >>>>>>>>>>> - Compute statistics, top of slowest tasks and their jobs [5]
> > >>>>>>>>>>> Soon I will add:
> > >>>>>>>>>>> - Topology and Ignite versions info
> > >>>>>>>>>>> - Client ID in case of operations from clients
> > >>>>>>>>>>>
> > >>>>>>>>>>> For now, I'm developing a binary logging format to reduce the
> > >>>>>> effect
> > >>>>>>>>>>> on performance. I'll try to reuse Ignite mechanisms.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I would like to hear suggestions for the profiling and the
> > >>>>>>>> performance
> > >>>>>>>>>>> report.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [1]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>>>>>>>>>> [2]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > >>>>>>>>>>> [3]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > >>>>>>>>>>> [4]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > >>>>>>>>>>> [5]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Best wishes,
> > >>>>>>>>>>> Amelchev Nikita
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Best wishes,
> > >>>>>>>> Amelchev Nikita
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best wishes,
> > >>>>>> Amelchev Nikita
> > >>>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> > >>
> > >> --
> > >> Best wishes,
> > >> Amelchev Nikita
> > >>
> >
> >

--
Best wishes,
Amelchev Nikita

[Saikat Maitra]

Hello Nikita,

Thank you for sharing the information, it is very helpful.

Regards,
Saikat

On Thu, Jun 11, 2020 at 3:20 AM Nikita Amelchev <[hidden email]>
wrote:

> Hello,
>
> > Can you please share more info on how we can use the profiling tool with
> > ignite-extensions modules?
>
> I have updated the module readme file. You can find instructions
> there. For now, I am working on review fixes and marked PR as a draft.
>
> чт, 11 июн. 2020 г. в 03:21, Saikat Maitra <[hidden email]>:
> >
> > Hello Nikita,
> >
> > I observed we have an open PR in ignite repo for this feature with
> > different set of changes compared to ignite extensions repo.
> >
> >
> > apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>
> >
> > https://github.com/apache/ignite-extensions/pull/16
> >
> > Can you please share more info on how we can use the profiling tool with
> > ignite-extensions modules?
> >
> >
> > Regards,
> >
> > Saikat
> >
> > On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <[hidden email]>
> wrote:
> >
> > > Hello, Alexey.
> > >
> > > Thanks for the review.
> > >
> > > My understanding if the following:
> > >
> > > We will have 3 in-depth tool to find issues in cluster:
> > >
> > > 1. Metrics + System views - data that describe Ignite entities very
> > > high-level.
> > >
> > > 2. Profiling - tool to know what specific query of transactions are
> slow.
> > > In many cases, this knowledge is enough to fix the issue(rewrite query,
> > > redesign transactions flow, etc)
> > >
> > > 3. Tracing - tool to know why one of 1000 of the same queries was slow.
> > > The most detailed view of the Ignite internal processes.
> > >
> > > > For example, a user would not be able to match a long task with a
> long
> > > job in that task.
> > >
> > > This is not true.
> > > Profiling report will aggregate data from all nodes.
> > > So there will be both
> > >
> > >  * summary time of the task
> > >  * time of the each job in the task.
> > >
> > >
> > > > 8 июня 2020 г., в 12:52, Alexey Goncharuk <
> [hidden email]>
> > > написал(а):
> > > >
> > > > Nikita, Igniters,
> > > >
> > > > I left a few comments on the tool itself in the PR.
> > > >
> > > > However, I would like to reiterate and discuss why a user would
> prefer to
> > > > use the profiling tool over tracing? Profiling tool only captures
> very
> > > > high-level details of the operations (a single cache operation, for
> > > > example), and does not interconnect operations happened on different
> > > nodes.
> > > > For example, a user would not be able to match a long task with a
> long
> > > job
> > > > in that task. In other words, profiling data is always a subset of
> data
> > > > collected by tracing.
> > > >
> > > > Maybe it makes sense to adopt local log file approach to write spans
> so
> > > we
> > > > can process those spans later to build a report?
> > > >
> > > > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <[hidden email]>:
> > > >
> > > >> Hi, Igniters.
> > > >>
> > > >> I have implemented cluster profiling and tool to build the
> performance
> > > >> report. It's ready to be reviewed. [1, 2]
> > > >>
> > > >> Profiling can be managed by JMX bean. I have plans to implement it
> to
> > > >> control.sh also.
> > > >>
> > > >> Nodes write statistics to the temporary off heap buffer and then one
> > > >> thread flushes to the profiling files. The write mechanics and
> format
> > > >> is like WAL logging.
> > > >> The report contains the following statistics:
> > > >> - nodes and caches info
> > > >> - cache operations and transaction statistics
> > > >> - SQL and scan queries statistics (include logical and physical
> reads
> > > per
> > > >> query)
> > > >> - tasks and jobs statistics.
> > > >>
> > > >> More details in the IEP [3].
> > > >>
> > > >> [1] https://github.com/apache/ignite/pull/7693
> > > >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> > > >> [3]
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >>
> > > >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <
> > > [hidden email]>:
> > > >>>
> > > >>> Hello Nikolay,
> > > >>>
> > > >>>> Who deprecated visor and when? Maybe I miss something?
> > > >>> On the one hand, there was technically no community consensus that
> this
> > > >>> tool should be obsolete.
> > > >>> On the other hand, my opinion based on the following topic:
> > > >>>
> > > >>
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> > > >>> Moreover, it seems to me, currently, the control utility is widely
> used
> > > >> and
> > > >>> actively developed, instead of the visor.
> > > >>>
> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy" I
> think
> > > >> it's
> > > >>> a big issue from the user's point of view.
> > > >>> I absolutely agree with that.
> > > >>>
> > > >>>> We should solve it in the nearest time. Feel free to start this
> > > >> activity
> > > >>> I have no plan at the moment. However, at the first stage, we could
> > > >>> understand the difference between visor and control utility.
> > > >>> All useful features from visor should be moved/implemented in
> control
> > > >>> utility and after that visor tool and should be marked as
> > > >>> deprecated/obsoleted.
> > > >>>
> > > >>>> You need to throw in control.sh also, which does some kind of
> > > >> statistics
> > > >>> too, such as idle_verify.
> > > >>>> Please, clarify your idea:
> > > >>>>   We should use some info from control.sh to the report?
> > > >>>>   The report should be generated by some control.sh subcommand?
> > > >>> If I am not mistaken, the oracle database has AWR tool (mentioned
> on
> > > the
> > > >>> IEP page) which is a command-line utility that generates HTML
> reports.
> > > >>> I like this idea and I think this is a good approach that can be
> > > realized
> > > >>> in the control utility.
> > > >>> If we have a case that cannot be implemented in this way, we have
> to
> > > >>> clearly states the difference between these tools so as not to
> confuse
> > > >> our
> > > >>> users.
> > > >>> What do you think?
> > > >>>
> > > >>> Thanks,
> > > >>> Slava.
> > > >>>
> > > >>>
> > > >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <[hidden email]
> >:
> > > >>>
> > > >>>> Hello, Slava, Ilya, Denis.
> > > >>>>
> > > >>>> Thanks for joining this discussion!
> > > >>>>
> > > >>>>> - visor (which is deprecated)
> > > >>>>
> > > >>>> Who deprecated visor and when?
> > > >>>> Maybe I miss something?
> > > >>>>
> > > >>>>> - web-console (to be honest, I don't quite understand the status
> of
> > > >> this
> > > >>>> tool)
> > > >>>>
> > > >>>> +1.
> > > >>>>
> > > >>>>> I am not against the new tool, I just want to understand the
> > > >> motivation
> > > >>>> to not improve the existing sub-projects.
> > > >>>>
> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
> > > >>>> I think it's a big issue from the user's point of view.
> > > >>>> We should solve it in the nearest time.
> > > >>>> Feel free to start this activity.
> > > >>>>
> > > >>>>> - new ignite-profiling (which is a monitoring tool as well,
> judging
> > > >> by
> > > >>>> the provided link [1] )
> > > >>>>
> > > >>>> The general idea is the following:
> > > >>>>
> > > >>>> 1. We should have some profiling mechanism that will generate a
> > > >> node-local
> > > >>>> event log
> > > >>>> 2. We should have a tool that can export events to some
> third-party
> > > >>>> system. This system can be an Elastic Search(Kibana) or Ignite
> > > >> performance
> > > >>>> report or Kafka log, whatever.
> > > >>>> 3. Ignite performance report, in the first release, should be a
> > > >> "static"
> > > >>>> tool.
> > > >>>>    This means we take static logs(that is not rewritten in the
> > > >> analysis
> > > >>>> time) and feed them in the script(or tool or control.sh, whatever)
> > > >>>>    The script produces static report that can be used for overall
> > > >>>> performance analysis.
> > > >>>>
> > > >>>> The primary users of this report is a developer of Ignite based
> > > >>>> applications and performance engineers.
> > > >>>>
> > > >>>> Ilya,
> > > >>>>
> > > >>>>> You need to throw in control.sh also, which does some kind of
> > > >> statistics
> > > >>>> too, such as idle_verify.
> > > >>>>
> > > >>>> Please, clarify your idea:
> > > >>>>    We should use some info from control.sh to the report?
> > > >>>>    The report should be generated by some control.sh subcommand?
> > > >>>>
> > > >>>>
> > > >>>> Denis,
> > > >>>>
> > > >>>>> Speaking of the probes/statistics collection approach, is it
> > > >> supposed to
> > > >>>> reuse tracing capabilities that are to be added as part of IEP-35?
> > > >>>>
> > > >>>> For now, we don't have any results of tracing development
> available in
> > > >>>> Apache Ignite.
> > > >>>> Hopefully, we got some in a couple of weeks.
> > > >>>> After it, we can start a discussion of how to merge two
> improvements.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]>
> > > >> написал(а):
> > > >>>>>
> > > >>>>>>
> > > >>>>>> Tracing is more deeply takes statistics. If it will be possible,
> > > >> I'm for
> > > >>>>>> reuse.
> > > >>>>>
> > > >>>>>
> > > >>>>> Looks like we need to sync up on these activities/initiatives to
> > > >> ensure
> > > >>>> we
> > > >>>>> don't do a duplicate job. If you think a separate discussion is
> > > >> necessary
> > > >>>>> let's kick it off.
> > > >>>>>
> > > >>>>> -
> > > >>>>> Denis
> > > >>>>>
> > > >>>>>
> > > >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
> > > >> [hidden email]>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Denis, Ilya,
> > > >>>>>>
> > > >>>>>> I will try to integrate profiling functionality into control.sh
> > > >> utility.
> > > >>>>>>
> > > >>>>>>> Speaking of the probes/statistics collection approach, is it
> > > >> supposed
> > > >>>> to
> > > >>>>>>> reuse tracing capabilities that are to be added as part of
> IEP-35?
> > > >>>>>> Tracing is more deeply takes statistics. If it will be possible,
> > > >> I'm for
> > > >>>>>> reuse.
> > > >>>>>>
> > > >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
> > > >> [hidden email]
> > > >>>>> :
> > > >>>>>>>
> > > >>>>>>> Hello!
> > > >>>>>>>
> > > >>>>>>> I suggest that it's one of the places where it could be put
> > > >> instead of
> > > >>>>>>> adding a new tool.
> > > >>>>>>>
> > > >>>>>>> Regards,
> > > >>>>>>> --
> > > >>>>>>> Ilya Kasnacheev
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <
> [hidden email]
> > > >>> :
> > > >>>>>>>
> > > >>>>>>>> Ilya,
> > > >>>>>>>>
> > > >>>>>>>> You suggest using control.sh to build the report?
> > > >>>>>>>>
> > > >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
> > > >>>>>> [hidden email]>:
> > > >>>>>>>>>
> > > >>>>>>>>> Hello!
> > > >>>>>>>>>
> > > >>>>>>>>> You need to throw in control.sh also, which does some kind of
> > > >>>>>> statistics
> > > >>>>>>>>> too, such as idle_verify.
> > > >>>>>>>>>
> > > >>>>>>>>> Regards,
> > > >>>>>>>>> --
> > > >>>>>>>>> Ilya Kasnacheev
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> > > >>>>>> [hidden email]
> > > >>>>>>>>> :
> > > >>>>>>>>>
> > > >>>>>>>>>> Hello Nikita,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Perhaps, I am missing something...
> > > >>>>>>>>>> Apache Ignite already has a web-console tool. Do we want to
> > > >>>>>> improve the
> > > >>>>>>>>>> existing tool instead of creating a new one?
> > > >>>>>>>>>> It seems to me, this can be confusing for users.
> > > >>>>>>>>>> - visor (which is deprecated)
> > > >>>>>>>>>> - web-console (to be honest, I don't quite understand the
> status
> > > >>>>>> of
> > > >>>>>>>> this
> > > >>>>>>>>>> tool)
> > > >>>>>>>>>> - new ignite-profiling (which is a monitoring tool as well,
> > > >>>>>> judging
> > > >>>>>>>> by the
> > > >>>>>>>>>> provided link [1] )
> > > >>>>>>>>>>
> > > >>>>>>>>>> I am not against the new tool, I just want to understand the
> > > >>>>>>>> motivation to
> > > >>>>>>>>>> not improve the existing sub-projects.
> > > >>>>>>>>>>
> > > >>>>>>>>>> [1]
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks,
> > > >>>>>>>>>> S.
> > > >>>>>>>>>>
> > > >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <
> > > >> [hidden email]
> > > >>>>>>> :
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi, Igniters.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm working on cluster profiling and the tool for creating
> a
> > > >>>>>>>>>>> performance report. [1] I have prepared PoC based on
> > > >> performance
> > > >>>>>>>>>>> logging to a separate category of Ignite log. The report
> > > >>>>>> contains:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Cache operations and its distribution by types [2]
> > > >>>>>>>>>>> - Transactions and histogram of durations [3]
> > > >>>>>>>>>>> - SQL and Scan query statistics, top of slowest queries,
> > > >> logical
> > > >>>>>> and
> > > >>>>>>>>>>> physical reads by query [4]
> > > >>>>>>>>>>> - Compute statistics, top of slowest tasks and their jobs
> [5]
> > > >>>>>>>>>>> Soon I will add:
> > > >>>>>>>>>>> - Topology and Ignite versions info
> > > >>>>>>>>>>> - Client ID in case of operations from clients
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> For now, I'm developing a binary logging format to reduce
> the
> > > >>>>>> effect
> > > >>>>>>>>>>> on performance. I'll try to reuse Ignite mechanisms.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I would like to hear suggestions for the profiling and the
> > > >>>>>>>> performance
> > > >>>>>>>>>>> report.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [1]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >>>>>>>>>>> [2]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > >>>>>>>>>>> [3]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > >>>>>>>>>>> [4]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > >>>>>>>>>>> [5]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> --
> > > >>>>>>>>>>> Best wishes,
> > > >>>>>>>>>>> Amelchev Nikita
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> --
> > > >>>>>>>> Best wishes,
> > > >>>>>>>> Amelchev Nikita
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Best wishes,
> > > >>>>>> Amelchev Nikita
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best wishes,
> > > >> Amelchev Nikita
> > > >>
> > >
> > >
>
>
>
> --
> Best wishes,
> Amelchev Nikita
>

[Saikat Maitra]

Hi Nikita,

I have reviewed the PR and shared comments. I also had a question on
including files with a MIT license.

Are we ok to use both Apache and MIT licenses in our source files?

Regards,
Saikat

On Fri, Jun 12, 2020 at 8:45 PM Saikat Maitra <[hidden email]>
wrote:

> Hello Nikita,
>
> Thank you for sharing the information, it is very helpful.
>
> Regards,
> Saikat
>
> On Thu, Jun 11, 2020 at 3:20 AM Nikita Amelchev <[hidden email]>
> wrote:
>
>> Hello,
>>
>> > Can you please share more info on how we can use the profiling tool with
>> > ignite-extensions modules?
>>
>> I have updated the module readme file. You can find instructions
>> there. For now, I am working on review fixes and marked PR as a draft.
>>
>> чт, 11 июн. 2020 г. в 03:21, Saikat Maitra <[hidden email]>:
>> >
>> > Hello Nikita,
>> >
>> > I observed we have an open PR in ignite repo for this feature with
>> > different set of changes compared to ignite extensions repo.
>> >
>> >
>> > apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>
>> >
>> > https://github.com/apache/ignite-extensions/pull/16
>> >
>> > Can you please share more info on how we can use the profiling tool with
>> > ignite-extensions modules?
>> >
>> >
>> > Regards,
>> >
>> > Saikat
>> >
>> > On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <[hidden email]>
>> wrote:
>> >
>> > > Hello, Alexey.
>> > >
>> > > Thanks for the review.
>> > >
>> > > My understanding if the following:
>> > >
>> > > We will have 3 in-depth tool to find issues in cluster:
>> > >
>> > > 1. Metrics + System views - data that describe Ignite entities very
>> > > high-level.
>> > >
>> > > 2. Profiling - tool to know what specific query of transactions are
>> slow.
>> > > In many cases, this knowledge is enough to fix the issue(rewrite
>> query,
>> > > redesign transactions flow, etc)
>> > >
>> > > 3. Tracing - tool to know why one of 1000 of the same queries was
>> slow.
>> > > The most detailed view of the Ignite internal processes.
>> > >
>> > > > For example, a user would not be able to match a long task with a
>> long
>> > > job in that task.
>> > >
>> > > This is not true.
>> > > Profiling report will aggregate data from all nodes.
>> > > So there will be both
>> > >
>> > >  * summary time of the task
>> > >  * time of the each job in the task.
>> > >
>> > >
>> > > > 8 июня 2020 г., в 12:52, Alexey Goncharuk <
>> [hidden email]>
>> > > написал(а):
>> > > >
>> > > > Nikita, Igniters,
>> > > >
>> > > > I left a few comments on the tool itself in the PR.
>> > > >
>> > > > However, I would like to reiterate and discuss why a user would
>> prefer to
>> > > > use the profiling tool over tracing? Profiling tool only captures
>> very
>> > > > high-level details of the operations (a single cache operation, for
>> > > > example), and does not interconnect operations happened on different
>> > > nodes.
>> > > > For example, a user would not be able to match a long task with a
>> long
>> > > job
>> > > > in that task. In other words, profiling data is always a subset of
>> data
>> > > > collected by tracing.
>> > > >
>> > > > Maybe it makes sense to adopt local log file approach to write
>> spans so
>> > > we
>> > > > can process those spans later to build a report?
>> > > >
>> > > > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <[hidden email]>:
>> > > >
>> > > >> Hi, Igniters.
>> > > >>
>> > > >> I have implemented cluster profiling and tool to build the
>> performance
>> > > >> report. It's ready to be reviewed. [1, 2]
>> > > >>
>> > > >> Profiling can be managed by JMX bean. I have plans to implement it
>> to
>> > > >> control.sh also.
>> > > >>
>> > > >> Nodes write statistics to the temporary off heap buffer and then
>> one
>> > > >> thread flushes to the profiling files. The write mechanics and
>> format
>> > > >> is like WAL logging.
>> > > >> The report contains the following statistics:
>> > > >> - nodes and caches info
>> > > >> - cache operations and transaction statistics
>> > > >> - SQL and scan queries statistics (include logical and physical
>> reads
>> > > per
>> > > >> query)
>> > > >> - tasks and jobs statistics.
>> > > >>
>> > > >> More details in the IEP [3].
>> > > >>
>> > > >> [1] https://github.com/apache/ignite/pull/7693
>> > > >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
>> > > >> [3]
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>> > > >>
>> > > >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <
>> > > [hidden email]>:
>> > > >>>
>> > > >>> Hello Nikolay,
>> > > >>>
>> > > >>>> Who deprecated visor and when? Maybe I miss something?
>> > > >>> On the one hand, there was technically no community consensus
>> that this
>> > > >>> tool should be obsolete.
>> > > >>> On the other hand, my opinion based on the following topic:
>> > > >>>
>> > > >>
>> > >
>> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
>> > > >>> Moreover, it seems to me, currently, the control utility is
>> widely used
>> > > >> and
>> > > >>> actively developed, instead of the visor.
>> > > >>>
>> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy" I
>> think
>> > > >> it's
>> > > >>> a big issue from the user's point of view.
>> > > >>> I absolutely agree with that.
>> > > >>>
>> > > >>>> We should solve it in the nearest time. Feel free to start this
>> > > >> activity
>> > > >>> I have no plan at the moment. However, at the first stage, we
>> could
>> > > >>> understand the difference between visor and control utility.
>> > > >>> All useful features from visor should be moved/implemented in
>> control
>> > > >>> utility and after that visor tool and should be marked as
>> > > >>> deprecated/obsoleted.
>> > > >>>
>> > > >>>> You need to throw in control.sh also, which does some kind of
>> > > >> statistics
>> > > >>> too, such as idle_verify.
>> > > >>>> Please, clarify your idea:
>> > > >>>>   We should use some info from control.sh to the report?
>> > > >>>>   The report should be generated by some control.sh subcommand?
>> > > >>> If I am not mistaken, the oracle database has AWR tool (mentioned
>> on
>> > > the
>> > > >>> IEP page) which is a command-line utility that generates HTML
>> reports.
>> > > >>> I like this idea and I think this is a good approach that can be
>> > > realized
>> > > >>> in the control utility.
>> > > >>> If we have a case that cannot be implemented in this way, we have
>> to
>> > > >>> clearly states the difference between these tools so as not to
>> confuse
>> > > >> our
>> > > >>> users.
>> > > >>> What do you think?
>> > > >>>
>> > > >>> Thanks,
>> > > >>> Slava.
>> > > >>>
>> > > >>>
>> > > >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <[hidden email]
>> >:
>> > > >>>
>> > > >>>> Hello, Slava, Ilya, Denis.
>> > > >>>>
>> > > >>>> Thanks for joining this discussion!
>> > > >>>>
>> > > >>>>> - visor (which is deprecated)
>> > > >>>>
>> > > >>>> Who deprecated visor and when?
>> > > >>>> Maybe I miss something?
>> > > >>>>
>> > > >>>>> - web-console (to be honest, I don't quite understand the
>> status of
>> > > >> this
>> > > >>>> tool)
>> > > >>>>
>> > > >>>> +1.
>> > > >>>>
>> > > >>>>> I am not against the new tool, I just want to understand the
>> > > >> motivation
>> > > >>>> to not improve the existing sub-projects.
>> > > >>>>
>> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
>> > > >>>> I think it's a big issue from the user's point of view.
>> > > >>>> We should solve it in the nearest time.
>> > > >>>> Feel free to start this activity.
>> > > >>>>
>> > > >>>>> - new ignite-profiling (which is a monitoring tool as well,
>> judging
>> > > >> by
>> > > >>>> the provided link [1] )
>> > > >>>>
>> > > >>>> The general idea is the following:
>> > > >>>>
>> > > >>>> 1. We should have some profiling mechanism that will generate a
>> > > >> node-local
>> > > >>>> event log
>> > > >>>> 2. We should have a tool that can export events to some
>> third-party
>> > > >>>> system. This system can be an Elastic Search(Kibana) or Ignite
>> > > >> performance
>> > > >>>> report or Kafka log, whatever.
>> > > >>>> 3. Ignite performance report, in the first release, should be a
>> > > >> "static"
>> > > >>>> tool.
>> > > >>>>    This means we take static logs(that is not rewritten in the
>> > > >> analysis
>> > > >>>> time) and feed them in the script(or tool or control.sh,
>> whatever)
>> > > >>>>    The script produces static report that can be used for overall
>> > > >>>> performance analysis.
>> > > >>>>
>> > > >>>> The primary users of this report is a developer of Ignite based
>> > > >>>> applications and performance engineers.
>> > > >>>>
>> > > >>>> Ilya,
>> > > >>>>
>> > > >>>>> You need to throw in control.sh also, which does some kind of
>> > > >> statistics
>> > > >>>> too, such as idle_verify.
>> > > >>>>
>> > > >>>> Please, clarify your idea:
>> > > >>>>    We should use some info from control.sh to the report?
>> > > >>>>    The report should be generated by some control.sh subcommand?
>> > > >>>>
>> > > >>>>
>> > > >>>> Denis,
>> > > >>>>
>> > > >>>>> Speaking of the probes/statistics collection approach, is it
>> > > >> supposed to
>> > > >>>> reuse tracing capabilities that are to be added as part of
>> IEP-35?
>> > > >>>>
>> > > >>>> For now, we don't have any results of tracing development
>> available in
>> > > >>>> Apache Ignite.
>> > > >>>> Hopefully, we got some in a couple of weeks.
>> > > >>>> After it, we can start a discussion of how to merge two
>> improvements.
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]>
>> > > >> написал(а):
>> > > >>>>>
>> > > >>>>>>
>> > > >>>>>> Tracing is more deeply takes statistics. If it will be
>> possible,
>> > > >> I'm for
>> > > >>>>>> reuse.
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> Looks like we need to sync up on these activities/initiatives to
>> > > >> ensure
>> > > >>>> we
>> > > >>>>> don't do a duplicate job. If you think a separate discussion is
>> > > >> necessary
>> > > >>>>> let's kick it off.
>> > > >>>>>
>> > > >>>>> -
>> > > >>>>> Denis
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
>> > > >> [hidden email]>
>> > > >>>>> wrote:
>> > > >>>>>
>> > > >>>>>> Denis, Ilya,
>> > > >>>>>>
>> > > >>>>>> I will try to integrate profiling functionality into control.sh
>> > > >> utility.
>> > > >>>>>>
>> > > >>>>>>> Speaking of the probes/statistics collection approach, is it
>> > > >> supposed
>> > > >>>> to
>> > > >>>>>>> reuse tracing capabilities that are to be added as part of
>> IEP-35?
>> > > >>>>>> Tracing is more deeply takes statistics. If it will be
>> possible,
>> > > >> I'm for
>> > > >>>>>> reuse.
>> > > >>>>>>
>> > > >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
>> > > >> [hidden email]
>> > > >>>>> :
>> > > >>>>>>>
>> > > >>>>>>> Hello!
>> > > >>>>>>>
>> > > >>>>>>> I suggest that it's one of the places where it could be put
>> > > >> instead of
>> > > >>>>>>> adding a new tool.
>> > > >>>>>>>
>> > > >>>>>>> Regards,
>> > > >>>>>>> --
>> > > >>>>>>> Ilya Kasnacheev
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <
>> [hidden email]
>> > > >>> :
>> > > >>>>>>>
>> > > >>>>>>>> Ilya,
>> > > >>>>>>>>
>> > > >>>>>>>> You suggest using control.sh to build the report?
>> > > >>>>>>>>
>> > > >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
>> > > >>>>>> [hidden email]>:
>> > > >>>>>>>>>
>> > > >>>>>>>>> Hello!
>> > > >>>>>>>>>
>> > > >>>>>>>>> You need to throw in control.sh also, which does some kind
>> of
>> > > >>>>>> statistics
>> > > >>>>>>>>> too, such as idle_verify.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Regards,
>> > > >>>>>>>>> --
>> > > >>>>>>>>> Ilya Kasnacheev
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
>> > > >>>>>> [hidden email]
>> > > >>>>>>>>> :
>> > > >>>>>>>>>
>> > > >>>>>>>>>> Hello Nikita,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Perhaps, I am missing something...
>> > > >>>>>>>>>> Apache Ignite already has a web-console tool. Do we want to
>> > > >>>>>> improve the
>> > > >>>>>>>>>> existing tool instead of creating a new one?
>> > > >>>>>>>>>> It seems to me, this can be confusing for users.
>> > > >>>>>>>>>> - visor (which is deprecated)
>> > > >>>>>>>>>> - web-console (to be honest, I don't quite understand the
>> status
>> > > >>>>>> of
>> > > >>>>>>>> this
>> > > >>>>>>>>>> tool)
>> > > >>>>>>>>>> - new ignite-profiling (which is a monitoring tool as well,
>> > > >>>>>> judging
>> > > >>>>>>>> by the
>> > > >>>>>>>>>> provided link [1] )
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> I am not against the new tool, I just want to understand
>> the
>> > > >>>>>>>> motivation to
>> > > >>>>>>>>>> not improve the existing sub-projects.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> [1]
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Thanks,
>> > > >>>>>>>>>> S.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <
>> > > >> [hidden email]
>> > > >>>>>>> :
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Hi, Igniters.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I'm working on cluster profiling and the tool for
>> creating a
>> > > >>>>>>>>>>> performance report. [1] I have prepared PoC based on
>> > > >> performance
>> > > >>>>>>>>>>> logging to a separate category of Ignite log. The report
>> > > >>>>>> contains:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> - Cache operations and its distribution by types [2]
>> > > >>>>>>>>>>> - Transactions and histogram of durations [3]
>> > > >>>>>>>>>>> - SQL and Scan query statistics, top of slowest queries,
>> > > >> logical
>> > > >>>>>> and
>> > > >>>>>>>>>>> physical reads by query [4]
>> > > >>>>>>>>>>> - Compute statistics, top of slowest tasks and their jobs
>> [5]
>> > > >>>>>>>>>>> Soon I will add:
>> > > >>>>>>>>>>> - Topology and Ignite versions info
>> > > >>>>>>>>>>> - Client ID in case of operations from clients
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> For now, I'm developing a binary logging format to reduce
>> the
>> > > >>>>>> effect
>> > > >>>>>>>>>>> on performance. I'll try to reuse Ignite mechanisms.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I would like to hear suggestions for the profiling and the
>> > > >>>>>>>> performance
>> > > >>>>>>>>>>> report.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> [1]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>> > > >>>>>>>>>>> [2]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
>> > > >>>>>>>>>>> [3]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
>> > > >>>>>>>>>>> [4]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
>> > > >>>>>>>>>>> [5]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> --
>> > > >>>>>>>>>>> Best wishes,
>> > > >>>>>>>>>>> Amelchev Nikita
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>> --
>> > > >>>>>>>> Best wishes,
>> > > >>>>>>>> Amelchev Nikita
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> --
>> > > >>>>>> Best wishes,
>> > > >>>>>> Amelchev Nikita
>> > > >>>>>>
>> > > >>>>
>> > > >>>>
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Best wishes,
>> > > >> Amelchev Nikita
>> > > >>
>> > >
>> > >
>>
>>
>>
>> --
>> Best wishes,
>> Amelchev Nikita
>>
>