Tool for performance statistics reports

[Ivan Daschinsky]

According to [1], it's absolutely OK to include files under MIT licence.
But we should mention these components or modules in LICENCE file.


[1] -- https://www.apache.org/legal/resolved.html

вс, 21 июн. 2020 г. в 23:34, Saikat Maitra <[hidden email]>:

> Hi Nikita,
>
> I have reviewed the PR and shared comments. I also had a question on
> including files with a MIT license.
>
> Are we ok to use both Apache and MIT licenses in our source files?
>
> Regards,
> Saikat
>
> On Fri, Jun 12, 2020 at 8:45 PM Saikat Maitra <[hidden email]>
> wrote:
>
> > Hello Nikita,
> >
> > Thank you for sharing the information, it is very helpful.
> >
> > Regards,
> > Saikat
> >
> > On Thu, Jun 11, 2020 at 3:20 AM Nikita Amelchev <[hidden email]>
> > wrote:
> >
> >> Hello,
> >>
> >> > Can you please share more info on how we can use the profiling tool
> with
> >> > ignite-extensions modules?
> >>
> >> I have updated the module readme file. You can find instructions
> >> there. For now, I am working on review fixes and marked PR as a draft.
> >>
> >> чт, 11 июн. 2020 г. в 03:21, Saikat Maitra <[hidden email]>:
> >> >
> >> > Hello Nikita,
> >> >
> >> > I observed we have an open PR in ignite repo for this feature with
> >> > different set of changes compared to ignite extensions repo.
> >> >
> >> >
> >> > apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>
> >> >
> >> > https://github.com/apache/ignite-extensions/pull/16
> >> >
> >> > Can you please share more info on how we can use the profiling tool
> with
> >> > ignite-extensions modules?
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Saikat
> >> >
> >> > On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <[hidden email]>
> >> wrote:
> >> >
> >> > > Hello, Alexey.
> >> > >
> >> > > Thanks for the review.
> >> > >
> >> > > My understanding if the following:
> >> > >
> >> > > We will have 3 in-depth tool to find issues in cluster:
> >> > >
> >> > > 1. Metrics + System views - data that describe Ignite entities very
> >> > > high-level.
> >> > >
> >> > > 2. Profiling - tool to know what specific query of transactions are
> >> slow.
> >> > > In many cases, this knowledge is enough to fix the issue(rewrite
> >> query,
> >> > > redesign transactions flow, etc)
> >> > >
> >> > > 3. Tracing - tool to know why one of 1000 of the same queries was
> >> slow.
> >> > > The most detailed view of the Ignite internal processes.
> >> > >
> >> > > > For example, a user would not be able to match a long task with a
> >> long
> >> > > job in that task.
> >> > >
> >> > > This is not true.
> >> > > Profiling report will aggregate data from all nodes.
> >> > > So there will be both
> >> > >
> >> > >  * summary time of the task
> >> > >  * time of the each job in the task.
> >> > >
> >> > >
> >> > > > 8 июня 2020 г., в 12:52, Alexey Goncharuk <
> >> [hidden email]>
> >> > > написал(а):
> >> > > >
> >> > > > Nikita, Igniters,
> >> > > >
> >> > > > I left a few comments on the tool itself in the PR.
> >> > > >
> >> > > > However, I would like to reiterate and discuss why a user would
> >> prefer to
> >> > > > use the profiling tool over tracing? Profiling tool only captures
> >> very
> >> > > > high-level details of the operations (a single cache operation,
> for
> >> > > > example), and does not interconnect operations happened on
> different
> >> > > nodes.
> >> > > > For example, a user would not be able to match a long task with a
> >> long
> >> > > job
> >> > > > in that task. In other words, profiling data is always a subset of
> >> data
> >> > > > collected by tracing.
> >> > > >
> >> > > > Maybe it makes sense to adopt local log file approach to write
> >> spans so
> >> > > we
> >> > > > can process those spans later to build a report?
> >> > > >
> >> > > > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <[hidden email]
> >:
> >> > > >
> >> > > >> Hi, Igniters.
> >> > > >>
> >> > > >> I have implemented cluster profiling and tool to build the
> >> performance
> >> > > >> report. It's ready to be reviewed. [1, 2]
> >> > > >>
> >> > > >> Profiling can be managed by JMX bean. I have plans to implement
> it
> >> to
> >> > > >> control.sh also.
> >> > > >>
> >> > > >> Nodes write statistics to the temporary off heap buffer and then
> >> one
> >> > > >> thread flushes to the profiling files. The write mechanics and
> >> format
> >> > > >> is like WAL logging.
> >> > > >> The report contains the following statistics:
> >> > > >> - nodes and caches info
> >> > > >> - cache operations and transaction statistics
> >> > > >> - SQL and scan queries statistics (include logical and physical
> >> reads
> >> > > per
> >> > > >> query)
> >> > > >> - tasks and jobs statistics.
> >> > > >>
> >> > > >> More details in the IEP [3].
> >> > > >>
> >> > > >> [1] https://github.com/apache/ignite/pull/7693
> >> > > >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> >> > > >> [3]
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >> > > >>
> >> > > >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <
> >> > > [hidden email]>:
> >> > > >>>
> >> > > >>> Hello Nikolay,
> >> > > >>>
> >> > > >>>> Who deprecated visor and when? Maybe I miss something?
> >> > > >>> On the one hand, there was technically no community consensus
> >> that this
> >> > > >>> tool should be obsolete.
> >> > > >>> On the other hand, my opinion based on the following topic:
> >> > > >>>
> >> > > >>
> >> > >
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> >> > > >>> Moreover, it seems to me, currently, the control utility is
> >> widely used
> >> > > >> and
> >> > > >>> actively developed, instead of the visor.
> >> > > >>>
> >> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy" I
> >> think
> >> > > >> it's
> >> > > >>> a big issue from the user's point of view.
> >> > > >>> I absolutely agree with that.
> >> > > >>>
> >> > > >>>> We should solve it in the nearest time. Feel free to start this
> >> > > >> activity
> >> > > >>> I have no plan at the moment. However, at the first stage, we
> >> could
> >> > > >>> understand the difference between visor and control utility.
> >> > > >>> All useful features from visor should be moved/implemented in
> >> control
> >> > > >>> utility and after that visor tool and should be marked as
> >> > > >>> deprecated/obsoleted.
> >> > > >>>
> >> > > >>>> You need to throw in control.sh also, which does some kind of
> >> > > >> statistics
> >> > > >>> too, such as idle_verify.
> >> > > >>>> Please, clarify your idea:
> >> > > >>>>   We should use some info from control.sh to the report?
> >> > > >>>>   The report should be generated by some control.sh subcommand?
> >> > > >>> If I am not mistaken, the oracle database has AWR tool
> (mentioned
> >> on
> >> > > the
> >> > > >>> IEP page) which is a command-line utility that generates HTML
> >> reports.
> >> > > >>> I like this idea and I think this is a good approach that can be
> >> > > realized
> >> > > >>> in the control utility.
> >> > > >>> If we have a case that cannot be implemented in this way, we
> have
> >> to
> >> > > >>> clearly states the difference between these tools so as not to
> >> confuse
> >> > > >> our
> >> > > >>> users.
> >> > > >>> What do you think?
> >> > > >>>
> >> > > >>> Thanks,
> >> > > >>> Slava.
> >> > > >>>
> >> > > >>>
> >> > > >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <
> [hidden email]
> >> >:
> >> > > >>>
> >> > > >>>> Hello, Slava, Ilya, Denis.
> >> > > >>>>
> >> > > >>>> Thanks for joining this discussion!
> >> > > >>>>
> >> > > >>>>> - visor (which is deprecated)
> >> > > >>>>
> >> > > >>>> Who deprecated visor and when?
> >> > > >>>> Maybe I miss something?
> >> > > >>>>
> >> > > >>>>> - web-console (to be honest, I don't quite understand the
> >> status of
> >> > > >> this
> >> > > >>>> tool)
> >> > > >>>>
> >> > > >>>> +1.
> >> > > >>>>
> >> > > >>>>> I am not against the new tool, I just want to understand the
> >> > > >> motivation
> >> > > >>>> to not improve the existing sub-projects.
> >> > > >>>>
> >> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
> >> > > >>>> I think it's a big issue from the user's point of view.
> >> > > >>>> We should solve it in the nearest time.
> >> > > >>>> Feel free to start this activity.
> >> > > >>>>
> >> > > >>>>> - new ignite-profiling (which is a monitoring tool as well,
> >> judging
> >> > > >> by
> >> > > >>>> the provided link [1] )
> >> > > >>>>
> >> > > >>>> The general idea is the following:
> >> > > >>>>
> >> > > >>>> 1. We should have some profiling mechanism that will generate a
> >> > > >> node-local
> >> > > >>>> event log
> >> > > >>>> 2. We should have a tool that can export events to some
> >> third-party
> >> > > >>>> system. This system can be an Elastic Search(Kibana) or Ignite
> >> > > >> performance
> >> > > >>>> report or Kafka log, whatever.
> >> > > >>>> 3. Ignite performance report, in the first release, should be a
> >> > > >> "static"
> >> > > >>>> tool.
> >> > > >>>>    This means we take static logs(that is not rewritten in the
> >> > > >> analysis
> >> > > >>>> time) and feed them in the script(or tool or control.sh,
> >> whatever)
> >> > > >>>>    The script produces static report that can be used for
> overall
> >> > > >>>> performance analysis.
> >> > > >>>>
> >> > > >>>> The primary users of this report is a developer of Ignite based
> >> > > >>>> applications and performance engineers.
> >> > > >>>>
> >> > > >>>> Ilya,
> >> > > >>>>
> >> > > >>>>> You need to throw in control.sh also, which does some kind of
> >> > > >> statistics
> >> > > >>>> too, such as idle_verify.
> >> > > >>>>
> >> > > >>>> Please, clarify your idea:
> >> > > >>>>    We should use some info from control.sh to the report?
> >> > > >>>>    The report should be generated by some control.sh
> subcommand?
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> Denis,
> >> > > >>>>
> >> > > >>>>> Speaking of the probes/statistics collection approach, is it
> >> > > >> supposed to
> >> > > >>>> reuse tracing capabilities that are to be added as part of
> >> IEP-35?
> >> > > >>>>
> >> > > >>>> For now, we don't have any results of tracing development
> >> available in
> >> > > >>>> Apache Ignite.
> >> > > >>>> Hopefully, we got some in a couple of weeks.
> >> > > >>>> After it, we can start a discussion of how to merge two
> >> improvements.
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <[hidden email]>
> >> > > >> написал(а):
> >> > > >>>>>
> >> > > >>>>>>
> >> > > >>>>>> Tracing is more deeply takes statistics. If it will be
> >> possible,
> >> > > >> I'm for
> >> > > >>>>>> reuse.
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>> Looks like we need to sync up on these activities/initiatives
> to
> >> > > >> ensure
> >> > > >>>> we
> >> > > >>>>> don't do a duplicate job. If you think a separate discussion
> is
> >> > > >> necessary
> >> > > >>>>> let's kick it off.
> >> > > >>>>>
> >> > > >>>>> -
> >> > > >>>>> Denis
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
> >> > > >> [hidden email]>
> >> > > >>>>> wrote:
> >> > > >>>>>
> >> > > >>>>>> Denis, Ilya,
> >> > > >>>>>>
> >> > > >>>>>> I will try to integrate profiling functionality into
> control.sh
> >> > > >> utility.
> >> > > >>>>>>
> >> > > >>>>>>> Speaking of the probes/statistics collection approach, is it
> >> > > >> supposed
> >> > > >>>> to
> >> > > >>>>>>> reuse tracing capabilities that are to be added as part of
> >> IEP-35?
> >> > > >>>>>> Tracing is more deeply takes statistics. If it will be
> >> possible,
> >> > > >> I'm for
> >> > > >>>>>> reuse.
> >> > > >>>>>>
> >> > > >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
> >> > > >> [hidden email]
> >> > > >>>>> :
> >> > > >>>>>>>
> >> > > >>>>>>> Hello!
> >> > > >>>>>>>
> >> > > >>>>>>> I suggest that it's one of the places where it could be put
> >> > > >> instead of
> >> > > >>>>>>> adding a new tool.
> >> > > >>>>>>>
> >> > > >>>>>>> Regards,
> >> > > >>>>>>> --
> >> > > >>>>>>> Ilya Kasnacheev
> >> > > >>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <
> >> [hidden email]
> >> > > >>> :
> >> > > >>>>>>>
> >> > > >>>>>>>> Ilya,
> >> > > >>>>>>>>
> >> > > >>>>>>>> You suggest using control.sh to build the report?
> >> > > >>>>>>>>
> >> > > >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
> >> > > >>>>>> [hidden email]>:
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> Hello!
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> You need to throw in control.sh also, which does some kind
> >> of
> >> > > >>>>>> statistics
> >> > > >>>>>>>>> too, such as idle_verify.
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> Regards,
> >> > > >>>>>>>>> --
> >> > > >>>>>>>>> Ilya Kasnacheev
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
> >> > > >>>>>> [hidden email]
> >> > > >>>>>>>>> :
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>> Hello Nikita,
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> Perhaps, I am missing something...
> >> > > >>>>>>>>>> Apache Ignite already has a web-console tool. Do we want
> to
> >> > > >>>>>> improve the
> >> > > >>>>>>>>>> existing tool instead of creating a new one?
> >> > > >>>>>>>>>> It seems to me, this can be confusing for users.
> >> > > >>>>>>>>>> - visor (which is deprecated)
> >> > > >>>>>>>>>> - web-console (to be honest, I don't quite understand the
> >> status
> >> > > >>>>>> of
> >> > > >>>>>>>> this
> >> > > >>>>>>>>>> tool)
> >> > > >>>>>>>>>> - new ignite-profiling (which is a monitoring tool as
> well,
> >> > > >>>>>> judging
> >> > > >>>>>>>> by the
> >> > > >>>>>>>>>> provided link [1] )
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> I am not against the new tool, I just want to understand
> >> the
> >> > > >>>>>>>> motivation to
> >> > > >>>>>>>>>> not improve the existing sub-projects.
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> [1]
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> Thanks,
> >> > > >>>>>>>>>> S.
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <
> >> > > >> [hidden email]
> >> > > >>>>>>> :
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>> Hi, Igniters.
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> I'm working on cluster profiling and the tool for
> >> creating a
> >> > > >>>>>>>>>>> performance report. [1] I have prepared PoC based on
> >> > > >> performance
> >> > > >>>>>>>>>>> logging to a separate category of Ignite log. The report
> >> > > >>>>>> contains:
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> - Cache operations and its distribution by types [2]
> >> > > >>>>>>>>>>> - Transactions and histogram of durations [3]
> >> > > >>>>>>>>>>> - SQL and Scan query statistics, top of slowest queries,
> >> > > >> logical
> >> > > >>>>>> and
> >> > > >>>>>>>>>>> physical reads by query [4]
> >> > > >>>>>>>>>>> - Compute statistics, top of slowest tasks and their
> jobs
> >> [5]
> >> > > >>>>>>>>>>> Soon I will add:
> >> > > >>>>>>>>>>> - Topology and Ignite versions info
> >> > > >>>>>>>>>>> - Client ID in case of operations from clients
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> For now, I'm developing a binary logging format to
> reduce
> >> the
> >> > > >>>>>> effect
> >> > > >>>>>>>>>>> on performance. I'll try to reuse Ignite mechanisms.
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> I would like to hear suggestions for the profiling and
> the
> >> > > >>>>>>>> performance
> >> > > >>>>>>>>>>> report.
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> [1]
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >> > > >>>>>>>>>>> [2]
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> >> > > >>>>>>>>>>> [3]
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> >> > > >>>>>>>>>>> [4]
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> >> > > >>>>>>>>>>> [5]
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> --
> >> > > >>>>>>>>>>> Best wishes,
> >> > > >>>>>>>>>>> Amelchev Nikita
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>> --
> >> > > >>>>>>>> Best wishes,
> >> > > >>>>>>>> Amelchev Nikita
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> --
> >> > > >>>>>> Best wishes,
> >> > > >>>>>> Amelchev Nikita
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> Best wishes,
> >> > > >> Amelchev Nikita
> >> > > >>
> >> > >
> >> > >
> >>
> >>
> >>
> >> --
> >> Best wishes,
> >> Amelchev Nikita
> >>
> >
>

--
Sincerely yours, Ivan Daschinskiy

[Nikita Amelchev]

Hi, guys.

I have mentioned components under the MIT license in the LICENSE file.

Saikat, I have fixed PR according to your suggestions. Thanks for taking a look.

[Nikolay Izhikov-2]

Hello, Igniters.

I think that inside Ignite core we should name this feature as «performance statistics»
We already have «cache statistics».
Data that is collected by performance statistics can be used not only for profiling but to solve other tasks.


> 22 июня 2020 г., в 14:00, Nikita Amelchev <[hidden email]> написал(а):
>
> Hi, guys.
>
> I have mentioned components under the MIT license in the LICENSE file.
>
> Saikat, I have fixed PR according to your suggestions. Thanks for taking a look.

[Saikat Maitra]

Hi Nikita,

The changes in this PR looks good.

https://github.com/apache/ignite-extensions/pull/16

Regards,
Saikat

On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov <[hidden email]>
wrote:

> Hello, Igniters.
>
> I think that inside Ignite core we should name this feature as
> «performance statistics»
> We already have «cache statistics».
> Data that is collected by performance statistics can be used not only for
> profiling but to solve other tasks.
>
>
> > 22 июня 2020 г., в 14:00, Nikita Amelchev <[hidden email]>
> написал(а):
> >
> > Hi, guys.
> >
> > I have mentioned components under the MIT license in the LICENSE file.
> >
> > Saikat, I have fixed PR according to your suggestions. Thanks for taking
> a look.
>
>

[Nikita Amelchev]

Hello, Igniters.

I have implemented the profiling tool [1, 2]. It writes duration and
other parameters of user operations (scan, SQL query, transactions,
tasks, jobs, CQ, etc) to a local file. This info can be used in
various cases. The main goal is to build the performance report to
analyze the count and duration of user queries [3].

We already have the tracing with similar functionality but I think
Ignite should have both tools - tracing and profiling.

Because the tracing causes performance drop 52% [4] and can not be
used for collecting statistics about all queries in production
deployments. The performance drop of the profiling tool is less than
2% and it can be used in production. I have benchmarked the tracing
and got the results:

 -2% when configured OpenCensusTracingSpi and all scopes disabled
 -52% for TX scope (IgnitePutTxBenchmark)
 -58% for SQL scope  (IgniteSqlQueryBenchmark)

Such a performance drop is significant to not use the tracing in production.

I have considered the possibility to reuse the tracing API. If
statistics collecting will be implemented with the TracingSpi then we
get a performance drop due to:
 - Transferring tracing context over the network.
 - Using ThreadLocal for spans
 - Converting primitives and objects to string and vice versa. (API
supports only String-based tags and values)
 - Generating span objects

I have benchmarked implementations on the yardstick’s
IgniteGetBenchmark. The tracing context transferring over the network
was disabled. The results:
 - Tracing API implementation - 8% performance drop.
 - Proposed implementation - 2% performance drop.

I think this is a significant drop and implementation with reuse
tracing API should not be used. The cluster profiling should have as
little performance drop as possible to be used in production. The
tracing will be used for the detailed investigation.

WDYT?

The tool is ready to be reviewed [3, 5].

[1] https://issues.apache.org/jira/browse/IGNITE-12666
[2] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
[3] https://github.com/apache/ignite-extensions/pull/16
[4] https://issues.apache.org/jira/secure/attachment/13016636/Tracing%20benchmarks.docx
[5] https://github.com/apache/ignite/pull/7693

ср, 24 июн. 2020 г. в 23:31, Saikat Maitra <[hidden email]>:

>
> Hi Nikita,
>
> The changes in this PR looks good.
>
> https://github.com/apache/ignite-extensions/pull/16
>
> Regards,
> Saikat
>
> On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov <[hidden email]>
> wrote:
>
> > Hello, Igniters.
> >
> > I think that inside Ignite core we should name this feature as
> > «performance statistics»
> > We already have «cache statistics».
> > Data that is collected by performance statistics can be used not only for
> > profiling but to solve other tasks.
> >
> >
> > > 22 июня 2020 г., в 14:00, Nikita Amelchev <[hidden email]>
> > написал(а):
> > >
> > > Hi, guys.
> > >
> > > I have mentioned components under the MIT license in the LICENSE file.
> > >
> > > Saikat, I have fixed PR according to your suggestions. Thanks for taking
> > a look.
> >
> >

--
Best wishes,
Amelchev Nikita

[Nikolay Izhikov-2]

Hello, Nikita.

Makes sense.

I will take a look.

> 7 дек. 2020 г., в 15:29, Nikita Amelchev <[hidden email]> написал(а):
>
> Hello, Igniters.
>
> I have implemented the profiling tool [1, 2]. It writes duration and
> other parameters of user operations (scan, SQL query, transactions,
> tasks, jobs, CQ, etc) to a local file. This info can be used in
> various cases. The main goal is to build the performance report to
> analyze the count and duration of user queries [3].
>
> We already have the tracing with similar functionality but I think
> Ignite should have both tools - tracing and profiling.
>
> Because the tracing causes performance drop 52% [4] and can not be
> used for collecting statistics about all queries in production
> deployments. The performance drop of the profiling tool is less than
> 2% and it can be used in production. I have benchmarked the tracing
> and got the results:
>
> -2% when configured OpenCensusTracingSpi and all scopes disabled
> -52% for TX scope (IgnitePutTxBenchmark)
> -58% for SQL scope  (IgniteSqlQueryBenchmark)
>
> Such a performance drop is significant to not use the tracing in production.
>
> I have considered the possibility to reuse the tracing API. If
> statistics collecting will be implemented with the TracingSpi then we
> get a performance drop due to:
> - Transferring tracing context over the network.
> - Using ThreadLocal for spans
> - Converting primitives and objects to string and vice versa. (API
> supports only String-based tags and values)
> - Generating span objects
>
> I have benchmarked implementations on the yardstick’s
> IgniteGetBenchmark. The tracing context transferring over the network
> was disabled. The results:
> - Tracing API implementation - 8% performance drop.
> - Proposed implementation - 2% performance drop.
>
> I think this is a significant drop and implementation with reuse
> tracing API should not be used. The cluster profiling should have as
> little performance drop as possible to be used in production. The
> tracing will be used for the detailed investigation.
>
> WDYT?
>
> The tool is ready to be reviewed [3, 5].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12666
> [2] https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> [3] https://github.com/apache/ignite-extensions/pull/16
> [4] https://issues.apache.org/jira/secure/attachment/13016636/Tracing%20benchmarks.docx
> [5] https://github.com/apache/ignite/pull/7693
>
> ср, 24 июн. 2020 г. в 23:31, Saikat Maitra <[hidden email]>:
>>
>> Hi Nikita,
>>
>> The changes in this PR looks good.
>>
>> https://github.com/apache/ignite-extensions/pull/16
>>
>> Regards,
>> Saikat
>>
>> On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov <[hidden email]>
>> wrote:
>>
>>> Hello, Igniters.
>>>
>>> I think that inside Ignite core we should name this feature as
>>> «performance statistics»
>>> We already have «cache statistics».
>>> Data that is collected by performance statistics can be used not only for
>>> profiling but to solve other tasks.
>>>
>>>
>>>> 22 июня 2020 г., в 14:00, Nikita Amelchev <[hidden email]>
>>> написал(а):
>>>>
>>>> Hi, guys.
>>>>
>>>> I have mentioned components under the MIT license in the LICENSE file.
>>>>
>>>> Saikat, I have fixed PR according to your suggestions. Thanks for taking
>>> a look.
>>>
>>>
>
>
>
> --
> Best wishes,
> Amelchev Nikita

[Alexander Lapin]

Hello Igniters,

Because the tracing causes performance drop 52% [4] and can not be

> used for collecting statistics about all queries in production
> deployments. The performance drop of the profiling tool is less than
> 2% and it can be used in production. I have benchmarked the tracing
> and got the results:
>
>  -2% when configured OpenCensusTracingSpi and all scopes disabled
>  -52% for TX scope (IgnitePutTxBenchmark)
>  -58% for SQL scope  (IgniteSqlQueryBenchmark)
>
> Such a performance drop is significant to not use the tracing in
> production.
>

We've rerun tracing benchmarks based on more realistic scenarios and got a
10-15% performance drop in case of sampling-rate 1 (all transactions were
traced). More realistic scenarios means that we don't test tracing
performance if the system is in overdraft state but add some sort of micro
throttling (1 millisecond) between operations, transactions in our case.
*IgnitePutTxBenchmark*

Green: Case 1: NoopTracingSpi

Blue: Case 2: OpenCensusTracingSpi (disabled)

Red: Case 3: OpenCensusTracingSpi, --scope TX --sampling-rate 0.1

Black: Case 5: *ControlCenter* + OpenCensusTracingSpi, --scope TX
--sampling-rate 0.1

Violet: Case 4: OpenCensusTracingSpi, --scope TX --sampling-rate 1
Yellow: Case 6: ControlCenter + OpenCensusTracingSpi, --scope TX
--sampling-rate

I have considered the possibility to reuse the tracing API. If
> statistics collecting will be implemented with the TracingSpi then we
> get a performance drop due to:
>  - Transferring tracing context over the network.
>  - Using ThreadLocal for spans
>  - Converting primitives and objects to string and vice versa. (API
> supports only String-based tags and values)
>  - Generating span objects
>
@Nikita Amelchev Could you please share numbers?

Best regards,
Alexander

пн, 7 дек. 2020 г. в 17:24, Nikolay Izhikov <[hidden email]>:

> Hello, Nikita.
>
> Makes sense.
>
> I will take a look.
>
> > 7 дек. 2020 г., в 15:29, Nikita Amelchev <[hidden email]>
> написал(а):
> >
> > Hello, Igniters.
> >
> > I have implemented the profiling tool [1, 2]. It writes duration and
> > other parameters of user operations (scan, SQL query, transactions,
> > tasks, jobs, CQ, etc) to a local file. This info can be used in
> > various cases. The main goal is to build the performance report to
> > analyze the count and duration of user queries [3].
> >
> > We already have the tracing with similar functionality but I think
> > Ignite should have both tools - tracing and profiling.
> >
> > Because the tracing causes performance drop 52% [4] and can not be
> > used for collecting statistics about all queries in production
> > deployments. The performance drop of the profiling tool is less than
> > 2% and it can be used in production. I have benchmarked the tracing
> > and got the results:
> >
> > -2% when configured OpenCensusTracingSpi and all scopes disabled
> > -52% for TX scope (IgnitePutTxBenchmark)
> > -58% for SQL scope  (IgniteSqlQueryBenchmark)
> >
> > Such a performance drop is significant to not use the tracing in
> production.
> >
> > I have considered the possibility to reuse the tracing API. If
> > statistics collecting will be implemented with the TracingSpi then we
> > get a performance drop due to:
> > - Transferring tracing context over the network.
> > - Using ThreadLocal for spans
> > - Converting primitives and objects to string and vice versa. (API
> > supports only String-based tags and values)
> > - Generating span objects
> >
> > I have benchmarked implementations on the yardstick’s
> > IgniteGetBenchmark. The tracing context transferring over the network
> > was disabled. The results:
> > - Tracing API implementation - 8% performance drop.
> > - Proposed implementation - 2% performance drop.
> >
> > I think this is a significant drop and implementation with reuse
> > tracing API should not be used. The cluster profiling should have as
> > little performance drop as possible to be used in production. The
> > tracing will be used for the detailed investigation.
> >
> > WDYT?
> >
> > The tool is ready to be reviewed [3, 5].
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12666
> > [2]
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > [3] https://github.com/apache/ignite-extensions/pull/16
> > [4]
> https://issues.apache.org/jira/secure/attachment/13016636/Tracing%20benchmarks.docx
> > [5] https://github.com/apache/ignite/pull/7693
> >
> > ср, 24 июн. 2020 г. в 23:31, Saikat Maitra <[hidden email]>:
> >>
> >> Hi Nikita,
> >>
> >> The changes in this PR looks good.
> >>
> >> https://github.com/apache/ignite-extensions/pull/16
> >>
> >> Regards,
> >> Saikat
> >>
> >> On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov <[hidden email]>
> >> wrote:
> >>
> >>> Hello, Igniters.
> >>>
> >>> I think that inside Ignite core we should name this feature as
> >>> «performance statistics»
> >>> We already have «cache statistics».
> >>> Data that is collected by performance statistics can be used not only
> for
> >>> profiling but to solve other tasks.
> >>>
> >>>
> >>>> 22 июня 2020 г., в 14:00, Nikita Amelchev <[hidden email]>
> >>> написал(а):
> >>>>
> >>>> Hi, guys.
> >>>>
> >>>> I have mentioned components under the MIT license in the LICENSE file.
> >>>>
> >>>> Saikat, I have fixed PR according to your suggestions. Thanks for
> taking
> >>> a look.
> >>>
> >>>
> >
> >
> >
> > --
> > Best wishes,
> > Amelchev Nikita
>
>

[Nikolay Izhikov-2]

Hello, Igniters.

We discussed this feature privately with Alexander and Nikita.
Here are the results we want to share with the community:

0. In the end, both, performance statistic tool and tracing should use the same API.
1. We should improve the Tracing API, so it able to be used for gathering information about all operations without a significant performance drop.

I propose to go as follows:

1. Merge current PR as is after final review. My intention is to provide a tool for users that can be used in the real-world production environment.
2. Improve the Tracing API.
3. Combine both tools under the same API.

> 14 дек. 2020 г., в 10:42, Alexander Lapin <[hidden email]> написал(а):
>
> Hello Igniters,
>
> Because the tracing causes performance drop 52% [4] and can not be
>> used for collecting statistics about all queries in production
>> deployments. The performance drop of the profiling tool is less than
>> 2% and it can be used in production. I have benchmarked the tracing
>> and got the results:
>>
>> -2% when configured OpenCensusTracingSpi and all scopes disabled
>> -52% for TX scope (IgnitePutTxBenchmark)
>> -58% for SQL scope  (IgniteSqlQueryBenchmark)
>>
>> Such a performance drop is significant to not use the tracing in
>> production.
>>
> We've rerun tracing benchmarks based on more realistic scenarios and got a
> 10-15% performance drop in case of sampling-rate 1 (all transactions were
> traced). More realistic scenarios means that we don't test tracing
> performance if the system is in overdraft state but add some sort of micro
> throttling (1 millisecond) between operations, transactions in our case.
> *IgnitePutTxBenchmark*
>
> Green: Case 1: NoopTracingSpi
>
> Blue: Case 2: OpenCensusTracingSpi (disabled)
>
> Red: Case 3: OpenCensusTracingSpi, --scope TX --sampling-rate 0.1
>
> Black: Case 5: *ControlCenter* + OpenCensusTracingSpi, --scope TX
> --sampling-rate 0.1
>
> Violet: Case 4: OpenCensusTracingSpi, --scope TX --sampling-rate 1
> Yellow: Case 6: ControlCenter + OpenCensusTracingSpi, --scope TX
> --sampling-rate
>
> I have considered the possibility to reuse the tracing API. If
>> statistics collecting will be implemented with the TracingSpi then we
>> get a performance drop due to:
>> - Transferring tracing context over the network.
>> - Using ThreadLocal for spans
>> - Converting primitives and objects to string and vice versa. (API
>> supports only String-based tags and values)
>> - Generating span objects
>>
> @Nikita Amelchev Could you please share numbers?
>
> Best regards,
> Alexander
>
> пн, 7 дек. 2020 г. в 17:24, Nikolay Izhikov <[hidden email]>:
>
>> Hello, Nikita.
>>
>> Makes sense.
>>
>> I will take a look.
>>
>>> 7 дек. 2020 г., в 15:29, Nikita Amelchev <[hidden email]>
>> написал(а):
>>>
>>> Hello, Igniters.
>>>
>>> I have implemented the profiling tool [1, 2]. It writes duration and
>>> other parameters of user operations (scan, SQL query, transactions,
>>> tasks, jobs, CQ, etc) to a local file. This info can be used in
>>> various cases. The main goal is to build the performance report to
>>> analyze the count and duration of user queries [3].
>>>
>>> We already have the tracing with similar functionality but I think
>>> Ignite should have both tools - tracing and profiling.
>>>
>>> Because the tracing causes performance drop 52% [4] and can not be
>>> used for collecting statistics about all queries in production
>>> deployments. The performance drop of the profiling tool is less than
>>> 2% and it can be used in production. I have benchmarked the tracing
>>> and got the results:
>>>
>>> -2% when configured OpenCensusTracingSpi and all scopes disabled
>>> -52% for TX scope (IgnitePutTxBenchmark)
>>> -58% for SQL scope  (IgniteSqlQueryBenchmark)
>>>
>>> Such a performance drop is significant to not use the tracing in
>> production.
>>>
>>> I have considered the possibility to reuse the tracing API. If
>>> statistics collecting will be implemented with the TracingSpi then we
>>> get a performance drop due to:
>>> - Transferring tracing context over the network.
>>> - Using ThreadLocal for spans
>>> - Converting primitives and objects to string and vice versa. (API
>>> supports only String-based tags and values)
>>> - Generating span objects
>>>
>>> I have benchmarked implementations on the yardstick’s
>>> IgniteGetBenchmark. The tracing context transferring over the network
>>> was disabled. The results:
>>> - Tracing API implementation - 8% performance drop.
>>> - Proposed implementation - 2% performance drop.
>>>
>>> I think this is a significant drop and implementation with reuse
>>> tracing API should not be used. The cluster profiling should have as
>>> little performance drop as possible to be used in production. The
>>> tracing will be used for the detailed investigation.
>>>
>>> WDYT?
>>>
>>> The tool is ready to be reviewed [3, 5].
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-12666
>>> [2]
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>>> [3] https://github.com/apache/ignite-extensions/pull/16
>>> [4]
>> https://issues.apache.org/jira/secure/attachment/13016636/Tracing%20benchmarks.docx
>>> [5] https://github.com/apache/ignite/pull/7693
>>>
>>> ср, 24 июн. 2020 г. в 23:31, Saikat Maitra <[hidden email]>:
>>>>
>>>> Hi Nikita,
>>>>
>>>> The changes in this PR looks good.
>>>>
>>>> https://github.com/apache/ignite-extensions/pull/16
>>>>
>>>> Regards,
>>>> Saikat
>>>>
>>>> On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hello, Igniters.
>>>>>
>>>>> I think that inside Ignite core we should name this feature as
>>>>> «performance statistics»
>>>>> We already have «cache statistics».
>>>>> Data that is collected by performance statistics can be used not only
>> for
>>>>> profiling but to solve other tasks.
>>>>>
>>>>>
>>>>>> 22 июня 2020 г., в 14:00, Nikita Amelchev <[hidden email]>
>>>>> написал(а):
>>>>>>
>>>>>> Hi, guys.
>>>>>>
>>>>>> I have mentioned components under the MIT license in the LICENSE file.
>>>>>>
>>>>>> Saikat, I have fixed PR according to your suggestions. Thanks for
>> taking
>>>>> a look.
>>>>>
>>>>>
>>>
>>>
>>>
>>> --
>>> Best wishes,
>>> Amelchev Nikita
>>
>>

[Alexander Lapin]

Ok from my side.
Few more details about tracing spi updates, based on mentioned above
discussion with Nikolay and Nikita.

Tracing provides enough data for a performance profiling tool, actually
only root spans are required. However, according to Nikinta,
root-span-tracing has a 7-8% performance drop in comparison to 1-2%
performance drop of the performance profiling tool. It's the main reason to
have given tool as is right now. In order to reuse TracingSPI for a
profiling tool internals, few modifications should be made to increase
tracing performance:

   - Add support for non-strings tags and log points: primitives, etc.
   - Add ability to postpone adding span tags and log points to the very
   and of span tree creation.
   - Probably some sort of tags caching could also help.

Best regards,
Alexander

пн, 14 дек. 2020 г. в 12:48, Nikolay Izhikov <[hidden email]>:

> Hello, Igniters.
>
> We discussed this feature privately with Alexander and Nikita.
> Here are the results we want to share with the community:
>
> 0. In the end, both, performance statistic tool and tracing should use the
> same API.
> 1. We should improve the Tracing API, so it able to be used for gathering
> information about all operations without a significant performance drop.
>
> I propose to go as follows:
>
> 1. Merge current PR as is after final review. My intention is to provide a
> tool for users that can be used in the real-world production environment.
> 2. Improve the Tracing API.
> 3. Combine both tools under the same API.
>
> > 14 дек. 2020 г., в 10:42, Alexander Lapin <[hidden email]>
> написал(а):
> >
> > Hello Igniters,
> >
> > Because the tracing causes performance drop 52% [4] and can not be
> >> used for collecting statistics about all queries in production
> >> deployments. The performance drop of the profiling tool is less than
> >> 2% and it can be used in production. I have benchmarked the tracing
> >> and got the results:
> >>
> >> -2% when configured OpenCensusTracingSpi and all scopes disabled
> >> -52% for TX scope (IgnitePutTxBenchmark)
> >> -58% for SQL scope  (IgniteSqlQueryBenchmark)
> >>
> >> Such a performance drop is significant to not use the tracing in
> >> production.
> >>
> > We've rerun tracing benchmarks based on more realistic scenarios and got
> a
> > 10-15% performance drop in case of sampling-rate 1 (all transactions were
> > traced). More realistic scenarios means that we don't test tracing
> > performance if the system is in overdraft state but add some sort of
> micro
> > throttling (1 millisecond) between operations, transactions in our case.
> > *IgnitePutTxBenchmark*
> >
> > Green: Case 1: NoopTracingSpi
> >
> > Blue: Case 2: OpenCensusTracingSpi (disabled)
> >
> > Red: Case 3: OpenCensusTracingSpi, --scope TX --sampling-rate 0.1
> >
> > Black: Case 5: *ControlCenter* + OpenCensusTracingSpi, --scope TX
> > --sampling-rate 0.1
> >
> > Violet: Case 4: OpenCensusTracingSpi, --scope TX --sampling-rate 1
> > Yellow: Case 6: ControlCenter + OpenCensusTracingSpi, --scope TX
> > --sampling-rate
> >
> > I have considered the possibility to reuse the tracing API. If
> >> statistics collecting will be implemented with the TracingSpi then we
> >> get a performance drop due to:
> >> - Transferring tracing context over the network.
> >> - Using ThreadLocal for spans
> >> - Converting primitives and objects to string and vice versa. (API
> >> supports only String-based tags and values)
> >> - Generating span objects
> >>
> > @Nikita Amelchev Could you please share numbers?
> >
> > Best regards,
> > Alexander
> >
> > пн, 7 дек. 2020 г. в 17:24, Nikolay Izhikov <[hidden email]>:
> >
> >> Hello, Nikita.
> >>
> >> Makes sense.
> >>
> >> I will take a look.
> >>
> >>> 7 дек. 2020 г., в 15:29, Nikita Amelchev <[hidden email]>
> >> написал(а):
> >>>
> >>> Hello, Igniters.
> >>>
> >>> I have implemented the profiling tool [1, 2]. It writes duration and
> >>> other parameters of user operations (scan, SQL query, transactions,
> >>> tasks, jobs, CQ, etc) to a local file. This info can be used in
> >>> various cases. The main goal is to build the performance report to
> >>> analyze the count and duration of user queries [3].
> >>>
> >>> We already have the tracing with similar functionality but I think
> >>> Ignite should have both tools - tracing and profiling.
> >>>
> >>> Because the tracing causes performance drop 52% [4] and can not be
> >>> used for collecting statistics about all queries in production
> >>> deployments. The performance drop of the profiling tool is less than
> >>> 2% and it can be used in production. I have benchmarked the tracing
> >>> and got the results:
> >>>
> >>> -2% when configured OpenCensusTracingSpi and all scopes disabled
> >>> -52% for TX scope (IgnitePutTxBenchmark)
> >>> -58% for SQL scope  (IgniteSqlQueryBenchmark)
> >>>
> >>> Such a performance drop is significant to not use the tracing in
> >> production.
> >>>
> >>> I have considered the possibility to reuse the tracing API. If
> >>> statistics collecting will be implemented with the TracingSpi then we
> >>> get a performance drop due to:
> >>> - Transferring tracing context over the network.
> >>> - Using ThreadLocal for spans
> >>> - Converting primitives and objects to string and vice versa. (API
> >>> supports only String-based tags and values)
> >>> - Generating span objects
> >>>
> >>> I have benchmarked implementations on the yardstick’s
> >>> IgniteGetBenchmark. The tracing context transferring over the network
> >>> was disabled. The results:
> >>> - Tracing API implementation - 8% performance drop.
> >>> - Proposed implementation - 2% performance drop.
> >>>
> >>> I think this is a significant drop and implementation with reuse
> >>> tracing API should not be used. The cluster profiling should have as
> >>> little performance drop as possible to be used in production. The
> >>> tracing will be used for the detailed investigation.
> >>>
> >>> WDYT?
> >>>
> >>> The tool is ready to be reviewed [3, 5].
> >>>
> >>> [1] https://issues.apache.org/jira/browse/IGNITE-12666
> >>> [2]
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>> [3] https://github.com/apache/ignite-extensions/pull/16
> >>> [4]
> >>
> https://issues.apache.org/jira/secure/attachment/13016636/Tracing%20benchmarks.docx
> >>> [5] https://github.com/apache/ignite/pull/7693
> >>>
> >>> ср, 24 июн. 2020 г. в 23:31, Saikat Maitra <[hidden email]>:
> >>>>
> >>>> Hi Nikita,
> >>>>
> >>>> The changes in this PR looks good.
> >>>>
> >>>> https://github.com/apache/ignite-extensions/pull/16
> >>>>
> >>>> Regards,
> >>>> Saikat
> >>>>
> >>>> On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov <[hidden email]
> >
> >>>> wrote:
> >>>>
> >>>>> Hello, Igniters.
> >>>>>
> >>>>> I think that inside Ignite core we should name this feature as
> >>>>> «performance statistics»
> >>>>> We already have «cache statistics».
> >>>>> Data that is collected by performance statistics can be used not only
> >> for
> >>>>> profiling but to solve other tasks.
> >>>>>
> >>>>>
> >>>>>> 22 июня 2020 г., в 14:00, Nikita Amelchev <[hidden email]>
> >>>>> написал(а):
> >>>>>>
> >>>>>> Hi, guys.
> >>>>>>
> >>>>>> I have mentioned components under the MIT license in the LICENSE
> file.
> >>>>>>
> >>>>>> Saikat, I have fixed PR according to your suggestions. Thanks for
> >> taking
> >>>>> a look.
> >>>>>
> >>>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best wishes,
> >>> Amelchev Nikita
> >>
> >>
>
>

[Nikita Amelchev]

Guys,

I have filed the ticket [1] to improve the TracingSpi interface and
further reusing it for cluster profiling. I'll assign it to me and
take it to work after profiling merge if nobody minds.

[1] https://issues.apache.org/jira/browse/IGNITE-13850

пн, 14 дек. 2020 г. в 13:38, Alexander Lapin <[hidden email]>:

>
> Ok from my side.
> Few more details about tracing spi updates, based on mentioned above
> discussion with Nikolay and Nikita.
>
> Tracing provides enough data for a performance profiling tool, actually
> only root spans are required. However, according to Nikinta,
> root-span-tracing has a 7-8% performance drop in comparison to 1-2%
> performance drop of the performance profiling tool. It's the main reason to
> have given tool as is right now. In order to reuse TracingSPI for a
> profiling tool internals, few modifications should be made to increase
> tracing performance:
>
>    - Add support for non-strings tags and log points: primitives, etc.
>    - Add ability to postpone adding span tags and log points to the very
>    and of span tree creation.
>    - Probably some sort of tags caching could also help.
>
> Best regards,
> Alexander
>
> пн, 14 дек. 2020 г. в 12:48, Nikolay Izhikov <[hidden email]>:
>
> > Hello, Igniters.
> >
> > We discussed this feature privately with Alexander and Nikita.
> > Here are the results we want to share with the community:
> >
> > 0. In the end, both, performance statistic tool and tracing should use the
> > same API.
> > 1. We should improve the Tracing API, so it able to be used for gathering
> > information about all operations without a significant performance drop.
> >
> > I propose to go as follows:
> >
> > 1. Merge current PR as is after final review. My intention is to provide a
> > tool for users that can be used in the real-world production environment.
> > 2. Improve the Tracing API.
> > 3. Combine both tools under the same API.
> >
> > > 14 дек. 2020 г., в 10:42, Alexander Lapin <[hidden email]>
> > написал(а):
> > >
> > > Hello Igniters,
> > >
> > > Because the tracing causes performance drop 52% [4] and can not be
> > >> used for collecting statistics about all queries in production
> > >> deployments. The performance drop of the profiling tool is less than
> > >> 2% and it can be used in production. I have benchmarked the tracing
> > >> and got the results:
> > >>
> > >> -2% when configured OpenCensusTracingSpi and all scopes disabled
> > >> -52% for TX scope (IgnitePutTxBenchmark)
> > >> -58% for SQL scope  (IgniteSqlQueryBenchmark)
> > >>
> > >> Such a performance drop is significant to not use the tracing in
> > >> production.
> > >>
> > > We've rerun tracing benchmarks based on more realistic scenarios and got
> > a
> > > 10-15% performance drop in case of sampling-rate 1 (all transactions were
> > > traced). More realistic scenarios means that we don't test tracing
> > > performance if the system is in overdraft state but add some sort of
> > micro
> > > throttling (1 millisecond) between operations, transactions in our case.
> > > *IgnitePutTxBenchmark*
> > >
> > > Green: Case 1: NoopTracingSpi
> > >
> > > Blue: Case 2: OpenCensusTracingSpi (disabled)
> > >
> > > Red: Case 3: OpenCensusTracingSpi, --scope TX --sampling-rate 0.1
> > >
> > > Black: Case 5: *ControlCenter* + OpenCensusTracingSpi, --scope TX
> > > --sampling-rate 0.1
> > >
> > > Violet: Case 4: OpenCensusTracingSpi, --scope TX --sampling-rate 1
> > > Yellow: Case 6: ControlCenter + OpenCensusTracingSpi, --scope TX
> > > --sampling-rate
> > >
> > > I have considered the possibility to reuse the tracing API. If
> > >> statistics collecting will be implemented with the TracingSpi then we
> > >> get a performance drop due to:
> > >> - Transferring tracing context over the network.
> > >> - Using ThreadLocal for spans
> > >> - Converting primitives and objects to string and vice versa. (API
> > >> supports only String-based tags and values)
> > >> - Generating span objects
> > >>
> > > @Nikita Amelchev Could you please share numbers?
> > >
> > > Best regards,
> > > Alexander
> > >
> > > пн, 7 дек. 2020 г. в 17:24, Nikolay Izhikov <[hidden email]>:
> > >
> > >> Hello, Nikita.
> > >>
> > >> Makes sense.
> > >>
> > >> I will take a look.
> > >>
> > >>> 7 дек. 2020 г., в 15:29, Nikita Amelchev <[hidden email]>
> > >> написал(а):
> > >>>
> > >>> Hello, Igniters.
> > >>>
> > >>> I have implemented the profiling tool [1, 2]. It writes duration and
> > >>> other parameters of user operations (scan, SQL query, transactions,
> > >>> tasks, jobs, CQ, etc) to a local file. This info can be used in
> > >>> various cases. The main goal is to build the performance report to
> > >>> analyze the count and duration of user queries [3].
> > >>>
> > >>> We already have the tracing with similar functionality but I think
> > >>> Ignite should have both tools - tracing and profiling.
> > >>>
> > >>> Because the tracing causes performance drop 52% [4] and can not be
> > >>> used for collecting statistics about all queries in production
> > >>> deployments. The performance drop of the profiling tool is less than
> > >>> 2% and it can be used in production. I have benchmarked the tracing
> > >>> and got the results:
> > >>>
> > >>> -2% when configured OpenCensusTracingSpi and all scopes disabled
> > >>> -52% for TX scope (IgnitePutTxBenchmark)
> > >>> -58% for SQL scope  (IgniteSqlQueryBenchmark)
> > >>>
> > >>> Such a performance drop is significant to not use the tracing in
> > >> production.
> > >>>
> > >>> I have considered the possibility to reuse the tracing API. If
> > >>> statistics collecting will be implemented with the TracingSpi then we
> > >>> get a performance drop due to:
> > >>> - Transferring tracing context over the network.
> > >>> - Using ThreadLocal for spans
> > >>> - Converting primitives and objects to string and vice versa. (API
> > >>> supports only String-based tags and values)
> > >>> - Generating span objects
> > >>>
> > >>> I have benchmarked implementations on the yardstick’s
> > >>> IgniteGetBenchmark. The tracing context transferring over the network
> > >>> was disabled. The results:
> > >>> - Tracing API implementation - 8% performance drop.
> > >>> - Proposed implementation - 2% performance drop.
> > >>>
> > >>> I think this is a significant drop and implementation with reuse
> > >>> tracing API should not be used. The cluster profiling should have as
> > >>> little performance drop as possible to be used in production. The
> > >>> tracing will be used for the detailed investigation.
> > >>>
> > >>> WDYT?
> > >>>
> > >>> The tool is ready to be reviewed [3, 5].
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-12666
> > >>> [2]
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>> [3] https://github.com/apache/ignite-extensions/pull/16
> > >>> [4]
> > >>
> > https://issues.apache.org/jira/secure/attachment/13016636/Tracing%20benchmarks.docx
> > >>> [5] https://github.com/apache/ignite/pull/7693
> > >>>
> > >>> ср, 24 июн. 2020 г. в 23:31, Saikat Maitra <[hidden email]>:
> > >>>>
> > >>>> Hi Nikita,
> > >>>>
> > >>>> The changes in this PR looks good.
> > >>>>
> > >>>> https://github.com/apache/ignite-extensions/pull/16
> > >>>>
> > >>>> Regards,
> > >>>> Saikat
> > >>>>
> > >>>> On Mon, Jun 22, 2020 at 12:03 PM Nikolay Izhikov <[hidden email]
> > >
> > >>>> wrote:
> > >>>>
> > >>>>> Hello, Igniters.
> > >>>>>
> > >>>>> I think that inside Ignite core we should name this feature as
> > >>>>> «performance statistics»
> > >>>>> We already have «cache statistics».
> > >>>>> Data that is collected by performance statistics can be used not only
> > >> for
> > >>>>> profiling but to solve other tasks.
> > >>>>>
> > >>>>>
> > >>>>>> 22 июня 2020 г., в 14:00, Nikita Amelchev <[hidden email]>
> > >>>>> написал(а):
> > >>>>>>
> > >>>>>> Hi, guys.
> > >>>>>>
> > >>>>>> I have mentioned components under the MIT license in the LICENSE
> > file.
> > >>>>>>
> > >>>>>> Saikat, I have fixed PR according to your suggestions. Thanks for
> > >> taking
> > >>>>> a look.
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best wishes,
> > >>> Amelchev Nikita
> > >>
> > >>
> >
> >

--
Best wishes,
Amelchev Nikita