Ignite internal events tracing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Ignite internal events tracing

Alexey Goncharuk
Igniters,

I've recently stumbled across a situation when occasionally Ignite
transactions commit may take up to several seconds while in general most of
the transactions completed in a period of milliseconds.

After a few attempts to analyze this situation with logs, I realized that
this is a no-go and I need a finer instrument for this. The idea is to
introduce several trace points along the way of an Ignite operation and
collect timings when an operation passes each of the trace points. When
enabled, this information should be available upon the operation completion.

I've implemented a prototype of this for TX commit operation, the
implementation is available in ignite-5797 branch.

I was wondering if something of this kind may be useful as a part of Ignite
product and available to users. If so, I would like to discuss the public
API for this so the feature can be finalized.

Thanks,
AG
Reply | Threaded
Open this post in threaded view
|

Re: Ignite internal events tracing

Valentin Kulichenko
Alex,

That's a great idea. I would also add an option to dump information on
demand, for case when operation hanged and can't complete.

-Val

On Fri, Jul 21, 2017 at 6:15 AM, Alexey Goncharuk <
[hidden email]> wrote:

> Igniters,
>
> I've recently stumbled across a situation when occasionally Ignite
> transactions commit may take up to several seconds while in general most of
> the transactions completed in a period of milliseconds.
>
> After a few attempts to analyze this situation with logs, I realized that
> this is a no-go and I need a finer instrument for this. The idea is to
> introduce several trace points along the way of an Ignite operation and
> collect timings when an operation passes each of the trace points. When
> enabled, this information should be available upon the operation
> completion.
>
> I've implemented a prototype of this for TX commit operation, the
> implementation is available in ignite-5797 branch.
>
> I was wondering if something of this kind may be useful as a part of Ignite
> product and available to users. If so, I would like to discuss the public
> API for this so the feature can be finalized.
>
> Thanks,
> AG
>
Reply | Threaded
Open this post in threaded view
|

Re: Ignite internal events tracing

dsetrakyan
I think this is a great idea, and should be implemented not only for
transactions, but also for SQL queries. Users should be able to turn this
tracing on and off through JMX and API. I am assuming that this information
will be dumped into the log as well, right?

D.

On Fri, Jul 21, 2017 at 12:04 PM, Valentin Kulichenko <
[hidden email]> wrote:

> Alex,
>
> That's a great idea. I would also add an option to dump information on
> demand, for case when operation hanged and can't complete.
>
> -Val
>
> On Fri, Jul 21, 2017 at 6:15 AM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > Igniters,
> >
> > I've recently stumbled across a situation when occasionally Ignite
> > transactions commit may take up to several seconds while in general most
> of
> > the transactions completed in a period of milliseconds.
> >
> > After a few attempts to analyze this situation with logs, I realized that
> > this is a no-go and I need a finer instrument for this. The idea is to
> > introduce several trace points along the way of an Ignite operation and
> > collect timings when an operation passes each of the trace points. When
> > enabled, this information should be available upon the operation
> > completion.
> >
> > I've implemented a prototype of this for TX commit operation, the
> > implementation is available in ignite-5797 branch.
> >
> > I was wondering if something of this kind may be useful as a part of
> Ignite
> > product and available to users. If so, I would like to discuss the public
> > API for this so the feature can be finalized.
> >
> > Thanks,
> > AG
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Ignite internal events tracing

yzhdanov
Alex, I like the idea very much, but I think we need to rethink the
implementation approach to make it more generic. Passing parameter to each
invocation seems dirty to me.

Val,  we already have this. Please
see org.apache.ignite.internal.IgniteDiagnosticAware

Dmitry, what you suggest will be pretty hard to implement. I would better
improve self-diagnostic system to extend list of metrics Ignite monitors.

--Yakov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite internal events tracing

dsetrakyan
On Mon, Jul 24, 2017 at 5:24 AM, Yakov Zhdanov <[hidden email]> wrote:

> Alex, I like the idea very much, but I think we need to rethink the
> implementation approach to make it more generic. Passing parameter to each
> invocation seems dirty to me.
>
> Val,  we already have this. Please
> see org.apache.ignite.internal.IgniteDiagnosticAware
>
> Dmitry, what you suggest will be pretty hard to implement. I would better
> improve self-diagnostic system to extend list of metrics Ignite monitors.
>

Yakov, I was thinking something along the lines of
"enableTransactionTracing(true)" or "enableSqlTracing" and the JMX or API
level. Why should it be hard?
Reply | Threaded
Open this post in threaded view
|

Re: Ignite internal events tracing

Valentin Kulichenko
In reply to this post by yzhdanov
Yakov,

How IgniteDiagnosticAware can be used? Is there any information?

-Val

On Mon, Jul 24, 2017 at 3:24 AM, Yakov Zhdanov <[hidden email]> wrote:

> Alex, I like the idea very much, but I think we need to rethink the
> implementation approach to make it more generic. Passing parameter to each
> invocation seems dirty to me.
>
> Val,  we already have this. Please
> see org.apache.ignite.internal.IgniteDiagnosticAware
>
> Dmitry, what you suggest will be pretty hard to implement. I would better
> improve self-diagnostic system to extend list of metrics Ignite monitors.
>
> --Yakov
>
Reply | Threaded
Open this post in threaded view
|

Re: Ignite internal events tracing

yzhdanov
Val, it is used by Ignite internals to report the hangs and output
diagnostic information.

--Yakov