Running single test multiple times on TeamCity

[Ivan Pavlukhin]

Hi,

During a couple of last weeks I was fixing several flaky tests.
Sometimes it was quite hard to reproduce a test locally. So, one
option was running a particular test on TC several times in a row. To
setup such run I did code modifications in several places.

I thought about how to simplify the thing. And I came up with some
sort of solution which I would like to share. Basically it is custom
junit runner DebugSuite and a configuration annotation
DebugSuite.Config which allows to choose a method to run and number of
executions. You can see a draft in PR [1].

As always there are several options to solve a problem. One
alternative way is creating something similar to parameterized build
job Jenkins employs [2] (I have not checked for TC analog yet) and
using maven features to run single test repeatedly (have not checked
as well). But all in all we need to answer following questions:
1. Do we need such tool? (Or perhaps we already have something and
there is no need to reinvent the wheel.)
2. What is the best way for us to implement the tool?

[1] https://github.com/apache/ignite/pull/6076
[2] https://wiki.jenkins.io/display/JENKINS/Parameterized+Build

--
Best regards,
Ivan Pavlukhin

[V.Pyatkov]

Hi,

I think more test falling on TC due to context in which running.
If you added it in debug suite, it can stop failing anymore like local run.

On Mon, Feb 11, 2019 at 9:37 AM Павлухин Иван <[hidden email]> wrote:

> Hi,
>
> During a couple of last weeks I was fixing several flaky tests.
> Sometimes it was quite hard to reproduce a test locally. So, one
> option was running a particular test on TC several times in a row. To
> setup such run I did code modifications in several places.
>
> I thought about how to simplify the thing. And I came up with some
> sort of solution which I would like to share. Basically it is custom
> junit runner DebugSuite and a configuration annotation
> DebugSuite.Config which allows to choose a method to run and number of
> executions. You can see a draft in PR [1].
>
> As always there are several options to solve a problem. One
> alternative way is creating something similar to parameterized build
> job Jenkins employs [2] (I have not checked for TC analog yet) and
> using maven features to run single test repeatedly (have not checked
> as well). But all in all we need to answer following questions:
> 1. Do we need such tool? (Or perhaps we already have something and
> there is no need to reinvent the wheel.)
> 2. What is the best way for us to implement the tool?
>
> [1] https://github.com/apache/ignite/pull/6076
> [2] https://wiki.jenkins.io/display/JENKINS/Parameterized+Build
>
> --
> Best regards,
> Ivan Pavlukhin
>

--
Vladislav Pyatkov

[Ivan Pavlukhin]

Vlad,

Your case is valid as well. And DebugSuite will not help there. But I
talk about another case. And I have already reproduced several tests
after repetitive execution of a single test on TC. So, my case also
seems valid.

пн, 11 февр. 2019 г. в 10:43, Vladislav Pyatkov <[hidden email]>:

>
> Hi,
>
> I think more test falling on TC due to context in which running.
> If you added it in debug suite, it can stop failing anymore like local run.
>
> On Mon, Feb 11, 2019 at 9:37 AM Павлухин Иван <[hidden email]> wrote:
>
> > Hi,
> >
> > During a couple of last weeks I was fixing several flaky tests.
> > Sometimes it was quite hard to reproduce a test locally. So, one
> > option was running a particular test on TC several times in a row. To
> > setup such run I did code modifications in several places.
> >
> > I thought about how to simplify the thing. And I came up with some
> > sort of solution which I would like to share. Basically it is custom
> > junit runner DebugSuite and a configuration annotation
> > DebugSuite.Config which allows to choose a method to run and number of
> > executions. You can see a draft in PR [1].
> >
> > As always there are several options to solve a problem. One
> > alternative way is creating something similar to parameterized build
> > job Jenkins employs [2] (I have not checked for TC analog yet) and
> > using maven features to run single test repeatedly (have not checked
> > as well). But all in all we need to answer following questions:
> > 1. Do we need such tool? (Or perhaps we already have something and
> > there is no need to reinvent the wheel.)
> > 2. What is the best way for us to implement the tool?
> >
> > [1] https://github.com/apache/ignite/pull/6076
> > [2] https://wiki.jenkins.io/display/JENKINS/Parameterized+Build
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>
>
> --
> Vladislav Pyatkov

--
Best regards,
Ivan Pavlukhin

[Eduard Shangareev]

Ivan,
We already have IgniteReproducingSuite, and there was a corresponding TC
configuration, but it was removed (maybe by accident).
We could resurrect TC configuration.
+ We can improve Suite to provide test class name by parameter instead of
changing code.

On Mon, Feb 11, 2019 at 11:22 AM Павлухин Иван <[hidden email]> wrote:

> Vlad,
>
> Your case is valid as well. And DebugSuite will not help there. But I
> talk about another case. And I have already reproduced several tests
> after repetitive execution of a single test on TC. So, my case also
> seems valid.
>
> пн, 11 февр. 2019 г. в 10:43, Vladislav Pyatkov <[hidden email]>:
> >
> > Hi,
> >
> > I think more test falling on TC due to context in which running.
> > If you added it in debug suite, it can stop failing anymore like local
> run.
> >
> > On Mon, Feb 11, 2019 at 9:37 AM Павлухин Иван <[hidden email]>
> wrote:
> >
> > > Hi,
> > >
> > > During a couple of last weeks I was fixing several flaky tests.
> > > Sometimes it was quite hard to reproduce a test locally. So, one
> > > option was running a particular test on TC several times in a row. To
> > > setup such run I did code modifications in several places.
> > >
> > > I thought about how to simplify the thing. And I came up with some
> > > sort of solution which I would like to share. Basically it is custom
> > > junit runner DebugSuite and a configuration annotation
> > > DebugSuite.Config which allows to choose a method to run and number of
> > > executions. You can see a draft in PR [1].
> > >
> > > As always there are several options to solve a problem. One
> > > alternative way is creating something similar to parameterized build
> > > job Jenkins employs [2] (I have not checked for TC analog yet) and
> > > using maven features to run single test repeatedly (have not checked
> > > as well). But all in all we need to answer following questions:
> > > 1. Do we need such tool? (Or perhaps we already have something and
> > > there is no need to reinvent the wheel.)
> > > 2. What is the best way for us to implement the tool?
> > >
> > > [1] https://github.com/apache/ignite/pull/6076
> > > [2] https://wiki.jenkins.io/display/JENKINS/Parameterized+Build
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
> >
> > --
> > Vladislav Pyatkov
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[Ivan Pavlukhin]

Ed,

Thank you for pointing out IgniteReproducingSuite. It looks like it
fits my needs. Do you we have a ticket for bringing back TC
configuration? Should I create one?

ср, 13 февр. 2019 г. в 17:33, Eduard Shangareev <[hidden email]>:

>
> Ivan,
> We already have IgniteReproducingSuite, and there was a corresponding TC
> configuration, but it was removed (maybe by accident).
> We could resurrect TC configuration.
> + We can improve Suite to provide test class name by parameter instead of
> changing code.
>
> On Mon, Feb 11, 2019 at 11:22 AM Павлухин Иван <[hidden email]> wrote:
>
> > Vlad,
> >
> > Your case is valid as well. And DebugSuite will not help there. But I
> > talk about another case. And I have already reproduced several tests
> > after repetitive execution of a single test on TC. So, my case also
> > seems valid.
> >
> > пн, 11 февр. 2019 г. в 10:43, Vladislav Pyatkov <[hidden email]>:
> > >
> > > Hi,
> > >
> > > I think more test falling on TC due to context in which running.
> > > If you added it in debug suite, it can stop failing anymore like local
> > run.
> > >
> > > On Mon, Feb 11, 2019 at 9:37 AM Павлухин Иван <[hidden email]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > During a couple of last weeks I was fixing several flaky tests.
> > > > Sometimes it was quite hard to reproduce a test locally. So, one
> > > > option was running a particular test on TC several times in a row. To
> > > > setup such run I did code modifications in several places.
> > > >
> > > > I thought about how to simplify the thing. And I came up with some
> > > > sort of solution which I would like to share. Basically it is custom
> > > > junit runner DebugSuite and a configuration annotation
> > > > DebugSuite.Config which allows to choose a method to run and number of
> > > > executions. You can see a draft in PR [1].
> > > >
> > > > As always there are several options to solve a problem. One
> > > > alternative way is creating something similar to parameterized build
> > > > job Jenkins employs [2] (I have not checked for TC analog yet) and
> > > > using maven features to run single test repeatedly (have not checked
> > > > as well). But all in all we need to answer following questions:
> > > > 1. Do we need such tool? (Or perhaps we already have something and
> > > > there is no need to reinvent the wheel.)
> > > > 2. What is the best way for us to implement the tool?
> > > >
> > > > [1] https://github.com/apache/ignite/pull/6076
> > > > [2] https://wiki.jenkins.io/display/JENKINS/Parameterized+Build
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> > >
> > > --
> > > Vladislav Pyatkov
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >

--
Best regards,
Ivan Pavlukhin

[daradurvs]

I'd recommend move the IgniteReproducingSuite to separate module to be
able use it for all modules.

For example, at the moment, we are not able use the suite to run tests
from 'indexing' module because of lack dependency.

On Thu, Feb 14, 2019 at 10:38 AM Павлухин Иван <[hidden email]> wrote:

>
> Ed,
>
> Thank you for pointing out IgniteReproducingSuite. It looks like it
> fits my needs. Do you we have a ticket for bringing back TC
> configuration? Should I create one?
>
> ср, 13 февр. 2019 г. в 17:33, Eduard Shangareev <[hidden email]>:
> >
> > Ivan,
> > We already have IgniteReproducingSuite, and there was a corresponding TC
> > configuration, but it was removed (maybe by accident).
> > We could resurrect TC configuration.
> > + We can improve Suite to provide test class name by parameter instead of
> > changing code.
> >
> > On Mon, Feb 11, 2019 at 11:22 AM Павлухин Иван <[hidden email]> wrote:
> >
> > > Vlad,
> > >
> > > Your case is valid as well. And DebugSuite will not help there. But I
> > > talk about another case. And I have already reproduced several tests
> > > after repetitive execution of a single test on TC. So, my case also
> > > seems valid.
> > >
> > > пн, 11 февр. 2019 г. в 10:43, Vladislav Pyatkov <[hidden email]>:
> > > >
> > > > Hi,
> > > >
> > > > I think more test falling on TC due to context in which running.
> > > > If you added it in debug suite, it can stop failing anymore like local
> > > run.
> > > >
> > > > On Mon, Feb 11, 2019 at 9:37 AM Павлухин Иван <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > During a couple of last weeks I was fixing several flaky tests.
> > > > > Sometimes it was quite hard to reproduce a test locally. So, one
> > > > > option was running a particular test on TC several times in a row. To
> > > > > setup such run I did code modifications in several places.
> > > > >
> > > > > I thought about how to simplify the thing. And I came up with some
> > > > > sort of solution which I would like to share. Basically it is custom
> > > > > junit runner DebugSuite and a configuration annotation
> > > > > DebugSuite.Config which allows to choose a method to run and number of
> > > > > executions. You can see a draft in PR [1].
> > > > >
> > > > > As always there are several options to solve a problem. One
> > > > > alternative way is creating something similar to parameterized build
> > > > > job Jenkins employs [2] (I have not checked for TC analog yet) and
> > > > > using maven features to run single test repeatedly (have not checked
> > > > > as well). But all in all we need to answer following questions:
> > > > > 1. Do we need such tool? (Or perhaps we already have something and
> > > > > there is no need to reinvent the wheel.)
> > > > > 2. What is the best way for us to implement the tool?
> > > > >
> > > > > [1] https://github.com/apache/ignite/pull/6076
> > > > > [2] https://wiki.jenkins.io/display/JENKINS/Parameterized+Build
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > >
> > > >
> > > >
> > > > --
> > > > Vladislav Pyatkov
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin

--
Best Regards, Vyacheslav D.