[DISCUSSION] Fixed test order to reduce flakyness

[Dmitry Pavlov]

Dear Ignite Developers,

The original idea came from our recent habr.ru post related to Apache
Ignite TeamCity Bot (for Russian native speakers, you can read an original
https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976 )

It is a known phenomenon when tests have an influence on each other. The
simplest case when Ignite Native persistence is used, and not properly
cleared after a test run. This can make some test failed afterward.

So, what if we will set predictable, for example, alphabetical tests
execution order (maven-surefire-plugin/runOrder/alphabetical). This may
have the following effect: the set of tests failed because of being
affected by the previous run will be constant, will be exactly the same
each run.

At some point, when we stabilize flaky tests enough, we may select random
order, but for now, this solution seems valid to me.

What do you think?

Sincerely,
Dmitriy Pavlov

[Ivan Pavlukhin]

Dmitriy,

Sounds like a good idea to me. Problems related to tests isolation are
very common in practice. And things can be complicated when order
varies.

But by the way does the order of Ignite tests vary today? Junit 4
javadocs claims something about "default deterministic order" [1].

[1] https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html

вт, 5 февр. 2019 г. в 17:40, Dmitriy Pavlov <[hidden email]>:

>
> Dear Ignite Developers,
>
> The original idea came from our recent habr.ru post related to Apache
> Ignite TeamCity Bot (for Russian native speakers, you can read an original
> https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976 )
>
> It is a known phenomenon when tests have an influence on each other. The
> simplest case when Ignite Native persistence is used, and not properly
> cleared after a test run. This can make some test failed afterward.
>
> So, what if we will set predictable, for example, alphabetical tests
> execution order (maven-surefire-plugin/runOrder/alphabetical). This may
> have the following effect: the set of tests failed because of being
> affected by the previous run will be constant, will be exactly the same
> each run.
>
> At some point, when we stabilize flaky tests enough, we may select random
> order, but for now, this solution seems valid to me.
>
> What do you think?
>
> Sincerely,
> Dmitriy Pavlov

--
Best regards,
Ivan Pavlukhin

[Eduard Shangareev]

Dmitriy,

Please, clarify the idea.
What do we want to achieve by this? Making tests more stable by hiding some
issues in our tests/codebase?


On Tue, Feb 5, 2019 at 6:57 PM Павлухин Иван <[hidden email]> wrote:

> Dmitriy,
>
> Sounds like a good idea to me. Problems related to tests isolation are
> very common in practice. And things can be complicated when order
> varies.
>
> But by the way does the order of Ignite tests vary today? Junit 4
> javadocs claims something about "default deterministic order" [1].
>
> [1] https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html
>
> вт, 5 февр. 2019 г. в 17:40, Dmitriy Pavlov <[hidden email]>:
> >
> > Dear Ignite Developers,
> >
> > The original idea came from our recent habr.ru post related to Apache
> > Ignite TeamCity Bot (for Russian native speakers, you can read an
> original
> > https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976 )
> >
> > It is a known phenomenon when tests have an influence on each other. The
> > simplest case when Ignite Native persistence is used, and not properly
> > cleared after a test run. This can make some test failed afterward.
> >
> > So, what if we will set predictable, for example, alphabetical tests
> > execution order (maven-surefire-plugin/runOrder/alphabetical). This may
> > have the following effect: the set of tests failed because of being
> > affected by the previous run will be constant, will be exactly the same
> > each run.
> >
> > At some point, when we stabilize flaky tests enough, we may select random
> > order, but for now, this solution seems valid to me.
> >
> > What do you think?
> >
> > Sincerely,
> > Dmitriy Pavlov
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[Dmitry Pavlov]

I mean changing parameter
http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrder
which
is fileSystem by default.

It is not about hiding. If a problematic test affects other test is will
continue to affect. The main point here is only about the test, which will
be affected. With unpredictable order, testA may break testB, testC, testD.

But for predictive & fixed order test affected by a failure of testA will
be always testC, and B&D will not be considered flaky because of the
randomized nature of execution. It will help for building at least good
statistics in the TC Bot.

ср, 6 февр. 2019 г. в 17:37, Eduard Shangareev <[hidden email]
>:

> Dmitriy,
>
> Please, clarify the idea.
> What do we want to achieve by this? Making tests more stable by hiding some
> issues in our tests/codebase?
>
>
> On Tue, Feb 5, 2019 at 6:57 PM Павлухин Иван <[hidden email]> wrote:
>
> > Dmitriy,
> >
> > Sounds like a good idea to me. Problems related to tests isolation are
> > very common in practice. And things can be complicated when order
> > varies.
> >
> > But by the way does the order of Ignite tests vary today? Junit 4
> > javadocs claims something about "default deterministic order" [1].
> >
> > [1]
> https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html
> >
> > вт, 5 февр. 2019 г. в 17:40, Dmitriy Pavlov <[hidden email]>:
> > >
> > > Dear Ignite Developers,
> > >
> > > The original idea came from our recent habr.ru post related to Apache
> > > Ignite TeamCity Bot (for Russian native speakers, you can read an
> > original
> > > https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976 )
> > >
> > > It is a known phenomenon when tests have an influence on each other.
> The
> > > simplest case when Ignite Native persistence is used, and not properly
> > > cleared after a test run. This can make some test failed afterward.
> > >
> > > So, what if we will set predictable, for example, alphabetical tests
> > > execution order (maven-surefire-plugin/runOrder/alphabetical). This may
> > > have the following effect: the set of tests failed because of being
> > > affected by the previous run will be constant, will be exactly the same
> > > each run.
> > >
> > > At some point, when we stabilize flaky tests enough, we may select
> random
> > > order, but for now, this solution seems valid to me.
> > >
> > > What do you think?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>

[Ivan Pavlukhin]

Dmitriy,

Initially I thought that the topic is about order of test METHODS
execution. You said about "fileSystem". I suppose file system order
have no meaning for methods but rather for test classes. Do other
possible values of the mentioned parameter affect order of methods
execution?

For convenience I remind a way how to define method order I am aware
of and wrote before [1].

[1] https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html

пт, 8 февр. 2019 г. в 22:18, Dmitriy Pavlov <[hidden email]>:

>
> I mean changing parameter
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrder
> which
> is fileSystem by default.
>
> It is not about hiding. If a problematic test affects other test is will
> continue to affect. The main point here is only about the test, which will
> be affected. With unpredictable order, testA may break testB, testC, testD.
>
> But for predictive & fixed order test affected by a failure of testA will
> be always testC, and B&D will not be considered flaky because of the
> randomized nature of execution. It will help for building at least good
> statistics in the TC Bot.
>
> ср, 6 февр. 2019 г. в 17:37, Eduard Shangareev <[hidden email]
> >:
>
> > Dmitriy,
> >
> > Please, clarify the idea.
> > What do we want to achieve by this? Making tests more stable by hiding some
> > issues in our tests/codebase?
> >
> >
> > On Tue, Feb 5, 2019 at 6:57 PM Павлухин Иван <[hidden email]> wrote:
> >
> > > Dmitriy,
> > >
> > > Sounds like a good idea to me. Problems related to tests isolation are
> > > very common in practice. And things can be complicated when order
> > > varies.
> > >
> > > But by the way does the order of Ignite tests vary today? Junit 4
> > > javadocs claims something about "default deterministic order" [1].
> > >
> > > [1]
> > https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html
> > >
> > > вт, 5 февр. 2019 г. в 17:40, Dmitriy Pavlov <[hidden email]>:
> > > >
> > > > Dear Ignite Developers,
> > > >
> > > > The original idea came from our recent habr.ru post related to Apache
> > > > Ignite TeamCity Bot (for Russian native speakers, you can read an
> > > original
> > > > https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976 )
> > > >
> > > > It is a known phenomenon when tests have an influence on each other.
> > The
> > > > simplest case when Ignite Native persistence is used, and not properly
> > > > cleared after a test run. This can make some test failed afterward.
> > > >
> > > > So, what if we will set predictable, for example, alphabetical tests
> > > > execution order (maven-surefire-plugin/runOrder/alphabetical). This may
> > > > have the following effect: the set of tests failed because of being
> > > > affected by the previous run will be constant, will be exactly the same
> > > > each run.
> > > >
> > > > At some point, when we stabilize flaky tests enough, we may select
> > random
> > > > order, but for now, this solution seems valid to me.
> > > >
> > > > What do you think?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >

--
Best regards,
Ivan Pavlukhin

[Stanislav Lukyanov]

HI Dmitry,

> It is not about hiding. If a problematic test affects other test is will continue to
> affect. The main point here is only about the test, which will be affected.
> With unpredictable order, testA may break testB, testC, testD.

The issue is that with a fixed order testA may not break any tests (e.g. because it will become last).
Also, tests may not only break but "fix" each other, i.e. testA may make a should-be-failing testB pass.

>
> But for predictive & fixed order test affected by a failure of testA will be
> always testC, and B&D will not be considered flaky because of the
> randomized nature of execution. It will help for building at least good
> statistics in the TC Bot.

I agree that random test order may sound too scary and that file-system dependent order is
no good because it's like a fixed one, but varies across platforms.

In my experience the best way to tackle execution order of tests is to vary
direct and reverse fixed order (say, alphabetical). It gives enough difference to make sure
order-dependent tests fail sometimes, and, most importantly, it gives a way to catch them
(you can tell that a problem is related to order when a test only fails on direct/reverse runs).

I see that surefire already has in a form of runOrder=hourly.
I suggest that if we change to something, we change to that.

Stan

> -----Original Message-----
> From: Dmitriy Pavlov <[hidden email]>
> Sent: Friday, February 8, 2019 10:19 PM
> To: [hidden email]
> Subject: Re: [DISCUSSION] Fixed test order to reduce flakyness
>
> I mean changing parameter
> http://maven.apache.org/surefire/maven-surefire-plugin/test-
> mojo.html#runOrder
> which
> is fileSystem by default.
>
> It is not about hiding. If a problematic test affects other test is will continue to
> affect. The main point here is only about the test, which will be affected.
> With unpredictable order, testA may break testB, testC, testD.
>
> But for predictive & fixed order test affected by a failure of testA will be
> always testC, and B&D will not be considered flaky because of the
> randomized nature of execution. It will help for building at least good
> statistics in the TC Bot.
>
> ср, 6 февр. 2019 г. в 17:37, Eduard Shangareev
> <[hidden email]
> >:
>
> > Dmitriy,
> >
> > Please, clarify the idea.
> > What do we want to achieve by this? Making tests more stable by hiding
> > some issues in our tests/codebase?
> >
> >
> > On Tue, Feb 5, 2019 at 6:57 PM Павлухин Иван <[hidden email]>
> wrote:
> >
> > > Dmitriy,
> > >
> > > Sounds like a good idea to me. Problems related to tests isolation
> > > are very common in practice. And things can be complicated when
> > > order varies.
> > >
> > > But by the way does the order of Ignite tests vary today? Junit 4
> > > javadocs claims something about "default deterministic order" [1].
> > >
> > > [1]
> > https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html
> > >
> > > вт, 5 февр. 2019 г. в 17:40, Dmitriy Pavlov <[hidden email]>:
> > > >
> > > > Dear Ignite Developers,
> > > >
> > > > The original idea came from our recent habr.ru post related to
> > > > Apache Ignite TeamCity Bot (for Russian native speakers, you can
> > > > read an
> > > original
> > > >
> https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976
> > > > )
> > > >
> > > > It is a known phenomenon when tests have an influence on each other.
> > The
> > > > simplest case when Ignite Native persistence is used, and not
> > > > properly cleared after a test run. This can make some test failed
> afterward.
> > > >
> > > > So, what if we will set predictable, for example, alphabetical
> > > > tests execution order
> > > > (maven-surefire-plugin/runOrder/alphabetical). This may have the
> > > > following effect: the set of tests failed because of being
> > > > affected by the previous run will be constant, will be exactly the same
> each run.
> > > >
> > > > At some point, when we stabilize flaky tests enough, we may select
> > random
> > > > order, but for now, this solution seems valid to me.
> > > >
> > > > What do you think?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >