Flaky tests

[Stanislav Kozlovski]

Hey there Ignite community,

I contribute to a fellow open-source project - Apache Kafka - and there we have been fighting flaky tests a lot. We run Java 8 and Java 11 builds on every Pull Request and due to test flakiness, almost all of them turn out red with 1 or 2 tests (completely unrelated to the change in the PR) failing. This has resulted in committers either ignoring them and merging the changes or in the worst case rerunning the hour-long build until it becomes green.
This test flakiness has also slowed down our releases significantly.

In general, I was just curious to understand if this is a problem that your project faces as well. Does your project have a lot of intermittently failing tests, do you have any active process of addressing such tests (during the initial review, after realizing it is flaky, etc). Any pointers will be greatly appreciated!

Thanks,
Stanislav


<http://aka.ms/weboutlook>

[dmagda]

Hello Stanislav,

Ignite also has some issues with Java 11 support. For a long time we have
been ignoring Java 11 existence at all. At the moment, the community has
been working on the full Java 11 support and this version should become a
default one soon. The tests will be done for both Java 8 and Java 11.

I'll let Dmitry Pavlov dig into the details since he is the one who is
driving or doing most of the adoption.

-
Denis


On Tue, Feb 26, 2019 at 2:39 PM Stanislav Kozlovski <
[hidden email]> wrote:

> Hey there Ignite community,
>
> I contribute to a fellow open-source project - Apache Kafka - and there we
> have been fighting flaky tests a lot. We run Java 8 and Java 11 builds on
> every Pull Request and due to test flakiness, almost all of them turn out
> red with 1 or 2 tests (completely unrelated to the change in the PR)
> failing. This has resulted in committers either ignoring them and merging
> the changes or in the worst case rerunning the hour-long build until it
> becomes green.
> This test flakiness has also slowed down our releases significantly.
>
> In general, I was just curious to understand if this is a problem that
> your project faces as well. Does your project have a lot of intermittently
> failing tests, do you have any active process of addressing such tests
> (during the initial review, after realizing it is flaky, etc). Any pointers
> will be greatly appreciated!
>
> Thanks,
> Stanislav
>
>
> <http://aka.ms/weboutlook>
>

[Ilya Kasnacheev]

Hello!

We also have this wonderful tool: https://mtcga.gridgain.com/

It keeps track of tests which are currently flaky or under investigation,
and given your change it will report non-flaky tests that failed, skipping
all problematic tests.

It talks to TeamCity so it can run tests for you, check their results and
comment JIRA with "Bot Visa", see:
https://issues.apache.org/jira/browse/IGNITE-11299 comment from "Ignite TC
Bot"

Since we don't have too many flaky tests thanks to Make TeamCity Green
Again movement, it works reasonably well.

A year ago we did not have any of this but now we do, and we were able to
shave 80% of test-checking busy work. This is mostly thanks to Dmitry
Pavlov.

Regards,
--
Ilya Kasnacheev


ср, 27 февр. 2019 г. в 01:39, Stanislav Kozlovski <
[hidden email]>:

> Hey there Ignite community,
>
> I contribute to a fellow open-source project - Apache Kafka - and there we
> have been fighting flaky tests a lot. We run Java 8 and Java 11 builds on
> every Pull Request and due to test flakiness, almost all of them turn out
> red with 1 or 2 tests (completely unrelated to the change in the PR)
> failing. This has resulted in committers either ignoring them and merging
> the changes or in the worst case rerunning the hour-long build until it
> becomes green.
> This test flakiness has also slowed down our releases significantly.
>
> In general, I was just curious to understand if this is a problem that
> your project faces as well. Does your project have a lot of intermittently
> failing tests, do you have any active process of addressing such tests
> (during the initial review, after realizing it is flaky, etc). Any pointers
> will be greatly appreciated!
>
> Thanks,
> Stanislav
>
>
> <http://aka.ms/weboutlook>
>

[Dmitry Pavlov]

Thanks, Ilya for your positive feedback.

Stanislav,

Apache Ignite TeamCity Bot requires CI credentials, so it may be easier to
check its code and description here
https://github.com/apache/ignite-teamcity-bot#ignite-teamcity-bot

(For Russian native speakers there is a blog post here
https://habr.com/ru/company/sberbank/blog/436070/ , it is not yet
translated to English)

As for Java 11, there is an idea to separate stable/flaky statistics from
the master for two Java versions, but it is only planned now.

Sincerely,
Dmitriy Pavlov

ср, 27 февр. 2019 г. в 12:01, Ilya Kasnacheev <[hidden email]>:

> Hello!
>
> We also have this wonderful tool: https://mtcga.gridgain.com/
>
> It keeps track of tests which are currently flaky or under investigation,
> and given your change it will report non-flaky tests that failed, skipping
> all problematic tests.
>
> It talks to TeamCity so it can run tests for you, check their results and
> comment JIRA with "Bot Visa", see:
> https://issues.apache.org/jira/browse/IGNITE-11299 comment from "Ignite TC
> Bot"
>
> Since we don't have too many flaky tests thanks to Make TeamCity Green
> Again movement, it works reasonably well.
>
> A year ago we did not have any of this but now we do, and we were able to
> shave 80% of test-checking busy work. This is mostly thanks to Dmitry
> Pavlov.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 27 февр. 2019 г. в 01:39, Stanislav Kozlovski <
> [hidden email]>:
>
> > Hey there Ignite community,
> >
> > I contribute to a fellow open-source project - Apache Kafka - and there
> we
> > have been fighting flaky tests a lot. We run Java 8 and Java 11 builds on
> > every Pull Request and due to test flakiness, almost all of them turn out
> > red with 1 or 2 tests (completely unrelated to the change in the PR)
> > failing. This has resulted in committers either ignoring them and merging
> > the changes or in the worst case rerunning the hour-long build until it
> > becomes green.
> > This test flakiness has also slowed down our releases significantly.
> >
> > In general, I was just curious to understand if this is a problem that
> > your project faces as well. Does your project have a lot of
> intermittently
> > failing tests, do you have any active process of addressing such tests
> > (during the initial review, after realizing it is flaky, etc). Any
> pointers
> > will be greatly appreciated!
> >
> > Thanks,
> > Stanislav
> >
> >
> > <http://aka.ms/weboutlook>
> >
>

[dmagda]

Dmitry,

Do you think Kafka community can adjust and adapt the bot for their needs?

-
Denis


On Wed, Feb 27, 2019 at 4:13 AM Dmitriy Pavlov <[hidden email]> wrote:

> Thanks, Ilya for your positive feedback.
>
> Stanislav,
>
> Apache Ignite TeamCity Bot requires CI credentials, so it may be easier to
> check its code and description here
> https://github.com/apache/ignite-teamcity-bot#ignite-teamcity-bot
>
> (For Russian native speakers there is a blog post here
> https://habr.com/ru/company/sberbank/blog/436070/ , it is not yet
> translated to English)
>
> As for Java 11, there is an idea to separate stable/flaky statistics from
> the master for two Java versions, but it is only planned now.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 27 февр. 2019 г. в 12:01, Ilya Kasnacheev <[hidden email]>:
>
> > Hello!
> >
> > We also have this wonderful tool: https://mtcga.gridgain.com/
> >
> > It keeps track of tests which are currently flaky or under investigation,
> > and given your change it will report non-flaky tests that failed,
> skipping
> > all problematic tests.
> >
> > It talks to TeamCity so it can run tests for you, check their results and
> > comment JIRA with "Bot Visa", see:
> > https://issues.apache.org/jira/browse/IGNITE-11299 comment from "Ignite
> TC
> > Bot"
> >
> > Since we don't have too many flaky tests thanks to Make TeamCity Green
> > Again movement, it works reasonably well.
> >
> > A year ago we did not have any of this but now we do, and we were able to
> > shave 80% of test-checking busy work. This is mostly thanks to Dmitry
> > Pavlov.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 27 февр. 2019 г. в 01:39, Stanislav Kozlovski <
> > [hidden email]>:
> >
> > > Hey there Ignite community,
> > >
> > > I contribute to a fellow open-source project - Apache Kafka - and there
> > we
> > > have been fighting flaky tests a lot. We run Java 8 and Java 11 builds
> on
> > > every Pull Request and due to test flakiness, almost all of them turn
> out
> > > red with 1 or 2 tests (completely unrelated to the change in the PR)
> > > failing. This has resulted in committers either ignoring them and
> merging
> > > the changes or in the worst case rerunning the hour-long build until it
> > > becomes green.
> > > This test flakiness has also slowed down our releases significantly.
> > >
> > > In general, I was just curious to understand if this is a problem that
> > > your project faces as well. Does your project have a lot of
> > intermittently
> > > failing tests, do you have any active process of addressing such tests
> > > (during the initial review, after realizing it is flaky, etc). Any
> > pointers
> > > will be greatly appreciated!
> > >
> > > Thanks,
> > > Stanislav
> > >
> > >
> > > <http://aka.ms/weboutlook>
> > >
> >
>

[Dmitry Pavlov]

Why not? If Apache Kafka uses TeamCity, JIRA and GitHub, then Apache Ignite
TeamCity Bot is reusable there. It may require to apply a number of patches
to make heuristics configurable, Ignite-specific logic disablable, etc, but
it is possible to reuse.

I would be happy if Bot will have a number more contributors, and I can
assist in developing it, as well.


ср, 27 февр. 2019 г., 21:44 Denis Magda <[hidden email]>:

> Dmitry,
>
> Do you think Kafka community can adjust and adapt the bot for their needs?
>
> -
> Denis
>
>
> On Wed, Feb 27, 2019 at 4:13 AM Dmitriy Pavlov <[hidden email]> wrote:
>
> > Thanks, Ilya for your positive feedback.
> >
> > Stanislav,
> >
> > Apache Ignite TeamCity Bot requires CI credentials, so it may be easier
> to
> > check its code and description here
> > https://github.com/apache/ignite-teamcity-bot#ignite-teamcity-bot
> >
> > (For Russian native speakers there is a blog post here
> > https://habr.com/ru/company/sberbank/blog/436070/ , it is not yet
> > translated to English)
> >
> > As for Java 11, there is an idea to separate stable/flaky statistics from
> > the master for two Java versions, but it is only planned now.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > ср, 27 февр. 2019 г. в 12:01, Ilya Kasnacheev <[hidden email]
> >:
> >
> > > Hello!
> > >
> > > We also have this wonderful tool: https://mtcga.gridgain.com/
> > >
> > > It keeps track of tests which are currently flaky or under
> investigation,
> > > and given your change it will report non-flaky tests that failed,
> > skipping
> > > all problematic tests.
> > >
> > > It talks to TeamCity so it can run tests for you, check their results
> and
> > > comment JIRA with "Bot Visa", see:
> > > https://issues.apache.org/jira/browse/IGNITE-11299 comment from
> "Ignite
> > TC
> > > Bot"
> > >
> > > Since we don't have too many flaky tests thanks to Make TeamCity Green
> > > Again movement, it works reasonably well.
> > >
> > > A year ago we did not have any of this but now we do, and we were able
> to
> > > shave 80% of test-checking busy work. This is mostly thanks to Dmitry
> > > Pavlov.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 27 февр. 2019 г. в 01:39, Stanislav Kozlovski <
> > > [hidden email]>:
> > >
> > > > Hey there Ignite community,
> > > >
> > > > I contribute to a fellow open-source project - Apache Kafka - and
> there
> > > we
> > > > have been fighting flaky tests a lot. We run Java 8 and Java 11
> builds
> > on
> > > > every Pull Request and due to test flakiness, almost all of them turn
> > out
> > > > red with 1 or 2 tests (completely unrelated to the change in the PR)
> > > > failing. This has resulted in committers either ignoring them and
> > merging
> > > > the changes or in the worst case rerunning the hour-long build until
> it
> > > > becomes green.
> > > > This test flakiness has also slowed down our releases significantly.
> > > >
> > > > In general, I was just curious to understand if this is a problem
> that
> > > > your project faces as well. Does your project have a lot of
> > > intermittently
> > > > failing tests, do you have any active process of addressing such
> tests
> > > > (during the initial review, after realizing it is flaky, etc). Any
> > > pointers
> > > > will be greatly appreciated!
> > > >
> > > > Thanks,
> > > > Stanislav
> > > >
> > > >
> > > > <http://aka.ms/weboutlook>
> > > >
> > >
> >
>

[Stanislav Kozlovski]

Thank you for clarifying everybody, this is valuable expertise.

On 2019/02/27 09:01:00, Ilya Kasnacheev <[hidden email]> wrote:

> Hello!>
>
> We also have this wonderful tool: https://mtcga.gridgain.com/>
>
> It keeps track of tests which are currently flaky or under investigation,>
> and given your change it will report non-flaky tests that failed, skipping>
> all problematic tests.>
>
> It talks to TeamCity so it can run tests for you, check their results and>
> comment JIRA with "Bot Visa", see:>
> https://issues.apache.org/jira/browse/IGNITE-11299 comment from "Ignite TC>
> Bot">
>
> Since we don't have too many flaky tests thanks to Make TeamCity Green>
> Again movement, it works reasonably well.>
>
> A year ago we did not have any of this but now we do, and we were able to>
> shave 80% of test-checking busy work. This is mostly thanks to Dmitry>
> Pavlov.>
>
> Regards,>
> -- >
> Ilya Kasnacheev>
>
>
> ср, 27 февр. 2019 г. в 01:39, Stanislav Kozlovski <>
> [hidden email]>:>
>
> > Hey there Ignite community,>
> >>
> > I contribute to a fellow open-source project - Apache Kafka - and there we>
> > have been fighting flaky tests a lot. We run Java 8 and Java 11 builds on>
> > every Pull Request and due to test flakiness, almost all of them turn out>
> > red with 1 or 2 tests (completely unrelated to the change in the PR)>
> > failing. This has resulted in committers either ignoring them and merging>
> > the changes or in the worst case rerunning the hour-long build until it>
> > becomes green.>
> > This test flakiness has also slowed down our releases significantly.>
> >>
> > In general, I was just curious to understand if this is a problem that>
> > your project faces as well. Does your project have a lot of intermittently>
> > failing tests, do you have any active process of addressing such tests>
> > (during the initial review, after realizing it is flaky, etc). Any pointers>
> > will be greatly appreciated!>
> >>
> > Thanks,>
> > Stanislav>
> >>
> >>
> > <http://aka.ms/weboutlook>>
> >>
>