Test failures

[voipp]

We have a lot of failed tests, which is frustrating. Some of them are
flaky(floating status randomly goes from succesful to failed) which adds to
frustration. Perhaps, we should fix all the tests in first place, and then
continue doing tickets ?
--

*Best Regards,*

*Kuznetsov Aleksey*

[dmagda]

Aleksey,

Bugs fixing and features development are two processes that usually co-exist.

Some of the committer/contributors fix tests/functionality while the others add new functionality. Someone does both.

You’re welcomed to start fixing the failing tests. Are there any specific that annoys you most?

—
Denis

> On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <[hidden email]> wrote:
>
> We have a lot of failed tests, which is frustrating. Some of them are
> flaky(floating status randomly goes from succesful to failed) which adds to
> frustration. Perhaps, we should fix all the tests in first place, and then
> continue doing tickets ?
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*

[voipp]

How could they co-exist ? When you developing some ticket you are risking
introduce bug which is reproduced by already failed test(s).
Moreover its time consuming to look up new failed tests when your build has
completed.
The last one, committers who introduced new bugs is responsible for them
and have to fix them, not other ones.
There are number of tests which i have to execute to ensure they are flaky
or permanently failed. For example this one :
CacheJdbcPojoStoreTest.testLoadCache()



вт, 7 февр. 2017 г. в 22:10, Denis Magda <[hidden email]>:

> Aleksey,
>
> Bugs fixing and features development are two processes that usually
> co-exist.
>
> Some of the committer/contributors fix tests/functionality while the
> others add new functionality. Someone does both.
>
> You’re welcomed to start fixing the failing tests. Are there any specific
> that annoys you most?
>
> —
> Denis
>
> > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <[hidden email]>
> wrote:
> >
> > We have a lot of failed tests, which is frustrating. Some of them are
> > flaky(floating status randomly goes from succesful to failed) which adds
> to
> > frustration. Perhaps, we should fix all the tests in first place, and
> then
> > continue doing tickets ?
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
>
> --

*Best Regards,*

*Kuznetsov Aleksey*

[afedotov]

Hi,

I would agree with Aleksey.
From the CI perspective, failing tests should be the main concern, because
it prevents a durable development of new features. Also, as Aleksey has
noted, developers working on different features could end up fixing the
same regressions, chances are - in different ways, resulting in merge
conflicts.
Having failing base branch, it is not possible to determine the reason of a
failure from the first sight.
All these points impact as the feedback from tests, so the whole process
agility.
Ideally, no new feature development should be started based on the failing
branch.
I think we should adopt this approach, otherwise, with growing amount of
features,
we will eventually end up spending more time dealing with regressions, than
developing features.

Regards,
Alexander

8 февр. 2017 г. 10:50 AM пользователь "ALEKSEY KUZNETSOV" <
[hidden email]> написал:

How could they co-exist ? When you developing some ticket you are risking
introduce bug which is reproduced by already failed test(s).
Moreover its time consuming to look up new failed tests when your build has
completed.
The last one, committers who introduced new bugs is responsible for them
and have to fix them, not other ones.
There are number of tests which i have to execute to ensure they are flaky
or permanently failed. For example this one :
CacheJdbcPojoStoreTest.testLoadCache()



вт, 7 февр. 2017 г. в 22:10, Denis Magda <[hidden email]>:

> Aleksey,
>
> Bugs fixing and features development are two processes that usually
> co-exist.
>
> Some of the committer/contributors fix tests/functionality while the
> others add new functionality. Someone does both.
>
> You’re welcomed to start fixing the failing tests. Are there any specific
> that annoys you most?
>
> —
> Denis
>
> > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <[hidden email]>
> wrote:
> >
> > We have a lot of failed tests, which is frustrating. Some of them are
> > flaky(floating status randomly goes from succesful to failed) which adds
> to
> > frustration. Perhaps, we should fix all the tests in first place, and
> then
> > continue doing tickets ?
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
>
> --

*Best Regards,*

*Kuznetsov Aleksey*

[daradurvs]

I vote for the master-branche without failed-tests)

I understand that impossible to make it quickly.

We shall aim at this approach.

It will be more comfortable to us to develop.

2017-02-08 12:17 GMT+03:00 Alexander Fedotov <[hidden email]>:

> Hi,
>
> I would agree with Aleksey.
> From the CI perspective, failing tests should be the main concern, because
> it prevents a durable development of new features. Also, as Aleksey has
> noted, developers working on different features could end up fixing the
> same regressions, chances are - in different ways, resulting in merge
> conflicts.
> Having failing base branch, it is not possible to determine the reason of a
> failure from the first sight.
> All these points impact as the feedback from tests, so the whole process
> agility.
> Ideally, no new feature development should be started based on the failing
> branch.
> I think we should adopt this approach, otherwise, with growing amount of
> features,
> we will eventually end up spending more time dealing with regressions, than
> developing features.
>
> Regards,
> Alexander
>
> 8 февр. 2017 г. 10:50 AM пользователь "ALEKSEY KUZNETSOV" <
> [hidden email]> написал:
>
> How could they co-exist ? When you developing some ticket you are risking
> introduce bug which is reproduced by already failed test(s).
> Moreover its time consuming to look up new failed tests when your build has
> completed.
> The last one, committers who introduced new bugs is responsible for them
> and have to fix them, not other ones.
> There are number of tests which i have to execute to ensure they are flaky
> or permanently failed. For example this one :
> CacheJdbcPojoStoreTest.testLoadCache()
>
>
>
> вт, 7 февр. 2017 г. в 22:10, Denis Magda <[hidden email]>:
>
> > Aleksey,
> >
> > Bugs fixing and features development are two processes that usually
> > co-exist.
> >
> > Some of the committer/contributors fix tests/functionality while the
> > others add new functionality. Someone does both.
> >
> > You’re welcomed to start fixing the failing tests. Are there any specific
> > that annoys you most?
> >
> > —
> > Denis
> >
> > > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <
> [hidden email]>
> > wrote:
> > >
> > > We have a lot of failed tests, which is frustrating. Some of them are
> > > flaky(floating status randomly goes from succesful to failed) which
> adds
> > to
> > > frustration. Perhaps, we should fix all the tests in first place, and
> > then
> > > continue doing tickets ?
> > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> >
> > --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>

[Александр Меньшиков]

+1 to Aleksey, Alexander and Vyacheslav.

I suppose that the best option is make issue for every master-failed-test.
And fix them all.

Floating tests should be normal. I think in most case we can just add
repeating.

All new test for some not ready future should be marked like "Should be fix
in IGNITE-9999".


2017-02-08 15:23 GMT+03:00 Vyacheslav Daradur <[hidden email]>:

> I vote for the master-branche without failed-tests)
>
> I understand that impossible to make it quickly.
>
> We shall aim at this approach.
>
> It will be more comfortable to us to develop.
>
> 2017-02-08 12:17 GMT+03:00 Alexander Fedotov <[hidden email]
> >:
>
> > Hi,
> >
> > I would agree with Aleksey.
> > From the CI perspective, failing tests should be the main concern,
> because
> > it prevents a durable development of new features. Also, as Aleksey has
> > noted, developers working on different features could end up fixing the
> > same regressions, chances are - in different ways, resulting in merge
> > conflicts.
> > Having failing base branch, it is not possible to determine the reason
> of a
> > failure from the first sight.
> > All these points impact as the feedback from tests, so the whole process
> > agility.
> > Ideally, no new feature development should be started based on the
> failing
> > branch.
> > I think we should adopt this approach, otherwise, with growing amount of
> > features,
> > we will eventually end up spending more time dealing with regressions,
> than
> > developing features.
> >
> > Regards,
> > Alexander
> >
> > 8 февр. 2017 г. 10:50 AM пользователь "ALEKSEY KUZNETSOV" <
> > [hidden email]> написал:
> >
> > How could they co-exist ? When you developing some ticket you are risking
> > introduce bug which is reproduced by already failed test(s).
> > Moreover its time consuming to look up new failed tests when your build
> has
> > completed.
> > The last one, committers who introduced new bugs is responsible for them
> > and have to fix them, not other ones.
> > There are number of tests which i have to execute to ensure they are
> flaky
> > or permanently failed. For example this one :
> > CacheJdbcPojoStoreTest.testLoadCache()
> >
> >
> >
> > вт, 7 февр. 2017 г. в 22:10, Denis Magda <[hidden email]>:
> >
> > > Aleksey,
> > >
> > > Bugs fixing and features development are two processes that usually
> > > co-exist.
> > >
> > > Some of the committer/contributors fix tests/functionality while the
> > > others add new functionality. Someone does both.
> > >
> > > You’re welcomed to start fixing the failing tests. Are there any
> specific
> > > that annoys you most?
> > >
> > > —
> > > Denis
> > >
> > > > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <
> > [hidden email]>
> > > wrote:
> > > >
> > > > We have a lot of failed tests, which is frustrating. Some of them are
> > > > flaky(floating status randomly goes from succesful to failed) which
> > adds
> > > to
> > > > frustration. Perhaps, we should fix all the tests in first place, and
> > > then
> > > > continue doing tickets ?
> > > > --
> > > >
> > > > *Best Regards,*
> > > >
> > > > *Kuznetsov Aleksey*
> > >
> > > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
>

[dsetrakyan]

I also agree. I think we should create tickets for all failing tests and
have the community grab them.

On Wed, Feb 8, 2017 at 5:50 AM, Александр Меньшиков <[hidden email]>
wrote:

> +1 to Aleksey, Alexander and Vyacheslav.
>
> I suppose that the best option is make issue for every master-failed-test.
> And fix them all.
>
> Floating tests should be normal. I think in most case we can just add
> repeating.
>
> All new test for some not ready future should be marked like "Should be fix
> in IGNITE-9999".
>
>
> 2017-02-08 15:23 GMT+03:00 Vyacheslav Daradur <[hidden email]>:
>
> > I vote for the master-branche without failed-tests)
> >
> > I understand that impossible to make it quickly.
> >
> > We shall aim at this approach.
> >
> > It will be more comfortable to us to develop.
> >
> > 2017-02-08 12:17 GMT+03:00 Alexander Fedotov <
> [hidden email]
> > >:
> >
> > > Hi,
> > >
> > > I would agree with Aleksey.
> > > From the CI perspective, failing tests should be the main concern,
> > because
> > > it prevents a durable development of new features. Also, as Aleksey has
> > > noted, developers working on different features could end up fixing the
> > > same regressions, chances are - in different ways, resulting in merge
> > > conflicts.
> > > Having failing base branch, it is not possible to determine the reason
> > of a
> > > failure from the first sight.
> > > All these points impact as the feedback from tests, so the whole
> process
> > > agility.
> > > Ideally, no new feature development should be started based on the
> > failing
> > > branch.
> > > I think we should adopt this approach, otherwise, with growing amount
> of
> > > features,
> > > we will eventually end up spending more time dealing with regressions,
> > than
> > > developing features.
> > >
> > > Regards,
> > > Alexander
> > >
> > > 8 февр. 2017 г. 10:50 AM пользователь "ALEKSEY KUZNETSOV" <
> > > [hidden email]> написал:
> > >
> > > How could they co-exist ? When you developing some ticket you are
> risking
> > > introduce bug which is reproduced by already failed test(s).
> > > Moreover its time consuming to look up new failed tests when your build
> > has
> > > completed.
> > > The last one, committers who introduced new bugs is responsible for
> them
> > > and have to fix them, not other ones.
> > > There are number of tests which i have to execute to ensure they are
> > flaky
> > > or permanently failed. For example this one :
> > > CacheJdbcPojoStoreTest.testLoadCache()
> > >
> > >
> > >
> > > вт, 7 февр. 2017 г. в 22:10, Denis Magda <[hidden email]>:
> > >
> > > > Aleksey,
> > > >
> > > > Bugs fixing and features development are two processes that usually
> > > > co-exist.
> > > >
> > > > Some of the committer/contributors fix tests/functionality while the
> > > > others add new functionality. Someone does both.
> > > >
> > > > You’re welcomed to start fixing the failing tests. Are there any
> > specific
> > > > that annoys you most?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <
> > > [hidden email]>
> > > > wrote:
> > > > >
> > > > > We have a lot of failed tests, which is frustrating. Some of them
> are
> > > > > flaky(floating status randomly goes from succesful to failed) which
> > > adds
> > > > to
> > > > > frustration. Perhaps, we should fix all the tests in first place,
> and
> > > > then
> > > > > continue doing tickets ?
> > > > > --
> > > > >
> > > > > *Best Regards,*
> > > > >
> > > > > *Kuznetsov Aleksey*
> > > >
> > > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> >
>

[dmagda]

> How could they co-exist ? When you developing some ticket you are risking
> introduce bug which is reproduced by already failed test(s).

Because there is no IT project that is bug-free. In my experience, it’s a usual situation when a part of the system is being evolved while the other is being fixed - this is why both feature development & bug fixing co-exist.

In any case, I do believe that most of the tests that fail used to be sporadic ones when a modification was applied and a contributor had been validating his changes. These casual failures can easily lead to the situation when the contributor commits the changes that introduce new bugs.

We should have JIRA tickets for most of the test failures:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20Ignite%20and%20summary~%22test%22%20and%20status%20!%3D%20closed <https://issues.apache.org/jira/issues/?jql=project%20=%20Ignite%20and%20summary~%22test%22%20and%20status%20!=%20closed>

Would you be interested picking some of them making the product more stable? Basically, this should help your the internals even more profoundly.

—
Denis

> On Feb 7, 2017, at 11:50 PM, ALEKSEY KUZNETSOV <[hidden email]> wrote:
>
> How could they co-exist ? When you developing some ticket you are risking
> introduce bug which is reproduced by already failed test(s).
> Moreover its time consuming to look up new failed tests when your build has
> completed.
> The last one, committers who introduced new bugs is responsible for them
> and have to fix them, not other ones.
> There are number of tests which i have to execute to ensure they are flaky
> or permanently failed. For example this one :
> CacheJdbcPojoStoreTest.testLoadCache()
>
>
>
> вт, 7 февр. 2017 г. в 22:10, Denis Magda <[hidden email]>:
>
>> Aleksey,
>>
>> Bugs fixing and features development are two processes that usually
>> co-exist.
>>
>> Some of the committer/contributors fix tests/functionality while the
>> others add new functionality. Someone does both.
>>
>> You’re welcomed to start fixing the failing tests. Are there any specific
>> that annoys you most?
>>
>> —
>> Denis
>>
>>> On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <[hidden email]>
>> wrote:
>>>
>>> We have a lot of failed tests, which is frustrating. Some of them are
>>> flaky(floating status randomly goes from succesful to failed) which adds
>> to
>>> frustration. Perhaps, we should fix all the tests in first place, and
>> then
>>> continue doing tickets ?
>>> --
>>>
>>> *Best Regards,*
>>>
>>> *Kuznetsov Aleksey*
>>
>> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*

[voipp]

yeah, i will

чт, 9 февр. 2017 г. в 0:01, Denis Magda <[hidden email]>:

> > How could they co-exist ? When you developing some ticket you are risking
> > introduce bug which is reproduced by already failed test(s).
>
> Because there is no IT project that is bug-free. In my experience, it’s a
> usual situation when a part of the system is being evolved while the other
> is being fixed - this is why both feature development & bug fixing co-exist.
>
> In any case, I do believe that most of the tests that fail used to be
> sporadic ones when a modification was applied and a contributor had been
> validating his changes. These casual failures can easily lead to the
> situation when the contributor commits the changes that introduce new bugs.
>
> We should have JIRA tickets for most of the test failures:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Ignite%20and%20summary~%22test%22%20and%20status%20!%3D%20closed
> <
> https://issues.apache.org/jira/issues/?jql=project%20=%20Ignite%20and%20summary~%22test%22%20and%20status%20!=%20closed
> >
>
> Would you be interested picking some of them making the product more
> stable? Basically, this should help your the internals even more profoundly.
>
> —
> Denis
>
> > On Feb 7, 2017, at 11:50 PM, ALEKSEY KUZNETSOV <[hidden email]>
> wrote:
> >
> > How could they co-exist ? When you developing some ticket you are risking
> > introduce bug which is reproduced by already failed test(s).
> > Moreover its time consuming to look up new failed tests when your build
> has
> > completed.
> > The last one, committers who introduced new bugs is responsible for them
> > and have to fix them, not other ones.
> > There are number of tests which i have to execute to ensure they are
> flaky
> > or permanently failed. For example this one :
> > CacheJdbcPojoStoreTest.testLoadCache()
> >
> >
> >
> > вт, 7 февр. 2017 г. в 22:10, Denis Magda <[hidden email]>:
> >
> >> Aleksey,
> >>
> >> Bugs fixing and features development are two processes that usually
> >> co-exist.
> >>
> >> Some of the committer/contributors fix tests/functionality while the
> >> others add new functionality. Someone does both.
> >>
> >> You’re welcomed to start fixing the failing tests. Are there any
> specific
> >> that annoys you most?
> >>
> >> —
> >> Denis
> >>
> >>> On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <
> [hidden email]>
> >> wrote:
> >>>
> >>> We have a lot of failed tests, which is frustrating. Some of them are
> >>> flaky(floating status randomly goes from succesful to failed) which
> adds
> >> to
> >>> frustration. Perhaps, we should fix all the tests in first place, and
> >> then
> >>> continue doing tickets ?
> >>> --
> >>>
> >>> *Best Regards,*
> >>>
> >>> *Kuznetsov Aleksey*
> >>
> >> --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
>
> --

*Best Regards,*

*Kuznetsov Aleksey*