Workflow improvement
Locked
 
|
Hello, Igniters!
I want to suggest improvement for TeamCity Helper [1] – we need an easy way to get list of failed tests that don’t fall in the master branch. These tests should: * fail in the PR * not fail in the master * not be flaky. Also, I want to suggest to create a GitHub plugin, which will notify PR if it has such tests. PR will have a marker, which allows/prohibits merge. This marker will be shown near PR conflicts. Allowing marker will be shown in case: * no new fails. Prohibiting marker will be shown in cases: * new fails – tests must be fixed. * new timed out test suite – suite should be restarted or tests must be fixed. * runAll wasn’t launched – tests must be launched. This will make test checks much faster and easier. Also, this will decrease the number of merges with new failed tests made by inattention to the tests. Further, we can expand the plugin by adding new checks, showing PR quality. [1] https://github.com/apache/ignite-teamcity-bot |
|
Hi Dmitrii,
I'm not sure we're able to install Github apps to Apache mirrors. The simplest solution, what can be as efficient as a plugin, is fake MTCGA bot account in Github, which will provide PR comments using Github program interface. What do you think? A new test failure can be identified by the Ignite TC Bot by master recent fail rate = 0.0%. The same rule can be applied to timed out suites. Sincerely, Dmitriy Pavlov вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <[hidden email]>: > Hello, Igniters! > > I want to suggest improvement for TeamCity Helper [1] – we need an easy way > to get list of failed tests that don’t fall in the master branch. These > tests should: > * fail in the PR > * not fail in the master > * not be flaky. > > Also, I want to suggest to create a GitHub plugin, which will notify PR if > it has such tests. PR will have a marker, which allows/prohibits merge. > This marker will be shown near PR conflicts. > > Allowing marker will be shown in case: > * no new fails. > > Prohibiting marker will be shown in cases: > * new fails – tests must be fixed. > * new timed out test suite – suite should be restarted or tests must be > fixed. > * runAll wasn’t launched – tests must be launched. > > This will make test checks much faster and easier. Also, this will decrease > the number of merges with new failed tests made by inattention to the > tests. > > Further, we can expand the plugin by adding new checks, showing PR quality. > > [1] https://github.com/apache/ignite-teamcity-bot > |
|
|
I think plugin will be more pretty looking, but comments can contain any
information, so they can be more usefull. I agree with your idea to create bot instead of plugin. As for fail rate - I'm not sure it is working as you describe. I'm looking on my runAll [1]. There is `IgniteCacheGroupsTest.testCacheApiTxReplicated` in `Cache 3` suite with fail rate = 0.0%. But it is flaky and fails in master branch [2]. [1] https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest [2] https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&buildTypeId=&tab=testDetails&testNameId=5628470782089555961&order=TEST_STATUS_DESC&branch_IgniteTests24Java8=__all_branches__&itemsCount=50 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > Hi Dmitrii, > > I'm not sure we're able to install Github apps to Apache mirrors. > > The simplest solution, what can be as efficient as a plugin, is fake MTCGA > bot account in Github, which will provide PR comments using Github program > interface. What do you think? > > A new test failure can be identified by the Ignite TC Bot by master recent > fail rate = 0.0%. The same rule can be applied to timed out suites. > > Sincerely, > Dmitriy Pavlov > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <[hidden email]>: > > > Hello, Igniters! > > > > I want to suggest improvement for TeamCity Helper [1] – we need an easy > way > > to get list of failed tests that don’t fall in the master branch. These > > tests should: > > * fail in the PR > > * not fail in the master > > * not be flaky. > > > > Also, I want to suggest to create a GitHub plugin, which will notify PR > if > > it has such tests. PR will have a marker, which allows/prohibits merge. > > This marker will be shown near PR conflicts. > > > > Allowing marker will be shown in case: > > * no new fails. > > > > Prohibiting marker will be shown in cases: > > * new fails – tests must be fixed. > > * new timed out test suite – suite should be restarted or tests must be > > fixed. > > * runAll wasn’t launched – tests must be launched. > > > > This will make test checks much faster and easier. Also, this will > decrease > > the number of merges with new failed tests made by inattention to the > > tests. > > > > Further, we can expand the plugin by adding new checks, showing PR > quality. > > > > [1] https://github.com/apache/ignite-teamcity-bot > > > |
|
|
Hi Dmitriy,
The Bot is able to detect a frequent change of test status, but currently only 50 last runs count. Same is true for the failure rate. This value can be easily changed to 70 or 100, moreover, the auto trigger feature gives us much more builds. We can improve these rules. We can add not only status change, but status change without any code changes. We can somehow save this data in RunStat class. Let's create a better rule, and later we can code it. Sincerely, Dmitriy Pavlov вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <[hidden email]>: > I think plugin will be more pretty looking, but comments can contain any > information, so they can be more usefull. I agree with your idea to create > bot instead of plugin. > > As for fail rate - I'm not sure it is working as you describe. > I'm looking on my runAll [1]. There is > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and fails in > master branch [2]. > > [1] > > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest > [2] > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&buildTypeId=&tab=testDetails&testNameId=5628470782089555961&order=TEST_STATUS_DESC&branch_IgniteTests24Java8=__all_branches__&itemsCount=50 > > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > Hi Dmitrii, > > > > I'm not sure we're able to install Github apps to Apache mirrors. > > > > The simplest solution, what can be as efficient as a plugin, is fake > MTCGA > > bot account in Github, which will provide PR comments using Github > program > > interface. What do you think? > > > > A new test failure can be identified by the Ignite TC Bot by master > recent > > fail rate = 0.0%. The same rule can be applied to timed out suites. > > > > Sincerely, > > Dmitriy Pavlov > > > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <[hidden email]>: > > > > > Hello, Igniters! > > > > > > I want to suggest improvement for TeamCity Helper [1] – we need an easy > > way > > > to get list of failed tests that don’t fall in the master branch. These > > > tests should: > > > * fail in the PR > > > * not fail in the master > > > * not be flaky. > > > > > > Also, I want to suggest to create a GitHub plugin, which will notify PR > > if > > > it has such tests. PR will have a marker, which allows/prohibits merge. > > > This marker will be shown near PR conflicts. > > > > > > Allowing marker will be shown in case: > > > * no new fails. > > > > > > Prohibiting marker will be shown in cases: > > > * new fails – tests must be fixed. > > > * new timed out test suite – suite should be restarted or tests must be > > > fixed. > > > * runAll wasn’t launched – tests must be launched. > > > > > > This will make test checks much faster and easier. Also, this will > > decrease > > > the number of merges with new failed tests made by inattention to the > > > tests. > > > > > > Further, we can expand the plugin by adding new checks, showing PR > > quality. > > > > > > [1] https://github.com/apache/ignite-teamcity-bot > > > > > > |
|
|
I think we should check not N last runs, but all runs in last N days.
A simple rule to detect flaky fails by hands - get test history ordered by TEST_STATUS_DESC and check its date. As I see, we can get list of failures from TC. We don't need to check successfull runs, because they are uninformative for our needs. 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > Hi Dmitriy, > > The Bot is able to detect a frequent change of test status, but currently > only 50 last runs count. Same is true for the failure rate. > > This value can be easily changed to 70 or 100, moreover, the auto > trigger feature gives us much more builds. > > We can improve these rules. We can add not only status change, but status > change without any code changes. We can somehow save this data in RunStat > class. Let's create a better rule, and later we can code it. > > Sincerely, > Dmitriy Pavlov > > вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <[hidden email]>: > > > I think plugin will be more pretty looking, but comments can contain any > > information, so they can be more usefull. I agree with your idea to > create > > bot instead of plugin. > > > > As for fail rate - I'm not sure it is working as you describe. > > I'm looking on my runAll [1]. There is > > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and fails in > > master branch [2]. > > > > [1] > > > > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId= > IgniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest > > [2] > > > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8& > buildTypeId=&tab=testDetails&testNameId=5628470782089555961&order= > TEST_STATUS_DESC&branch_IgniteTests24Java8=__all_branches__&itemsCount=50 > > > > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > Hi Dmitrii, > > > > > > I'm not sure we're able to install Github apps to Apache mirrors. > > > > > > The simplest solution, what can be as efficient as a plugin, is fake > > MTCGA > > > bot account in Github, which will provide PR comments using Github > > program > > > interface. What do you think? > > > > > > A new test failure can be identified by the Ignite TC Bot by master > > recent > > > fail rate = 0.0%. The same rule can be applied to timed out suites. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <[hidden email]>: > > > > > > > Hello, Igniters! > > > > > > > > I want to suggest improvement for TeamCity Helper [1] – we need an > easy > > > way > > > > to get list of failed tests that don’t fall in the master branch. > These > > > > tests should: > > > > * fail in the PR > > > > * not fail in the master > > > > * not be flaky. > > > > > > > > Also, I want to suggest to create a GitHub plugin, which will notify > PR > > > if > > > > it has such tests. PR will have a marker, which allows/prohibits > merge. > > > > This marker will be shown near PR conflicts. > > > > > > > > Allowing marker will be shown in case: > > > > * no new fails. > > > > > > > > Prohibiting marker will be shown in cases: > > > > * new fails – tests must be fixed. > > > > * new timed out test suite – suite should be restarted or tests must > be > > > > fixed. > > > > * runAll wasn’t launched – tests must be launched. > > > > > > > > This will make test checks much faster and easier. Also, this will > > > decrease > > > > the number of merges with new failed tests made by inattention to the > > > > tests. > > > > > > > > Further, we can expand the plugin by adding new checks, showing PR > > > quality. > > > > > > > > [1] https://github.com/apache/ignite-teamcity-bot > > > > > > > > > > |
|
|
Hi, Dmitriy,
I propose the next steps: 1. Show current 0% tests in a separate table at the top of the analysis results page. Thus, we'll see most suspicious (new or very flaky) tests firstly. Or we can hide other tests under "More >>" button, like top long running tests. 2. Create a button by clicking on which the info about 0% tests will be written in the PR. 3. Replace button by webhook for TeamCity (for Run All), which will start analysis on TCH and write results in the PR. 4. ... What do you think? 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > I think we should check not N last runs, but all runs in last N days. > > A simple rule to detect flaky fails by hands - get test history ordered > by TEST_STATUS_DESC and check its date. As I see, we can get list of > failures from TC. We don't need to check successfull runs, because they are > uninformative for our needs. > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > >> Hi Dmitriy, >> >> The Bot is able to detect a frequent change of test status, but currently >> only 50 last runs count. Same is true for the failure rate. >> >> This value can be easily changed to 70 or 100, moreover, the auto >> trigger feature gives us much more builds. >> >> We can improve these rules. We can add not only status change, but status >> change without any code changes. We can somehow save this data in RunStat >> class. Let's create a better rule, and later we can code it. >> >> Sincerely, >> Dmitriy Pavlov >> >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <[hidden email]>: >> >> > I think plugin will be more pretty looking, but comments can contain any >> > information, so they can be more usefull. I agree with your idea to >> create >> > bot instead of plugin. >> > >> > As for fail rate - I'm not sure it is working as you describe. >> > I'm looking on my runAll [1]. There is >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and fails in >> > master branch [2]. >> > >> > [1] >> > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I >> gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest >> > [2] >> > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests >> 24Java8=__all_branches__&itemsCount=50 >> > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: >> > >> > > Hi Dmitrii, >> > > >> > > I'm not sure we're able to install Github apps to Apache mirrors. >> > > >> > > The simplest solution, what can be as efficient as a plugin, is fake >> > MTCGA >> > > bot account in Github, which will provide PR comments using Github >> > program >> > > interface. What do you think? >> > > >> > > A new test failure can be identified by the Ignite TC Bot by master >> > recent >> > > fail rate = 0.0%. The same rule can be applied to timed out suites. >> > > >> > > Sincerely, >> > > Dmitriy Pavlov >> > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <[hidden email]>: >> > > >> > > > Hello, Igniters! >> > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we need an >> easy >> > > way >> > > > to get list of failed tests that don’t fall in the master branch. >> These >> > > > tests should: >> > > > * fail in the PR >> > > > * not fail in the master >> > > > * not be flaky. >> > > > >> > > > Also, I want to suggest to create a GitHub plugin, which will >> notify PR >> > > if >> > > > it has such tests. PR will have a marker, which allows/prohibits >> merge. >> > > > This marker will be shown near PR conflicts. >> > > > >> > > > Allowing marker will be shown in case: >> > > > * no new fails. >> > > > >> > > > Prohibiting marker will be shown in cases: >> > > > * new fails – tests must be fixed. >> > > > * new timed out test suite – suite should be restarted or tests >> must be >> > > > fixed. >> > > > * runAll wasn’t launched – tests must be launched. >> > > > >> > > > This will make test checks much faster and easier. Also, this will >> > > decrease >> > > > the number of merges with new failed tests made by inattention to >> the >> > > > tests. >> > > > >> > > > Further, we can expand the plugin by adding new checks, showing PR >> > > quality. >> > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot >> > > > >> > > >> > >> > > |
|
|
Hi Dmitriy,
Sounds like a plan ;) I totally agree. Just one proposal: I would like to avoid hiding any test failures. 2 separate tables named 'Possible Blockers' and 'Other failures' should be completely clear. Comment to PR can contain only first category. We had a private discussion with Anton K. and he proposed a very interesting thing, I would like to bring it here. We can add configuration into TC bot and mark some tests and some suites as 'Known Stable' It means that suite or test failure should be considered as a blocker for merge every time it fails, even if fail rate is non-zero. Very first list of such suites are - Build Apache Ignite - License check And tests: - .Net API Parity check Please share your vision. Meanwhile, I've updated the TC bot. - Underlying Apache Ignite DB was refactored to use lower partitions count, so restart should be faster. - Now 100 builds are saved as our failure rate statistics basis. Flakiness border was not changed, so more test now will be considered as flaky. - Now 4 builds required to be failed or timed out in a row before notification is sent. Sincerely, Dmitriy Pavlov чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <[hidden email]>: > Hi, Dmitriy, > > I propose the next steps: > > 1. Show current 0% tests in a separate table at the top of the analysis > results page. Thus, we'll see most suspicious (new or very flaky) tests > firstly. Or we can hide other tests under "More >>" button, like top long > running tests. > 2. Create a button by clicking on which the info about 0% tests will be > written in the PR. > 3. Replace button by webhook for TeamCity (for Run All), which will start > analysis on TCH and write results in the PR. > 4. ... > > What do you think? > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > > > I think we should check not N last runs, but all runs in last N days. > > > > A simple rule to detect flaky fails by hands - get test history ordered > > by TEST_STATUS_DESC and check its date. As I see, we can get list of > > failures from TC. We don't need to check successfull runs, because they > are > > uninformative for our needs. > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > >> Hi Dmitriy, > >> > >> The Bot is able to detect a frequent change of test status, but > currently > >> only 50 last runs count. Same is true for the failure rate. > >> > >> This value can be easily changed to 70 or 100, moreover, the auto > >> trigger feature gives us much more builds. > >> > >> We can improve these rules. We can add not only status change, but > status > >> change without any code changes. We can somehow save this data in > RunStat > >> class. Let's create a better rule, and later we can code it. > >> > >> Sincerely, > >> Dmitriy Pavlov > >> > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <[hidden email]>: > >> > >> > I think plugin will be more pretty looking, but comments can contain > any > >> > information, so they can be more usefull. I agree with your idea to > >> create > >> > bot instead of plugin. > >> > > >> > As for fail rate - I'm not sure it is working as you describe. > >> > I'm looking on my runAll [1]. There is > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and fails in > >> > master branch [2]. > >> > > >> > [1] > >> > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I > >> gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest > >> > [2] > >> > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > >> 24Java8=__all_branches__&itemsCount=50 > >> > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > >> > > >> > > Hi Dmitrii, > >> > > > >> > > I'm not sure we're able to install Github apps to Apache mirrors. > >> > > > >> > > The simplest solution, what can be as efficient as a plugin, is fake > >> > MTCGA > >> > > bot account in Github, which will provide PR comments using Github > >> > program > >> > > interface. What do you think? > >> > > > >> > > A new test failure can be identified by the Ignite TC Bot by master > >> > recent > >> > > fail rate = 0.0%. The same rule can be applied to timed out suites. > >> > > > >> > > Sincerely, > >> > > Dmitriy Pavlov > >> > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <[hidden email] > >: > >> > > > >> > > > Hello, Igniters! > >> > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we need an > >> easy > >> > > way > >> > > > to get list of failed tests that don’t fall in the master branch. > >> These > >> > > > tests should: > >> > > > * fail in the PR > >> > > > * not fail in the master > >> > > > * not be flaky. > >> > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which will > >> notify PR > >> > > if > >> > > > it has such tests. PR will have a marker, which allows/prohibits > >> merge. > >> > > > This marker will be shown near PR conflicts. > >> > > > > >> > > > Allowing marker will be shown in case: > >> > > > * no new fails. > >> > > > > >> > > > Prohibiting marker will be shown in cases: > >> > > > * new fails – tests must be fixed. > >> > > > * new timed out test suite – suite should be restarted or tests > >> must be > >> > > > fixed. > >> > > > * runAll wasn’t launched – tests must be launched. > >> > > > > >> > > > This will make test checks much faster and easier. Also, this will > >> > > decrease > >> > > > the number of merges with new failed tests made by inattention to > >> the > >> > > > tests. > >> > > > > >> > > > Further, we can expand the plugin by adding new checks, showing PR > >> > > quality. > >> > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > >> > > > > >> > > > >> > > >> > > > > > |
|
|
Hi, Dmitriy,
Good, I'll create tickets for this steps. Stable suites are a good idea, but also we need to do some actions when a flaky test will appear in stable suite of master branch run. To find out a guilty commit, we should run previous commits on empty TeamCity's queue. This runs will reduce bot's statistics for the latest commits. 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > Hi Dmitriy, > > Sounds like a plan ;) I totally agree. > Just one proposal: I would like to avoid hiding any test failures. 2 > separate tables named 'Possible Blockers' and 'Other failures' should be > completely clear. Comment to PR can contain only first category. > > We had a private discussion with Anton K. and he proposed a very > interesting thing, I would like to bring it here. We can add configuration > into TC bot and mark some tests and some suites as 'Known Stable' It means > that suite or test failure should be considered as a blocker for merge > every time it fails, even if fail rate is non-zero. Very first list of such > suites are > - Build Apache Ignite > - License check > And tests: > - .Net API Parity check > Please share your vision. > > Meanwhile, I've updated the TC bot. > - Underlying Apache Ignite DB was refactored to use lower partitions count, > so restart should be faster. > - Now 100 builds are saved as our failure rate statistics basis. Flakiness > border was not changed, so more test now will be considered as flaky. > - Now 4 builds required to be failed or timed out in a row before > notification is sent. > > Sincerely, > Dmitriy Pavlov > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <[hidden email]>: > > > Hi, Dmitriy, > > > > I propose the next steps: > > > > 1. Show current 0% tests in a separate table at the top of the analysis > > results page. Thus, we'll see most suspicious (new or very flaky) tests > > firstly. Or we can hide other tests under "More >>" button, like top long > > running tests. > > 2. Create a button by clicking on which the info about 0% tests will be > > written in the PR. > > 3. Replace button by webhook for TeamCity (for Run All), which will start > > analysis on TCH and write results in the PR. > > 4. ... > > > > What do you think? > > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > > > > > I think we should check not N last runs, but all runs in last N days. > > > > > > A simple rule to detect flaky fails by hands - get test history ordered > > > by TEST_STATUS_DESC and check its date. As I see, we can get list of > > > failures from TC. We don't need to check successfull runs, because they > > are > > > uninformative for our needs. > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > >> Hi Dmitriy, > > >> > > >> The Bot is able to detect a frequent change of test status, but > > currently > > >> only 50 last runs count. Same is true for the failure rate. > > >> > > >> This value can be easily changed to 70 or 100, moreover, the auto > > >> trigger feature gives us much more builds. > > >> > > >> We can improve these rules. We can add not only status change, but > > status > > >> change without any code changes. We can somehow save this data in > > RunStat > > >> class. Let's create a better rule, and later we can code it. > > >> > > >> Sincerely, > > >> Dmitriy Pavlov > > >> > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <[hidden email]>: > > >> > > >> > I think plugin will be more pretty looking, but comments can contain > > any > > >> > information, so they can be more usefull. I agree with your idea to > > >> create > > >> > bot instead of plugin. > > >> > > > >> > As for fail rate - I'm not sure it is working as you describe. > > >> > I'm looking on my runAll [1]. There is > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and fails > in > > >> > master branch [2]. > > >> > > > >> > [1] > > >> > > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I > > >> gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest > > >> > [2] > > >> > > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > > >> 24Java8=__all_branches__&itemsCount=50 > > >> > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > >> > > > >> > > Hi Dmitrii, > > >> > > > > >> > > I'm not sure we're able to install Github apps to Apache mirrors. > > >> > > > > >> > > The simplest solution, what can be as efficient as a plugin, is > fake > > >> > MTCGA > > >> > > bot account in Github, which will provide PR comments using Github > > >> > program > > >> > > interface. What do you think? > > >> > > > > >> > > A new test failure can be identified by the Ignite TC Bot by > master > > >> > recent > > >> > > fail rate = 0.0%. The same rule can be applied to timed out > suites. > > >> > > > > >> > > Sincerely, > > >> > > Dmitriy Pavlov > > >> > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < > [hidden email] > > >: > > >> > > > > >> > > > Hello, Igniters! > > >> > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we need > an > > >> easy > > >> > > way > > >> > > > to get list of failed tests that don’t fall in the master > branch. > > >> These > > >> > > > tests should: > > >> > > > * fail in the PR > > >> > > > * not fail in the master > > >> > > > * not be flaky. > > >> > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which will > > >> notify PR > > >> > > if > > >> > > > it has such tests. PR will have a marker, which allows/prohibits > > >> merge. > > >> > > > This marker will be shown near PR conflicts. > > >> > > > > > >> > > > Allowing marker will be shown in case: > > >> > > > * no new fails. > > >> > > > > > >> > > > Prohibiting marker will be shown in cases: > > >> > > > * new fails – tests must be fixed. > > >> > > > * new timed out test suite – suite should be restarted or tests > > >> must be > > >> > > > fixed. > > >> > > > * runAll wasn’t launched – tests must be launched. > > >> > > > > > >> > > > This will make test checks much faster and easier. Also, this > will > > >> > > decrease > > >> > > > the number of merges with new failed tests made by inattention > to > > >> the > > >> > > > tests. > > >> > > > > > >> > > > Further, we can expand the plugin by adding new checks, showing > PR > > >> > > quality. > > >> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > > |
|
|
Not sure I understood what bot statistic reduction means.
As far as I understood, you also mean locating failure automatically with re-runs on specific git revisions (hashes). I found it will be a good contribution to TC Bot for both flaky and non-flaky failures detection. E.g failure occurred after 10 commits merged to master. How to find out bad commit? Automatic location (analog of bisecting, but using TC test) will be really helpful. But it is not a simple contribution. пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <[hidden email]>: > Hi, Dmitriy, > > Good, I'll create tickets for this steps. > > Stable suites are a good idea, but also we need to do some actions when a > flaky test will appear in stable suite of master branch run. To find out a > guilty commit, we should run previous commits on empty TeamCity's queue. > This runs will reduce bot's statistics for the latest commits. > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > Hi Dmitriy, > > > > Sounds like a plan ;) I totally agree. > > Just one proposal: I would like to avoid hiding any test failures. 2 > > separate tables named 'Possible Blockers' and 'Other failures' should be > > completely clear. Comment to PR can contain only first category. > > > > We had a private discussion with Anton K. and he proposed a very > > interesting thing, I would like to bring it here. We can add > configuration > > into TC bot and mark some tests and some suites as 'Known Stable' It > means > > that suite or test failure should be considered as a blocker for merge > > every time it fails, even if fail rate is non-zero. Very first list of > such > > suites are > > - Build Apache Ignite > > - License check > > And tests: > > - .Net API Parity check > > Please share your vision. > > > > Meanwhile, I've updated the TC bot. > > - Underlying Apache Ignite DB was refactored to use lower partitions > count, > > so restart should be faster. > > - Now 100 builds are saved as our failure rate statistics basis. > Flakiness > > border was not changed, so more test now will be considered as flaky. > > - Now 4 builds required to be failed or timed out in a row before > > notification is sent. > > > > Sincerely, > > Dmitriy Pavlov > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <[hidden email]>: > > > > > Hi, Dmitriy, > > > > > > I propose the next steps: > > > > > > 1. Show current 0% tests in a separate table at the top of the analysis > > > results page. Thus, we'll see most suspicious (new or very flaky) tests > > > firstly. Or we can hide other tests under "More >>" button, like top > long > > > running tests. > > > 2. Create a button by clicking on which the info about 0% tests will be > > > written in the PR. > > > 3. Replace button by webhook for TeamCity (for Run All), which will > start > > > analysis on TCH and write results in the PR. > > > 4. ... > > > > > > What do you think? > > > > > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > > > > > > > I think we should check not N last runs, but all runs in last N days. > > > > > > > > A simple rule to detect flaky fails by hands - get test history > ordered > > > > by TEST_STATUS_DESC and check its date. As I see, we can get list of > > > > failures from TC. We don't need to check successfull runs, because > they > > > are > > > > uninformative for our needs. > > > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > > > >> Hi Dmitriy, > > > >> > > > >> The Bot is able to detect a frequent change of test status, but > > > currently > > > >> only 50 last runs count. Same is true for the failure rate. > > > >> > > > >> This value can be easily changed to 70 or 100, moreover, the auto > > > >> trigger feature gives us much more builds. > > > >> > > > >> We can improve these rules. We can add not only status change, but > > > status > > > >> change without any code changes. We can somehow save this data in > > > RunStat > > > >> class. Let's create a better rule, and later we can code it. > > > >> > > > >> Sincerely, > > > >> Dmitriy Pavlov > > > >> > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <[hidden email] > >: > > > >> > > > >> > I think plugin will be more pretty looking, but comments can > contain > > > any > > > >> > information, so they can be more usefull. I agree with your idea > to > > > >> create > > > >> > bot instead of plugin. > > > >> > > > > >> > As for fail rate - I'm not sure it is working as you describe. > > > >> > I'm looking on my runAll [1]. There is > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and > fails > > in > > > >> > master branch [2]. > > > >> > > > > >> > [1] > > > >> > > > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I > > > >> > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest > > > >> > [2] > > > >> > > > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > > > >> 24Java8=__all_branches__&itemsCount=50 > > > >> > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <[hidden email] > >: > > > >> > > > > >> > > Hi Dmitrii, > > > >> > > > > > >> > > I'm not sure we're able to install Github apps to Apache > mirrors. > > > >> > > > > > >> > > The simplest solution, what can be as efficient as a plugin, is > > fake > > > >> > MTCGA > > > >> > > bot account in Github, which will provide PR comments using > Github > > > >> > program > > > >> > > interface. What do you think? > > > >> > > > > > >> > > A new test failure can be identified by the Ignite TC Bot by > > master > > > >> > recent > > > >> > > fail rate = 0.0%. The same rule can be applied to timed out > > suites. > > > >> > > > > > >> > > Sincerely, > > > >> > > Dmitriy Pavlov > > > >> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < > > [hidden email] > > > >: > > > >> > > > > > >> > > > Hello, Igniters! > > > >> > > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we > need > > an > > > >> easy > > > >> > > way > > > >> > > > to get list of failed tests that don’t fall in the master > > branch. > > > >> These > > > >> > > > tests should: > > > >> > > > * fail in the PR > > > >> > > > * not fail in the master > > > >> > > > * not be flaky. > > > >> > > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which will > > > >> notify PR > > > >> > > if > > > >> > > > it has such tests. PR will have a marker, which > allows/prohibits > > > >> merge. > > > >> > > > This marker will be shown near PR conflicts. > > > >> > > > > > > >> > > > Allowing marker will be shown in case: > > > >> > > > * no new fails. > > > >> > > > > > > >> > > > Prohibiting marker will be shown in cases: > > > >> > > > * new fails – tests must be fixed. > > > >> > > > * new timed out test suite – suite should be restarted or > tests > > > >> must be > > > >> > > > fixed. > > > >> > > > * runAll wasn’t launched – tests must be launched. > > > >> > > > > > > >> > > > This will make test checks much faster and easier. Also, this > > will > > > >> > > decrease > > > >> > > > the number of merges with new failed tests made by inattention > > to > > > >> the > > > >> > > > tests. > > > >> > > > > > > >> > > > Further, we can expand the plugin by adding new checks, > showing > > PR > > > >> > > quality. > > > >> > > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > > |
|
|
Hi, Dmitriy,
I created a table with possible blockers [1] for the page with the latest build analysis. Anton K. has reviewed it. Can you check and merge it? Also, I created a button to notify PR about analisis results [2]. I used GitHub statuses (example [3]), but to work it needs a token with push permission. As Apache PMC, can you acquire such token? I think statuses are much more notable than comments. [1] https://issues.apache.org/jira/browse/IGNITE-9376 [2] https://issues.apache.org/jira/browse/IGNITE-9377 [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > Not sure I understood what bot statistic reduction means. > > As far as I understood, you also mean locating failure automatically with > re-runs on specific git revisions (hashes). I found it will be a good > contribution to TC Bot for both flaky and non-flaky failures detection. E.g > failure occurred after 10 commits merged to master. How to find out bad > commit? Automatic location (analog of bisecting, but using TC test) will be > really helpful. But it is not a simple contribution. > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <[hidden email]>: > > > Hi, Dmitriy, > > > > Good, I'll create tickets for this steps. > > > > Stable suites are a good idea, but also we need to do some actions when a > > flaky test will appear in stable suite of master branch run. To find out > a > > guilty commit, we should run previous commits on empty TeamCity's queue. > > This runs will reduce bot's statistics for the latest commits. > > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > Hi Dmitriy, > > > > > > Sounds like a plan ;) I totally agree. > > > Just one proposal: I would like to avoid hiding any test failures. 2 > > > separate tables named 'Possible Blockers' and 'Other failures' should > be > > > completely clear. Comment to PR can contain only first category. > > > > > > We had a private discussion with Anton K. and he proposed a very > > > interesting thing, I would like to bring it here. We can add > > configuration > > > into TC bot and mark some tests and some suites as 'Known Stable' It > > means > > > that suite or test failure should be considered as a blocker for merge > > > every time it fails, even if fail rate is non-zero. Very first list of > > such > > > suites are > > > - Build Apache Ignite > > > - License check > > > And tests: > > > - .Net API Parity check > > > Please share your vision. > > > > > > Meanwhile, I've updated the TC bot. > > > - Underlying Apache Ignite DB was refactored to use lower partitions > > count, > > > so restart should be faster. > > > - Now 100 builds are saved as our failure rate statistics basis. > > Flakiness > > > border was not changed, so more test now will be considered as flaky. > > > - Now 4 builds required to be failed or timed out in a row before > > > notification is sent. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <[hidden email]>: > > > > > > > Hi, Dmitriy, > > > > > > > > I propose the next steps: > > > > > > > > 1. Show current 0% tests in a separate table at the top of the > analysis > > > > results page. Thus, we'll see most suspicious (new or very flaky) > tests > > > > firstly. Or we can hide other tests under "More >>" button, like top > > long > > > > running tests. > > > > 2. Create a button by clicking on which the info about 0% tests will > be > > > > written in the PR. > > > > 3. Replace button by webhook for TeamCity (for Run All), which will > > start > > > > analysis on TCH and write results in the PR. > > > > 4. ... > > > > > > > > What do you think? > > > > > > > > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > > > > > > > > > I think we should check not N last runs, but all runs in last N > days. > > > > > > > > > > A simple rule to detect flaky fails by hands - get test history > > ordered > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get list > of > > > > > failures from TC. We don't need to check successfull runs, because > > they > > > > are > > > > > uninformative for our needs. > > > > > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > > > > > >> Hi Dmitriy, > > > > >> > > > > >> The Bot is able to detect a frequent change of test status, but > > > > currently > > > > >> only 50 last runs count. Same is true for the failure rate. > > > > >> > > > > >> This value can be easily changed to 70 or 100, moreover, the auto > > > > >> trigger feature gives us much more builds. > > > > >> > > > > >> We can improve these rules. We can add not only status change, but > > > > status > > > > >> change without any code changes. We can somehow save this data in > > > > RunStat > > > > >> class. Let's create a better rule, and later we can code it. > > > > >> > > > > >> Sincerely, > > > > >> Dmitriy Pavlov > > > > >> > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov < > [hidden email] > > >: > > > > >> > > > > >> > I think plugin will be more pretty looking, but comments can > > contain > > > > any > > > > >> > information, so they can be more usefull. I agree with your idea > > to > > > > >> create > > > > >> > bot instead of plugin. > > > > >> > > > > > >> > As for fail rate - I'm not sure it is working as you describe. > > > > >> > I'm looking on my runAll [1]. There is > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and > > fails > > > in > > > > >> > master branch [2]. > > > > >> > > > > > >> > [1] > > > > >> > > > > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I > > > > >> > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest > > > > >> > [2] > > > > >> > > > > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > > > > >> 24Java8=__all_branches__&itemsCount=50 > > > > >> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov < > [hidden email] > > >: > > > > >> > > > > > >> > > Hi Dmitrii, > > > > >> > > > > > > >> > > I'm not sure we're able to install Github apps to Apache > > mirrors. > > > > >> > > > > > > >> > > The simplest solution, what can be as efficient as a plugin, > is > > > fake > > > > >> > MTCGA > > > > >> > > bot account in Github, which will provide PR comments using > > Github > > > > >> > program > > > > >> > > interface. What do you think? > > > > >> > > > > > > >> > > A new test failure can be identified by the Ignite TC Bot by > > > master > > > > >> > recent > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed out > > > suites. > > > > >> > > > > > > >> > > Sincerely, > > > > >> > > Dmitriy Pavlov > > > > >> > > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < > > > [hidden email] > > > > >: > > > > >> > > > > > > >> > > > Hello, Igniters! > > > > >> > > > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we > > need > > > an > > > > >> easy > > > > >> > > way > > > > >> > > > to get list of failed tests that don’t fall in the master > > > branch. > > > > >> These > > > > >> > > > tests should: > > > > >> > > > * fail in the PR > > > > >> > > > * not fail in the master > > > > >> > > > * not be flaky. > > > > >> > > > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which > will > > > > >> notify PR > > > > >> > > if > > > > >> > > > it has such tests. PR will have a marker, which > > allows/prohibits > > > > >> merge. > > > > >> > > > This marker will be shown near PR conflicts. > > > > >> > > > > > > > >> > > > Allowing marker will be shown in case: > > > > >> > > > * no new fails. > > > > >> > > > > > > > >> > > > Prohibiting marker will be shown in cases: > > > > >> > > > * new fails – tests must be fixed. > > > > >> > > > * new timed out test suite – suite should be restarted or > > tests > > > > >> must be > > > > >> > > > fixed. > > > > >> > > > * runAll wasn’t launched – tests must be launched. > > > > >> > > > > > > > >> > > > This will make test checks much faster and easier. Also, > this > > > will > > > > >> > > decrease > > > > >> > > > the number of merges with new failed tests made by > inattention > > > to > > > > >> the > > > > >> > > > tests. > > > > >> > > > > > > > >> > > > Further, we can expand the plugin by adding new checks, > > showing > > > PR > > > > >> > > quality. > > > > >> > > > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > |
|
|
Hi Dmitrii,
Great news. I love the fact that you did these improvements. I have limited access to the internet so for now I'm not able to review and merge. I can do it next week. I appreciate your effort. Sincerely, Dmitriy Pavlov пн, 3 сент. 2018 г., 18:20 Dmitrii Ryabov <[hidden email]>: > Hi, Dmitriy, > > I created a table with possible blockers [1] for the page with the latest > build analysis. Anton K. has reviewed it. Can you check and merge it? > > Also, I created a button to notify PR about analisis results [2]. I used > GitHub statuses (example [3]), but to work it needs a token with push > permission. As Apache PMC, can you acquire such token? I think statuses are > much more notable than comments. > > [1] https://issues.apache.org/jira/browse/IGNITE-9376 > [2] https://issues.apache.org/jira/browse/IGNITE-9377 > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls > > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > Not sure I understood what bot statistic reduction means. > > > > As far as I understood, you also mean locating failure automatically with > > re-runs on specific git revisions (hashes). I found it will be a good > > contribution to TC Bot for both flaky and non-flaky failures detection. > E.g > > failure occurred after 10 commits merged to master. How to find out bad > > commit? Automatic location (analog of bisecting, but using TC test) will > be > > really helpful. But it is not a simple contribution. > > > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <[hidden email]>: > > > > > Hi, Dmitriy, > > > > > > Good, I'll create tickets for this steps. > > > > > > Stable suites are a good idea, but also we need to do some actions > when a > > > flaky test will appear in stable suite of master branch run. To find > out > > a > > > guilty commit, we should run previous commits on empty TeamCity's > queue. > > > This runs will reduce bot's statistics for the latest commits. > > > > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > > > Hi Dmitriy, > > > > > > > > Sounds like a plan ;) I totally agree. > > > > Just one proposal: I would like to avoid hiding any test failures. 2 > > > > separate tables named 'Possible Blockers' and 'Other failures' should > > be > > > > completely clear. Comment to PR can contain only first category. > > > > > > > > We had a private discussion with Anton K. and he proposed a very > > > > interesting thing, I would like to bring it here. We can add > > > configuration > > > > into TC bot and mark some tests and some suites as 'Known Stable' It > > > means > > > > that suite or test failure should be considered as a blocker for > merge > > > > every time it fails, even if fail rate is non-zero. Very first list > of > > > such > > > > suites are > > > > - Build Apache Ignite > > > > - License check > > > > And tests: > > > > - .Net API Parity check > > > > Please share your vision. > > > > > > > > Meanwhile, I've updated the TC bot. > > > > - Underlying Apache Ignite DB was refactored to use lower partitions > > > count, > > > > so restart should be faster. > > > > - Now 100 builds are saved as our failure rate statistics basis. > > > Flakiness > > > > border was not changed, so more test now will be considered as flaky. > > > > - Now 4 builds required to be failed or timed out in a row before > > > > notification is sent. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <[hidden email]>: > > > > > > > > > Hi, Dmitriy, > > > > > > > > > > I propose the next steps: > > > > > > > > > > 1. Show current 0% tests in a separate table at the top of the > > analysis > > > > > results page. Thus, we'll see most suspicious (new or very flaky) > > tests > > > > > firstly. Or we can hide other tests under "More >>" button, like > top > > > long > > > > > running tests. > > > > > 2. Create a button by clicking on which the info about 0% tests > will > > be > > > > > written in the PR. > > > > > 3. Replace button by webhook for TeamCity (for Run All), which will > > > start > > > > > analysis on TCH and write results in the PR. > > > > > 4. ... > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > > > > > > > > > > > I think we should check not N last runs, but all runs in last N > > days. > > > > > > > > > > > > A simple rule to detect flaky fails by hands - get test history > > > ordered > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get list > > of > > > > > > failures from TC. We don't need to check successfull runs, > because > > > they > > > > > are > > > > > > uninformative for our needs. > > > > > > > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <[hidden email] > >: > > > > > > > > > > > >> Hi Dmitriy, > > > > > >> > > > > > >> The Bot is able to detect a frequent change of test status, but > > > > > currently > > > > > >> only 50 last runs count. Same is true for the failure rate. > > > > > >> > > > > > >> This value can be easily changed to 70 or 100, moreover, the > auto > > > > > >> trigger feature gives us much more builds. > > > > > >> > > > > > >> We can improve these rules. We can add not only status change, > but > > > > > status > > > > > >> change without any code changes. We can somehow save this data > in > > > > > RunStat > > > > > >> class. Let's create a better rule, and later we can code it. > > > > > >> > > > > > >> Sincerely, > > > > > >> Dmitriy Pavlov > > > > > >> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov < > > [hidden email] > > > >: > > > > > >> > > > > > >> > I think plugin will be more pretty looking, but comments can > > > contain > > > > > any > > > > > >> > information, so they can be more usefull. I agree with your > idea > > > to > > > > > >> create > > > > > >> > bot instead of plugin. > > > > > >> > > > > > > >> > As for fail rate - I'm not sure it is working as you describe. > > > > > >> > I'm looking on my runAll [1]. There is > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and > > > fails > > > > in > > > > > >> > master branch [2]. > > > > > >> > > > > > > >> > [1] > > > > > >> > > > > > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I > > > > > >> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest > > > > > >> > [2] > > > > > >> > > > > > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > > > > > >> 24Java8=__all_branches__&itemsCount=50 > > > > > >> > > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov < > > [hidden email] > > > >: > > > > > >> > > > > > > >> > > Hi Dmitrii, > > > > > >> > > > > > > > >> > > I'm not sure we're able to install Github apps to Apache > > > mirrors. > > > > > >> > > > > > > > >> > > The simplest solution, what can be as efficient as a plugin, > > is > > > > fake > > > > > >> > MTCGA > > > > > >> > > bot account in Github, which will provide PR comments using > > > Github > > > > > >> > program > > > > > >> > > interface. What do you think? > > > > > >> > > > > > > > >> > > A new test failure can be identified by the Ignite TC Bot by > > > > master > > > > > >> > recent > > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed out > > > > suites. > > > > > >> > > > > > > > >> > > Sincerely, > > > > > >> > > Dmitriy Pavlov > > > > > >> > > > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < > > > > [hidden email] > > > > > >: > > > > > >> > > > > > > > >> > > > Hello, Igniters! > > > > > >> > > > > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we > > > need > > > > an > > > > > >> easy > > > > > >> > > way > > > > > >> > > > to get list of failed tests that don’t fall in the master > > > > branch. > > > > > >> These > > > > > >> > > > tests should: > > > > > >> > > > * fail in the PR > > > > > >> > > > * not fail in the master > > > > > >> > > > * not be flaky. > > > > > >> > > > > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which > > will > > > > > >> notify PR > > > > > >> > > if > > > > > >> > > > it has such tests. PR will have a marker, which > > > allows/prohibits > > > > > >> merge. > > > > > >> > > > This marker will be shown near PR conflicts. > > > > > >> > > > > > > > > >> > > > Allowing marker will be shown in case: > > > > > >> > > > * no new fails. > > > > > >> > > > > > > > > >> > > > Prohibiting marker will be shown in cases: > > > > > >> > > > * new fails – tests must be fixed. > > > > > >> > > > * new timed out test suite – suite should be restarted or > > > tests > > > > > >> must be > > > > > >> > > > fixed. > > > > > >> > > > * runAll wasn’t launched – tests must be launched. > > > > > >> > > > > > > > > >> > > > This will make test checks much faster and easier. Also, > > this > > > > will > > > > > >> > > decrease > > > > > >> > > > the number of merges with new failed tests made by > > inattention > > > > to > > > > > >> the > > > > > >> > > > tests. > > > > > >> > > > > > > > > >> > > > Further, we can expand the plugin by adding new checks, > > > showing > > > > PR > > > > > >> > > quality. > > > > > >> > > > > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
In reply to this post by Dmitrii Ryabov
Hi Dmitrii,
I deployed change with blockers summary of failures at the top of PR result page. The Bot is migrating entries now, I hope it will be done in 1-4 hours. I noticed your PRs are created in your own fork, not in https://github.com/apache/ignite-teamcity-bot Could you please create unmerged PR in GitHub mirror so it could be merged after reviewing by the apply-pr script. Sincerely, Dmitriy Pavlov пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <[hidden email]>: > Hi, Dmitriy, > > I created a table with possible blockers [1] for the page with the latest > build analysis. Anton K. has reviewed it. Can you check and merge it? > > Also, I created a button to notify PR about analisis results [2]. I used > GitHub statuses (example [3]), but to work it needs a token with push > permission. As Apache PMC, can you acquire such token? I think statuses are > much more notable than comments. > > [1] https://issues.apache.org/jira/browse/IGNITE-9376 > [2] https://issues.apache.org/jira/browse/IGNITE-9377 > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls > > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > Not sure I understood what bot statistic reduction means. > > > > As far as I understood, you also mean locating failure automatically with > > re-runs on specific git revisions (hashes). I found it will be a good > > contribution to TC Bot for both flaky and non-flaky failures detection. > E.g > > failure occurred after 10 commits merged to master. How to find out bad > > commit? Automatic location (analog of bisecting, but using TC test) will > be > > really helpful. But it is not a simple contribution. > > > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <[hidden email]>: > > > > > Hi, Dmitriy, > > > > > > Good, I'll create tickets for this steps. > > > > > > Stable suites are a good idea, but also we need to do some actions > when a > > > flaky test will appear in stable suite of master branch run. To find > out > > a > > > guilty commit, we should run previous commits on empty TeamCity's > queue. > > > This runs will reduce bot's statistics for the latest commits. > > > > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > > > Hi Dmitriy, > > > > > > > > Sounds like a plan ;) I totally agree. > > > > Just one proposal: I would like to avoid hiding any test failures. 2 > > > > separate tables named 'Possible Blockers' and 'Other failures' should > > be > > > > completely clear. Comment to PR can contain only first category. > > > > > > > > We had a private discussion with Anton K. and he proposed a very > > > > interesting thing, I would like to bring it here. We can add > > > configuration > > > > into TC bot and mark some tests and some suites as 'Known Stable' It > > > means > > > > that suite or test failure should be considered as a blocker for > merge > > > > every time it fails, even if fail rate is non-zero. Very first list > of > > > such > > > > suites are > > > > - Build Apache Ignite > > > > - License check > > > > And tests: > > > > - .Net API Parity check > > > > Please share your vision. > > > > > > > > Meanwhile, I've updated the TC bot. > > > > - Underlying Apache Ignite DB was refactored to use lower partitions > > > count, > > > > so restart should be faster. > > > > - Now 100 builds are saved as our failure rate statistics basis. > > > Flakiness > > > > border was not changed, so more test now will be considered as flaky. > > > > - Now 4 builds required to be failed or timed out in a row before > > > > notification is sent. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <[hidden email]>: > > > > > > > > > Hi, Dmitriy, > > > > > > > > > > I propose the next steps: > > > > > > > > > > 1. Show current 0% tests in a separate table at the top of the > > analysis > > > > > results page. Thus, we'll see most suspicious (new or very flaky) > > tests > > > > > firstly. Or we can hide other tests under "More >>" button, like > top > > > long > > > > > running tests. > > > > > 2. Create a button by clicking on which the info about 0% tests > will > > be > > > > > written in the PR. > > > > > 3. Replace button by webhook for TeamCity (for Run All), which will > > > start > > > > > analysis on TCH and write results in the PR. > > > > > 4. ... > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > > > > > > > > > > > I think we should check not N last runs, but all runs in last N > > days. > > > > > > > > > > > > A simple rule to detect flaky fails by hands - get test history > > > ordered > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get list > > of > > > > > > failures from TC. We don't need to check successfull runs, > because > > > they > > > > > are > > > > > > uninformative for our needs. > > > > > > > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <[hidden email] > >: > > > > > > > > > > > >> Hi Dmitriy, > > > > > >> > > > > > >> The Bot is able to detect a frequent change of test status, but > > > > > currently > > > > > >> only 50 last runs count. Same is true for the failure rate. > > > > > >> > > > > > >> This value can be easily changed to 70 or 100, moreover, the > auto > > > > > >> trigger feature gives us much more builds. > > > > > >> > > > > > >> We can improve these rules. We can add not only status change, > but > > > > > status > > > > > >> change without any code changes. We can somehow save this data > in > > > > > RunStat > > > > > >> class. Let's create a better rule, and later we can code it. > > > > > >> > > > > > >> Sincerely, > > > > > >> Dmitriy Pavlov > > > > > >> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov < > > [hidden email] > > > >: > > > > > >> > > > > > >> > I think plugin will be more pretty looking, but comments can > > > contain > > > > > any > > > > > >> > information, so they can be more usefull. I agree with your > idea > > > to > > > > > >> create > > > > > >> > bot instead of plugin. > > > > > >> > > > > > > >> > As for fail rate - I'm not sure it is working as you describe. > > > > > >> > I'm looking on my runAll [1]. There is > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and > > > fails > > > > in > > > > > >> > master branch [2]. > > > > > >> > > > > > > >> > [1] > > > > > >> > > > > > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I > > > > > >> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest > > > > > >> > [2] > > > > > >> > > > > > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > > > > > >> 24Java8=__all_branches__&itemsCount=50 > > > > > >> > > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov < > > [hidden email] > > > >: > > > > > >> > > > > > > >> > > Hi Dmitrii, > > > > > >> > > > > > > > >> > > I'm not sure we're able to install Github apps to Apache > > > mirrors. > > > > > >> > > > > > > > >> > > The simplest solution, what can be as efficient as a plugin, > > is > > > > fake > > > > > >> > MTCGA > > > > > >> > > bot account in Github, which will provide PR comments using > > > Github > > > > > >> > program > > > > > >> > > interface. What do you think? > > > > > >> > > > > > > > >> > > A new test failure can be identified by the Ignite TC Bot by > > > > master > > > > > >> > recent > > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed out > > > > suites. > > > > > >> > > > > > > > >> > > Sincerely, > > > > > >> > > Dmitriy Pavlov > > > > > >> > > > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < > > > > [hidden email] > > > > > >: > > > > > >> > > > > > > > >> > > > Hello, Igniters! > > > > > >> > > > > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we > > > need > > > > an > > > > > >> easy > > > > > >> > > way > > > > > >> > > > to get list of failed tests that don’t fall in the master > > > > branch. > > > > > >> These > > > > > >> > > > tests should: > > > > > >> > > > * fail in the PR > > > > > >> > > > * not fail in the master > > > > > >> > > > * not be flaky. > > > > > >> > > > > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which > > will > > > > > >> notify PR > > > > > >> > > if > > > > > >> > > > it has such tests. PR will have a marker, which > > > allows/prohibits > > > > > >> merge. > > > > > >> > > > This marker will be shown near PR conflicts. > > > > > >> > > > > > > > > >> > > > Allowing marker will be shown in case: > > > > > >> > > > * no new fails. > > > > > >> > > > > > > > > >> > > > Prohibiting marker will be shown in cases: > > > > > >> > > > * new fails – tests must be fixed. > > > > > >> > > > * new timed out test suite – suite should be restarted or > > > tests > > > > > >> must be > > > > > >> > > > fixed. > > > > > >> > > > * runAll wasn’t launched – tests must be launched. > > > > > >> > > > > > > > > >> > > > This will make test checks much faster and easier. Also, > > this > > > > will > > > > > >> > > decrease > > > > > >> > > > the number of merges with new failed tests made by > > inattention > > > > to > > > > > >> the > > > > > >> > > > tests. > > > > > >> > > > > > > > > >> > > > Further, we can expand the plugin by adding new checks, > > > showing > > > > PR > > > > > >> > > quality. > > > > > >> > > > > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
Hi, Dmitriy,
I made PRs in my fork for test purposes. Real PRs were made to the GitHub mirror, and one of them is already merged by D. Govorukhin. PR with GitHub statuses [1] is ready for review. PR with JIRA comment will be ready in a few days. [1] https://github.com/apache/ignite-teamcity-bot/pull/5 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > Hi Dmitrii, > > I deployed change with blockers summary of failures at the top of PR result > page. The Bot is migrating entries now, I hope it will be done in 1-4 > hours. > > I noticed your PRs are created in your own fork, not in > https://github.com/apache/ignite-teamcity-bot > Could you please create unmerged PR in GitHub mirror so it could be merged > after reviewing by the apply-pr script. > > Sincerely, > Dmitriy Pavlov > > пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <[hidden email]>: > > > Hi, Dmitriy, > > > > I created a table with possible blockers [1] for the page with the latest > > build analysis. Anton K. has reviewed it. Can you check and merge it? > > > > Also, I created a button to notify PR about analisis results [2]. I used > > GitHub statuses (example [3]), but to work it needs a token with push > > permission. As Apache PMC, can you acquire such token? I think statuses > are > > much more notable than comments. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9376 > > [2] https://issues.apache.org/jira/browse/IGNITE-9377 > > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls > > > > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > Not sure I understood what bot statistic reduction means. > > > > > > As far as I understood, you also mean locating failure automatically > with > > > re-runs on specific git revisions (hashes). I found it will be a good > > > contribution to TC Bot for both flaky and non-flaky failures detection. > > E.g > > > failure occurred after 10 commits merged to master. How to find out bad > > > commit? Automatic location (analog of bisecting, but using TC test) > will > > be > > > really helpful. But it is not a simple contribution. > > > > > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <[hidden email]>: > > > > > > > Hi, Dmitriy, > > > > > > > > Good, I'll create tickets for this steps. > > > > > > > > Stable suites are a good idea, but also we need to do some actions > > when a > > > > flaky test will appear in stable suite of master branch run. To find > > out > > > a > > > > guilty commit, we should run previous commits on empty TeamCity's > > queue. > > > > This runs will reduce bot's statistics for the latest commits. > > > > > > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > > > > > Hi Dmitriy, > > > > > > > > > > Sounds like a plan ;) I totally agree. > > > > > Just one proposal: I would like to avoid hiding any test failures. > 2 > > > > > separate tables named 'Possible Blockers' and 'Other failures' > should > > > be > > > > > completely clear. Comment to PR can contain only first category. > > > > > > > > > > We had a private discussion with Anton K. and he proposed a very > > > > > interesting thing, I would like to bring it here. We can add > > > > configuration > > > > > into TC bot and mark some tests and some suites as 'Known Stable' > It > > > > means > > > > > that suite or test failure should be considered as a blocker for > > merge > > > > > every time it fails, even if fail rate is non-zero. Very first list > > of > > > > such > > > > > suites are > > > > > - Build Apache Ignite > > > > > - License check > > > > > And tests: > > > > > - .Net API Parity check > > > > > Please share your vision. > > > > > > > > > > Meanwhile, I've updated the TC bot. > > > > > - Underlying Apache Ignite DB was refactored to use lower > partitions > > > > count, > > > > > so restart should be faster. > > > > > - Now 100 builds are saved as our failure rate statistics basis. > > > > Flakiness > > > > > border was not changed, so more test now will be considered as > flaky. > > > > > - Now 4 builds required to be failed or timed out in a row before > > > > > notification is sent. > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <[hidden email] > >: > > > > > > > > > > > Hi, Dmitriy, > > > > > > > > > > > > I propose the next steps: > > > > > > > > > > > > 1. Show current 0% tests in a separate table at the top of the > > > analysis > > > > > > results page. Thus, we'll see most suspicious (new or very flaky) > > > tests > > > > > > firstly. Or we can hide other tests under "More >>" button, like > > top > > > > long > > > > > > running tests. > > > > > > 2. Create a button by clicking on which the info about 0% tests > > will > > > be > > > > > > written in the PR. > > > > > > 3. Replace button by webhook for TeamCity (for Run All), which > will > > > > start > > > > > > analysis on TCH and write results in the PR. > > > > > > 4. ... > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email] > >: > > > > > > > > > > > > > I think we should check not N last runs, but all runs in last N > > > days. > > > > > > > > > > > > > > A simple rule to detect flaky fails by hands - get test history > > > > ordered > > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get > list > > > of > > > > > > > failures from TC. We don't need to check successfull runs, > > because > > > > they > > > > > > are > > > > > > > uninformative for our needs. > > > > > > > > > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov < > [hidden email] > > >: > > > > > > > > > > > > > >> Hi Dmitriy, > > > > > > >> > > > > > > >> The Bot is able to detect a frequent change of test status, > but > > > > > > currently > > > > > > >> only 50 last runs count. Same is true for the failure rate. > > > > > > >> > > > > > > >> This value can be easily changed to 70 or 100, moreover, the > > auto > > > > > > >> trigger feature gives us much more builds. > > > > > > >> > > > > > > >> We can improve these rules. We can add not only status change, > > but > > > > > > status > > > > > > >> change without any code changes. We can somehow save this data > > in > > > > > > RunStat > > > > > > >> class. Let's create a better rule, and later we can code it. > > > > > > >> > > > > > > >> Sincerely, > > > > > > >> Dmitriy Pavlov > > > > > > >> > > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov < > > > [hidden email] > > > > >: > > > > > > >> > > > > > > >> > I think plugin will be more pretty looking, but comments can > > > > contain > > > > > > any > > > > > > >> > information, so they can be more usefull. I agree with your > > idea > > > > to > > > > > > >> create > > > > > > >> > bot instead of plugin. > > > > > > >> > > > > > > > >> > As for fail rate - I'm not sure it is working as you > describe. > > > > > > >> > I'm looking on my runAll [1]. There is > > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky > and > > > > fails > > > > > in > > > > > > >> > master branch [2]. > > > > > > >> > > > > > > > >> > [1] > > > > > > >> > > > > > > > >> > https://mtcga.gridgain.com/pr. > html?serverId=apache&suiteId=I > > > > > > >> > > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519% > 2Fhead&action=Latest > > > > > > >> > [2] > > > > > > >> > > > > > > > >> > https://ci.ignite.apache.org/project.html?projectId= > IgniteTe > > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 > > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > > > > > > >> 24Java8=__all_branches__&itemsCount=50 > > > > > > >> > > > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov < > > > [hidden email] > > > > >: > > > > > > >> > > > > > > > >> > > Hi Dmitrii, > > > > > > >> > > > > > > > > >> > > I'm not sure we're able to install Github apps to Apache > > > > mirrors. > > > > > > >> > > > > > > > > >> > > The simplest solution, what can be as efficient as a > plugin, > > > is > > > > > fake > > > > > > >> > MTCGA > > > > > > >> > > bot account in Github, which will provide PR comments > using > > > > Github > > > > > > >> > program > > > > > > >> > > interface. What do you think? > > > > > > >> > > > > > > > > >> > > A new test failure can be identified by the Ignite TC Bot > by > > > > > master > > > > > > >> > recent > > > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed > out > > > > > suites. > > > > > > >> > > > > > > > > >> > > Sincerely, > > > > > > >> > > Dmitriy Pavlov > > > > > > >> > > > > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < > > > > > [hidden email] > > > > > > >: > > > > > > >> > > > > > > > > >> > > > Hello, Igniters! > > > > > > >> > > > > > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – > we > > > > need > > > > > an > > > > > > >> easy > > > > > > >> > > way > > > > > > >> > > > to get list of failed tests that don’t fall in the > master > > > > > branch. > > > > > > >> These > > > > > > >> > > > tests should: > > > > > > >> > > > * fail in the PR > > > > > > >> > > > * not fail in the master > > > > > > >> > > > * not be flaky. > > > > > > >> > > > > > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which > > > will > > > > > > >> notify PR > > > > > > >> > > if > > > > > > >> > > > it has such tests. PR will have a marker, which > > > > allows/prohibits > > > > > > >> merge. > > > > > > >> > > > This marker will be shown near PR conflicts. > > > > > > >> > > > > > > > > > >> > > > Allowing marker will be shown in case: > > > > > > >> > > > * no new fails. > > > > > > >> > > > > > > > > > >> > > > Prohibiting marker will be shown in cases: > > > > > > >> > > > * new fails – tests must be fixed. > > > > > > >> > > > * new timed out test suite – suite should be restarted > or > > > > tests > > > > > > >> must be > > > > > > >> > > > fixed. > > > > > > >> > > > * runAll wasn’t launched – tests must be launched. > > > > > > >> > > > > > > > > > >> > > > This will make test checks much faster and easier. Also, > > > this > > > > > will > > > > > > >> > > decrease > > > > > > >> > > > the number of merges with new failed tests made by > > > inattention > > > > > to > > > > > > >> the > > > > > > >> > > > tests. > > > > > > >> > > > > > > > > > >> > > > Further, we can expand the plugin by adding new checks, > > > > showing > > > > > PR > > > > > > >> > > quality. > > > > > > >> > > > > > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
Hi, Dmitriy,
Can you give me rights to stop my builds on TeamCity? Working on the TCH, I run a lot of builds, and I don't like to ask other people to stop builds too often. 2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > Hi, Dmitriy, > > I made PRs in my fork for test purposes. Real PRs were made to the GitHub > mirror, and one of them is already merged by D. Govorukhin. PR with GitHub > statuses [1] is ready for review. PR with JIRA comment will be ready in a > few days. > > [1] https://github.com/apache/ignite-teamcity-bot/pull/5 > > 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > >> Hi Dmitrii, >> >> I deployed change with blockers summary of failures at the top of PR >> result >> page. The Bot is migrating entries now, I hope it will be done in 1-4 >> hours. >> >> I noticed your PRs are created in your own fork, not in >> https://github.com/apache/ignite-teamcity-bot >> Could you please create unmerged PR in GitHub mirror so it could be merged >> after reviewing by the apply-pr script. >> >> Sincerely, >> Dmitriy Pavlov >> >> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <[hidden email]>: >> >> > Hi, Dmitriy, >> > >> > I created a table with possible blockers [1] for the page with the >> latest >> > build analysis. Anton K. has reviewed it. Can you check and merge it? >> > >> > Also, I created a button to notify PR about analisis results [2]. I used >> > GitHub statuses (example [3]), but to work it needs a token with push >> > permission. As Apache PMC, can you acquire such token? I think statuses >> are >> > much more notable than comments. >> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-9376 >> > [2] https://issues.apache.org/jira/browse/IGNITE-9377 >> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls >> > >> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <[hidden email]>: >> > >> > > Not sure I understood what bot statistic reduction means. >> > > >> > > As far as I understood, you also mean locating failure automatically >> with >> > > re-runs on specific git revisions (hashes). I found it will be a good >> > > contribution to TC Bot for both flaky and non-flaky failures >> detection. >> > E.g >> > > failure occurred after 10 commits merged to master. How to find out >> bad >> > > commit? Automatic location (analog of bisecting, but using TC test) >> will >> > be >> > > really helpful. But it is not a simple contribution. >> > > >> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <[hidden email]>: >> > > >> > > > Hi, Dmitriy, >> > > > >> > > > Good, I'll create tickets for this steps. >> > > > >> > > > Stable suites are a good idea, but also we need to do some actions >> > when a >> > > > flaky test will appear in stable suite of master branch run. To find >> > out >> > > a >> > > > guilty commit, we should run previous commits on empty TeamCity's >> > queue. >> > > > This runs will reduce bot's statistics for the latest commits. >> > > > >> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <[hidden email]>: >> > > > >> > > > > Hi Dmitriy, >> > > > > >> > > > > Sounds like a plan ;) I totally agree. >> > > > > Just one proposal: I would like to avoid hiding any test >> failures. 2 >> > > > > separate tables named 'Possible Blockers' and 'Other failures' >> should >> > > be >> > > > > completely clear. Comment to PR can contain only first category. >> > > > > >> > > > > We had a private discussion with Anton K. and he proposed a very >> > > > > interesting thing, I would like to bring it here. We can add >> > > > configuration >> > > > > into TC bot and mark some tests and some suites as 'Known Stable' >> It >> > > > means >> > > > > that suite or test failure should be considered as a blocker for >> > merge >> > > > > every time it fails, even if fail rate is non-zero. Very first >> list >> > of >> > > > such >> > > > > suites are >> > > > > - Build Apache Ignite >> > > > > - License check >> > > > > And tests: >> > > > > - .Net API Parity check >> > > > > Please share your vision. >> > > > > >> > > > > Meanwhile, I've updated the TC bot. >> > > > > - Underlying Apache Ignite DB was refactored to use lower >> partitions >> > > > count, >> > > > > so restart should be faster. >> > > > > - Now 100 builds are saved as our failure rate statistics basis. >> > > > Flakiness >> > > > > border was not changed, so more test now will be considered as >> flaky. >> > > > > - Now 4 builds required to be failed or timed out in a row before >> > > > > notification is sent. >> > > > > >> > > > > Sincerely, >> > > > > Dmitriy Pavlov >> > > > > >> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov < >> [hidden email]>: >> > > > > >> > > > > > Hi, Dmitriy, >> > > > > > >> > > > > > I propose the next steps: >> > > > > > >> > > > > > 1. Show current 0% tests in a separate table at the top of the >> > > analysis >> > > > > > results page. Thus, we'll see most suspicious (new or very >> flaky) >> > > tests >> > > > > > firstly. Or we can hide other tests under "More >>" button, like >> > top >> > > > long >> > > > > > running tests. >> > > > > > 2. Create a button by clicking on which the info about 0% tests >> > will >> > > be >> > > > > > written in the PR. >> > > > > > 3. Replace button by webhook for TeamCity (for Run All), which >> will >> > > > start >> > > > > > analysis on TCH and write results in the PR. >> > > > > > 4. ... >> > > > > > >> > > > > > What do you think? >> > > > > > >> > > > > > >> > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <[hidden email] >> >: >> > > > > > >> > > > > > > I think we should check not N last runs, but all runs in last >> N >> > > days. >> > > > > > > >> > > > > > > A simple rule to detect flaky fails by hands - get test >> history >> > > > ordered >> > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get >> list >> > > of >> > > > > > > failures from TC. We don't need to check successfull runs, >> > because >> > > > they >> > > > > > are >> > > > > > > uninformative for our needs. >> > > > > > > >> > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov < >> [hidden email] >> > >: >> > > > > > > >> > > > > > >> Hi Dmitriy, >> > > > > > >> >> > > > > > >> The Bot is able to detect a frequent change of test status, >> but >> > > > > > currently >> > > > > > >> only 50 last runs count. Same is true for the failure rate. >> > > > > > >> >> > > > > > >> This value can be easily changed to 70 or 100, moreover, the >> > auto >> > > > > > >> trigger feature gives us much more builds. >> > > > > > >> >> > > > > > >> We can improve these rules. We can add not only status >> change, >> > but >> > > > > > status >> > > > > > >> change without any code changes. We can somehow save this >> data >> > in >> > > > > > RunStat >> > > > > > >> class. Let's create a better rule, and later we can code it. >> > > > > > >> >> > > > > > >> Sincerely, >> > > > > > >> Dmitriy Pavlov >> > > > > > >> >> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov < >> > > [hidden email] >> > > > >: >> > > > > > >> >> > > > > > >> > I think plugin will be more pretty looking, but comments >> can >> > > > contain >> > > > > > any >> > > > > > >> > information, so they can be more usefull. I agree with your >> > idea >> > > > to >> > > > > > >> create >> > > > > > >> > bot instead of plugin. >> > > > > > >> > >> > > > > > >> > As for fail rate - I'm not sure it is working as you >> describe. >> > > > > > >> > I'm looking on my runAll [1]. There is >> > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` >> > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky >> and >> > > > fails >> > > > > in >> > > > > > >> > master branch [2]. >> > > > > > >> > >> > > > > > >> > [1] >> > > > > > >> > >> > > > > > >> > https://mtcga.gridgain.com/pr. >> html?serverId=apache&suiteId=I >> > > > > > >> >> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead& >> action=Latest >> > > > > > >> > [2] >> > > > > > >> > >> > > > > > >> > https://ci.ignite.apache.org/p >> roject.html?projectId=IgniteTe >> > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 >> > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests >> > > > > > >> 24Java8=__all_branches__&itemsCount=50 >> > > > > > >> > >> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov < >> > > [hidden email] >> > > > >: >> > > > > > >> > >> > > > > > >> > > Hi Dmitrii, >> > > > > > >> > > >> > > > > > >> > > I'm not sure we're able to install Github apps to Apache >> > > > mirrors. >> > > > > > >> > > >> > > > > > >> > > The simplest solution, what can be as efficient as a >> plugin, >> > > is >> > > > > fake >> > > > > > >> > MTCGA >> > > > > > >> > > bot account in Github, which will provide PR comments >> using >> > > > Github >> > > > > > >> > program >> > > > > > >> > > interface. What do you think? >> > > > > > >> > > >> > > > > > >> > > A new test failure can be identified by the Ignite TC >> Bot by >> > > > > master >> > > > > > >> > recent >> > > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed >> out >> > > > > suites. >> > > > > > >> > > >> > > > > > >> > > Sincerely, >> > > > > > >> > > Dmitriy Pavlov >> > > > > > >> > > >> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < >> > > > > [hidden email] >> > > > > > >: >> > > > > > >> > > >> > > > > > >> > > > Hello, Igniters! >> > > > > > >> > > > >> > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] >> – we >> > > > need >> > > > > an >> > > > > > >> easy >> > > > > > >> > > way >> > > > > > >> > > > to get list of failed tests that don’t fall in the >> master >> > > > > branch. >> > > > > > >> These >> > > > > > >> > > > tests should: >> > > > > > >> > > > * fail in the PR >> > > > > > >> > > > * not fail in the master >> > > > > > >> > > > * not be flaky. >> > > > > > >> > > > >> > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, >> which >> > > will >> > > > > > >> notify PR >> > > > > > >> > > if >> > > > > > >> > > > it has such tests. PR will have a marker, which >> > > > allows/prohibits >> > > > > > >> merge. >> > > > > > >> > > > This marker will be shown near PR conflicts. >> > > > > > >> > > > >> > > > > > >> > > > Allowing marker will be shown in case: >> > > > > > >> > > > * no new fails. >> > > > > > >> > > > >> > > > > > >> > > > Prohibiting marker will be shown in cases: >> > > > > > >> > > > * new fails – tests must be fixed. >> > > > > > >> > > > * new timed out test suite – suite should be restarted >> or >> > > > tests >> > > > > > >> must be >> > > > > > >> > > > fixed. >> > > > > > >> > > > * runAll wasn’t launched – tests must be launched. >> > > > > > >> > > > >> > > > > > >> > > > This will make test checks much faster and easier. >> Also, >> > > this >> > > > > will >> > > > > > >> > > decrease >> > > > > > >> > > > the number of merges with new failed tests made by >> > > inattention >> > > > > to >> > > > > > >> the >> > > > > > >> > > > tests. >> > > > > > >> > > > >> > > > > > >> > > > Further, we can expand the plugin by adding new checks, >> > > > showing >> > > > > PR >> > > > > > >> > > quality. >> > > > > > >> > > > >> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > > |
|
|
Hi Dmitriy,
Sure, I agree with extra permissions assignment. Done. Could you please check if you can manage your builds now? Sincerely, Dmitriy Pavlov чт, 13 сент. 2018 г. в 16:05, Dmitrii Ryabov <[hidden email]>: > Hi, Dmitriy, > > Can you give me rights to stop my builds on TeamCity? Working on the TCH, I > run a lot of builds, and I don't like to ask other people to stop builds > too often. > > 2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > > > Hi, Dmitriy, > > > > I made PRs in my fork for test purposes. Real PRs were made to the GitHub > > mirror, and one of them is already merged by D. Govorukhin. PR with > GitHub > > statuses [1] is ready for review. PR with JIRA comment will be ready in a > > few days. > > > > [1] https://github.com/apache/ignite-teamcity-bot/pull/5 > > > > 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > >> Hi Dmitrii, > >> > >> I deployed change with blockers summary of failures at the top of PR > >> result > >> page. The Bot is migrating entries now, I hope it will be done in 1-4 > >> hours. > >> > >> I noticed your PRs are created in your own fork, not in > >> https://github.com/apache/ignite-teamcity-bot > >> Could you please create unmerged PR in GitHub mirror so it could be > merged > >> after reviewing by the apply-pr script. > >> > >> Sincerely, > >> Dmitriy Pavlov > >> > >> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <[hidden email]>: > >> > >> > Hi, Dmitriy, > >> > > >> > I created a table with possible blockers [1] for the page with the > >> latest > >> > build analysis. Anton K. has reviewed it. Can you check and merge it? > >> > > >> > Also, I created a button to notify PR about analisis results [2]. I > used > >> > GitHub statuses (example [3]), but to work it needs a token with push > >> > permission. As Apache PMC, can you acquire such token? I think > statuses > >> are > >> > much more notable than comments. > >> > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-9376 > >> > [2] https://issues.apache.org/jira/browse/IGNITE-9377 > >> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls > >> > > >> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > >> > > >> > > Not sure I understood what bot statistic reduction means. > >> > > > >> > > As far as I understood, you also mean locating failure automatically > >> with > >> > > re-runs on specific git revisions (hashes). I found it will be a > good > >> > > contribution to TC Bot for both flaky and non-flaky failures > >> detection. > >> > E.g > >> > > failure occurred after 10 commits merged to master. How to find out > >> bad > >> > > commit? Automatic location (analog of bisecting, but using TC test) > >> will > >> > be > >> > > really helpful. But it is not a simple contribution. > >> > > > >> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <[hidden email] > >: > >> > > > >> > > > Hi, Dmitriy, > >> > > > > >> > > > Good, I'll create tickets for this steps. > >> > > > > >> > > > Stable suites are a good idea, but also we need to do some actions > >> > when a > >> > > > flaky test will appear in stable suite of master branch run. To > find > >> > out > >> > > a > >> > > > guilty commit, we should run previous commits on empty TeamCity's > >> > queue. > >> > > > This runs will reduce bot's statistics for the latest commits. > >> > > > > >> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <[hidden email] > >: > >> > > > > >> > > > > Hi Dmitriy, > >> > > > > > >> > > > > Sounds like a plan ;) I totally agree. > >> > > > > Just one proposal: I would like to avoid hiding any test > >> failures. 2 > >> > > > > separate tables named 'Possible Blockers' and 'Other failures' > >> should > >> > > be > >> > > > > completely clear. Comment to PR can contain only first category. > >> > > > > > >> > > > > We had a private discussion with Anton K. and he proposed a very > >> > > > > interesting thing, I would like to bring it here. We can add > >> > > > configuration > >> > > > > into TC bot and mark some tests and some suites as 'Known > Stable' > >> It > >> > > > means > >> > > > > that suite or test failure should be considered as a blocker for > >> > merge > >> > > > > every time it fails, even if fail rate is non-zero. Very first > >> list > >> > of > >> > > > such > >> > > > > suites are > >> > > > > - Build Apache Ignite > >> > > > > - License check > >> > > > > And tests: > >> > > > > - .Net API Parity check > >> > > > > Please share your vision. > >> > > > > > >> > > > > Meanwhile, I've updated the TC bot. > >> > > > > - Underlying Apache Ignite DB was refactored to use lower > >> partitions > >> > > > count, > >> > > > > so restart should be faster. > >> > > > > - Now 100 builds are saved as our failure rate statistics basis. > >> > > > Flakiness > >> > > > > border was not changed, so more test now will be considered as > >> flaky. > >> > > > > - Now 4 builds required to be failed or timed out in a row > before > >> > > > > notification is sent. > >> > > > > > >> > > > > Sincerely, > >> > > > > Dmitriy Pavlov > >> > > > > > >> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov < > >> [hidden email]>: > >> > > > > > >> > > > > > Hi, Dmitriy, > >> > > > > > > >> > > > > > I propose the next steps: > >> > > > > > > >> > > > > > 1. Show current 0% tests in a separate table at the top of the > >> > > analysis > >> > > > > > results page. Thus, we'll see most suspicious (new or very > >> flaky) > >> > > tests > >> > > > > > firstly. Or we can hide other tests under "More >>" button, > like > >> > top > >> > > > long > >> > > > > > running tests. > >> > > > > > 2. Create a button by clicking on which the info about 0% > tests > >> > will > >> > > be > >> > > > > > written in the PR. > >> > > > > > 3. Replace button by webhook for TeamCity (for Run All), which > >> will > >> > > > start > >> > > > > > analysis on TCH and write results in the PR. > >> > > > > > 4. ... > >> > > > > > > >> > > > > > What do you think? > >> > > > > > > >> > > > > > > >> > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov < > [hidden email] > >> >: > >> > > > > > > >> > > > > > > I think we should check not N last runs, but all runs in > last > >> N > >> > > days. > >> > > > > > > > >> > > > > > > A simple rule to detect flaky fails by hands - get test > >> history > >> > > > ordered > >> > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get > >> list > >> > > of > >> > > > > > > failures from TC. We don't need to check successfull runs, > >> > because > >> > > > they > >> > > > > > are > >> > > > > > > uninformative for our needs. > >> > > > > > > > >> > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov < > >> [hidden email] > >> > >: > >> > > > > > > > >> > > > > > >> Hi Dmitriy, > >> > > > > > >> > >> > > > > > >> The Bot is able to detect a frequent change of test status, > >> but > >> > > > > > currently > >> > > > > > >> only 50 last runs count. Same is true for the failure rate. > >> > > > > > >> > >> > > > > > >> This value can be easily changed to 70 or 100, moreover, > the > >> > auto > >> > > > > > >> trigger feature gives us much more builds. > >> > > > > > >> > >> > > > > > >> We can improve these rules. We can add not only status > >> change, > >> > but > >> > > > > > status > >> > > > > > >> change without any code changes. We can somehow save this > >> data > >> > in > >> > > > > > RunStat > >> > > > > > >> class. Let's create a better rule, and later we can code > it. > >> > > > > > >> > >> > > > > > >> Sincerely, > >> > > > > > >> Dmitriy Pavlov > >> > > > > > >> > >> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov < > >> > > [hidden email] > >> > > > >: > >> > > > > > >> > >> > > > > > >> > I think plugin will be more pretty looking, but comments > >> can > >> > > > contain > >> > > > > > any > >> > > > > > >> > information, so they can be more usefull. I agree with > your > >> > idea > >> > > > to > >> > > > > > >> create > >> > > > > > >> > bot instead of plugin. > >> > > > > > >> > > >> > > > > > >> > As for fail rate - I'm not sure it is working as you > >> describe. > >> > > > > > >> > I'm looking on my runAll [1]. There is > >> > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > >> > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky > >> and > >> > > > fails > >> > > > > in > >> > > > > > >> > master branch [2]. > >> > > > > > >> > > >> > > > > > >> > [1] > >> > > > > > >> > > >> > > > > > >> > https://mtcga.gridgain.com/pr. > >> html?serverId=apache&suiteId=I > >> > > > > > >> > >> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead& > >> action=Latest > >> > > > > > >> > [2] > >> > > > > > >> > > >> > > > > > >> > https://ci.ignite.apache.org/p > >> roject.html?projectId=IgniteTe > >> > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470 > >> > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > >> > > > > > >> 24Java8=__all_branches__&itemsCount=50 > >> > > > > > >> > > >> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov < > >> > > [hidden email] > >> > > > >: > >> > > > > > >> > > >> > > > > > >> > > Hi Dmitrii, > >> > > > > > >> > > > >> > > > > > >> > > I'm not sure we're able to install Github apps to > Apache > >> > > > mirrors. > >> > > > > > >> > > > >> > > > > > >> > > The simplest solution, what can be as efficient as a > >> plugin, > >> > > is > >> > > > > fake > >> > > > > > >> > MTCGA > >> > > > > > >> > > bot account in Github, which will provide PR comments > >> using > >> > > > Github > >> > > > > > >> > program > >> > > > > > >> > > interface. What do you think? > >> > > > > > >> > > > >> > > > > > >> > > A new test failure can be identified by the Ignite TC > >> Bot by > >> > > > > master > >> > > > > > >> > recent > >> > > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed > >> out > >> > > > > suites. > >> > > > > > >> > > > >> > > > > > >> > > Sincerely, > >> > > > > > >> > > Dmitriy Pavlov > >> > > > > > >> > > > >> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < > >> > > > > [hidden email] > >> > > > > > >: > >> > > > > > >> > > > >> > > > > > >> > > > Hello, Igniters! > >> > > > > > >> > > > > >> > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] > >> – we > >> > > > need > >> > > > > an > >> > > > > > >> easy > >> > > > > > >> > > way > >> > > > > > >> > > > to get list of failed tests that don’t fall in the > >> master > >> > > > > branch. > >> > > > > > >> These > >> > > > > > >> > > > tests should: > >> > > > > > >> > > > * fail in the PR > >> > > > > > >> > > > * not fail in the master > >> > > > > > >> > > > * not be flaky. > >> > > > > > >> > > > > >> > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, > >> which > >> > > will > >> > > > > > >> notify PR > >> > > > > > >> > > if > >> > > > > > >> > > > it has such tests. PR will have a marker, which > >> > > > allows/prohibits > >> > > > > > >> merge. > >> > > > > > >> > > > This marker will be shown near PR conflicts. > >> > > > > > >> > > > > >> > > > > > >> > > > Allowing marker will be shown in case: > >> > > > > > >> > > > * no new fails. > >> > > > > > >> > > > > >> > > > > > >> > > > Prohibiting marker will be shown in cases: > >> > > > > > >> > > > * new fails – tests must be fixed. > >> > > > > > >> > > > * new timed out test suite – suite should be > restarted > >> or > >> > > > tests > >> > > > > > >> must be > >> > > > > > >> > > > fixed. > >> > > > > > >> > > > * runAll wasn’t launched – tests must be launched. > >> > > > > > >> > > > > >> > > > > > >> > > > This will make test checks much faster and easier. > >> Also, > >> > > this > >> > > > > will > >> > > > > > >> > > decrease > >> > > > > > >> > > > the number of merges with new failed tests made by > >> > > inattention > >> > > > > to > >> > > > > > >> the > >> > > > > > >> > > > tests. > >> > > > > > >> > > > > >> > > > > > >> > > > Further, we can expand the plugin by adding new > checks, > >> > > > showing > >> > > > > PR > >> > > > > > >> > > quality. > >> > > > > > >> > > > > >> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > > |
|
|
Yep, it's working. Thank you.
2018-09-13 19:09 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > Hi Dmitriy, > > Sure, I agree with extra permissions assignment. Done. > > Could you please check if you can manage your builds now? > > Sincerely, > Dmitriy Pavlov > > чт, 13 сент. 2018 г. в 16:05, Dmitrii Ryabov <[hidden email]>: > > > Hi, Dmitriy, > > > > Can you give me rights to stop my builds on TeamCity? Working on the > TCH, I > > run a lot of builds, and I don't like to ask other people to stop builds > > too often. > > > > 2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov <[hidden email]>: > > > > > Hi, Dmitriy, > > > > > > I made PRs in my fork for test purposes. Real PRs were made to the > GitHub > > > mirror, and one of them is already merged by D. Govorukhin. PR with > > GitHub > > > statuses [1] is ready for review. PR with JIRA comment will be ready > in a > > > few days. > > > > > > [1] https://github.com/apache/ignite-teamcity-bot/pull/5 > > > > > > 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > > > > >> Hi Dmitrii, > > >> > > >> I deployed change with blockers summary of failures at the top of PR > > >> result > > >> page. The Bot is migrating entries now, I hope it will be done in 1-4 > > >> hours. > > >> > > >> I noticed your PRs are created in your own fork, not in > > >> https://github.com/apache/ignite-teamcity-bot > > >> Could you please create unmerged PR in GitHub mirror so it could be > > merged > > >> after reviewing by the apply-pr script. > > >> > > >> Sincerely, > > >> Dmitriy Pavlov > > >> > > >> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <[hidden email]>: > > >> > > >> > Hi, Dmitriy, > > >> > > > >> > I created a table with possible blockers [1] for the page with the > > >> latest > > >> > build analysis. Anton K. has reviewed it. Can you check and merge > it? > > >> > > > >> > Also, I created a button to notify PR about analisis results [2]. I > > used > > >> > GitHub statuses (example [3]), but to work it needs a token with > push > > >> > permission. As Apache PMC, can you acquire such token? I think > > statuses > > >> are > > >> > much more notable than comments. > > >> > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-9376 > > >> > [2] https://issues.apache.org/jira/browse/IGNITE-9377 > > >> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls > > >> > > > >> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <[hidden email]>: > > >> > > > >> > > Not sure I understood what bot statistic reduction means. > > >> > > > > >> > > As far as I understood, you also mean locating failure > automatically > > >> with > > >> > > re-runs on specific git revisions (hashes). I found it will be a > > good > > >> > > contribution to TC Bot for both flaky and non-flaky failures > > >> detection. > > >> > E.g > > >> > > failure occurred after 10 commits merged to master. How to find > out > > >> bad > > >> > > commit? Automatic location (analog of bisecting, but using TC > test) > > >> will > > >> > be > > >> > > really helpful. But it is not a simple contribution. > > >> > > > > >> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov < > [hidden email] > > >: > > >> > > > > >> > > > Hi, Dmitriy, > > >> > > > > > >> > > > Good, I'll create tickets for this steps. > > >> > > > > > >> > > > Stable suites are a good idea, but also we need to do some > actions > > >> > when a > > >> > > > flaky test will appear in stable suite of master branch run. To > > find > > >> > out > > >> > > a > > >> > > > guilty commit, we should run previous commits on empty > TeamCity's > > >> > queue. > > >> > > > This runs will reduce bot's statistics for the latest commits. > > >> > > > > > >> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov < > [hidden email] > > >: > > >> > > > > > >> > > > > Hi Dmitriy, > > >> > > > > > > >> > > > > Sounds like a plan ;) I totally agree. > > >> > > > > Just one proposal: I would like to avoid hiding any test > > >> failures. 2 > > >> > > > > separate tables named 'Possible Blockers' and 'Other failures' > > >> should > > >> > > be > > >> > > > > completely clear. Comment to PR can contain only first > category. > > >> > > > > > > >> > > > > We had a private discussion with Anton K. and he proposed a > very > > >> > > > > interesting thing, I would like to bring it here. We can add > > >> > > > configuration > > >> > > > > into TC bot and mark some tests and some suites as 'Known > > Stable' > > >> It > > >> > > > means > > >> > > > > that suite or test failure should be considered as a blocker > for > > >> > merge > > >> > > > > every time it fails, even if fail rate is non-zero. Very first > > >> list > > >> > of > > >> > > > such > > >> > > > > suites are > > >> > > > > - Build Apache Ignite > > >> > > > > - License check > > >> > > > > And tests: > > >> > > > > - .Net API Parity check > > >> > > > > Please share your vision. > > >> > > > > > > >> > > > > Meanwhile, I've updated the TC bot. > > >> > > > > - Underlying Apache Ignite DB was refactored to use lower > > >> partitions > > >> > > > count, > > >> > > > > so restart should be faster. > > >> > > > > - Now 100 builds are saved as our failure rate statistics > basis. > > >> > > > Flakiness > > >> > > > > border was not changed, so more test now will be considered as > > >> flaky. > > >> > > > > - Now 4 builds required to be failed or timed out in a row > > before > > >> > > > > notification is sent. > > >> > > > > > > >> > > > > Sincerely, > > >> > > > > Dmitriy Pavlov > > >> > > > > > > >> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov < > > >> [hidden email]>: > > >> > > > > > > >> > > > > > Hi, Dmitriy, > > >> > > > > > > > >> > > > > > I propose the next steps: > > >> > > > > > > > >> > > > > > 1. Show current 0% tests in a separate table at the top of > the > > >> > > analysis > > >> > > > > > results page. Thus, we'll see most suspicious (new or very > > >> flaky) > > >> > > tests > > >> > > > > > firstly. Or we can hide other tests under "More >>" button, > > like > > >> > top > > >> > > > long > > >> > > > > > running tests. > > >> > > > > > 2. Create a button by clicking on which the info about 0% > > tests > > >> > will > > >> > > be > > >> > > > > > written in the PR. > > >> > > > > > 3. Replace button by webhook for TeamCity (for Run All), > which > > >> will > > >> > > > start > > >> > > > > > analysis on TCH and write results in the PR. > > >> > > > > > 4. ... > > >> > > > > > > > >> > > > > > What do you think? > > >> > > > > > > > >> > > > > > > > >> > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov < > > [hidden email] > > >> >: > > >> > > > > > > > >> > > > > > > I think we should check not N last runs, but all runs in > > last > > >> N > > >> > > days. > > >> > > > > > > > > >> > > > > > > A simple rule to detect flaky fails by hands - get test > > >> history > > >> > > > ordered > > >> > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can > get > > >> list > > >> > > of > > >> > > > > > > failures from TC. We don't need to check successfull runs, > > >> > because > > >> > > > they > > >> > > > > > are > > >> > > > > > > uninformative for our needs. > > >> > > > > > > > > >> > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov < > > >> [hidden email] > > >> > >: > > >> > > > > > > > > >> > > > > > >> Hi Dmitriy, > > >> > > > > > >> > > >> > > > > > >> The Bot is able to detect a frequent change of test > status, > > >> but > > >> > > > > > currently > > >> > > > > > >> only 50 last runs count. Same is true for the failure > rate. > > >> > > > > > >> > > >> > > > > > >> This value can be easily changed to 70 or 100, moreover, > > the > > >> > auto > > >> > > > > > >> trigger feature gives us much more builds. > > >> > > > > > >> > > >> > > > > > >> We can improve these rules. We can add not only status > > >> change, > > >> > but > > >> > > > > > status > > >> > > > > > >> change without any code changes. We can somehow save this > > >> data > > >> > in > > >> > > > > > RunStat > > >> > > > > > >> class. Let's create a better rule, and later we can code > > it. > > >> > > > > > >> > > >> > > > > > >> Sincerely, > > >> > > > > > >> Dmitriy Pavlov > > >> > > > > > >> > > >> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov < > > >> > > [hidden email] > > >> > > > >: > > >> > > > > > >> > > >> > > > > > >> > I think plugin will be more pretty looking, but > comments > > >> can > > >> > > > contain > > >> > > > > > any > > >> > > > > > >> > information, so they can be more usefull. I agree with > > your > > >> > idea > > >> > > > to > > >> > > > > > >> create > > >> > > > > > >> > bot instead of plugin. > > >> > > > > > >> > > > >> > > > > > >> > As for fail rate - I'm not sure it is working as you > > >> describe. > > >> > > > > > >> > I'm looking on my runAll [1]. There is > > >> > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated` > > >> > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is > flaky > > >> and > > >> > > > fails > > >> > > > > in > > >> > > > > > >> > master branch [2]. > > >> > > > > > >> > > > >> > > > > > >> > [1] > > >> > > > > > >> > > > >> > > > > > >> > https://mtcga.gridgain.com/pr. > > >> html?serverId=apache&suiteId=I > > >> > > > > > >> > > >> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead& > > >> action=Latest > > >> > > > > > >> > [2] > > >> > > > > > >> > > > >> > > > > > >> > https://ci.ignite.apache.org/p > > >> roject.html?projectId=IgniteTe > > >> > > > > > >> sts24Java8&buildTypeId=&tab= > testDetails&testNameId=5628470 > > >> > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests > > >> > > > > > >> 24Java8=__all_branches__&itemsCount=50 > > >> > > > > > >> > > > >> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov < > > >> > > [hidden email] > > >> > > > >: > > >> > > > > > >> > > > >> > > > > > >> > > Hi Dmitrii, > > >> > > > > > >> > > > > >> > > > > > >> > > I'm not sure we're able to install Github apps to > > Apache > > >> > > > mirrors. > > >> > > > > > >> > > > > >> > > > > > >> > > The simplest solution, what can be as efficient as a > > >> plugin, > > >> > > is > > >> > > > > fake > > >> > > > > > >> > MTCGA > > >> > > > > > >> > > bot account in Github, which will provide PR comments > > >> using > > >> > > > Github > > >> > > > > > >> > program > > >> > > > > > >> > > interface. What do you think? > > >> > > > > > >> > > > > >> > > > > > >> > > A new test failure can be identified by the Ignite TC > > >> Bot by > > >> > > > > master > > >> > > > > > >> > recent > > >> > > > > > >> > > fail rate = 0.0%. The same rule can be applied to > timed > > >> out > > >> > > > > suites. > > >> > > > > > >> > > > > >> > > > > > >> > > Sincerely, > > >> > > > > > >> > > Dmitriy Pavlov > > >> > > > > > >> > > > > >> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov < > > >> > > > > [hidden email] > > >> > > > > > >: > > >> > > > > > >> > > > > >> > > > > > >> > > > Hello, Igniters! > > >> > > > > > >> > > > > > >> > > > > > >> > > > I want to suggest improvement for TeamCity Helper > [1] > > >> – we > > >> > > > need > > >> > > > > an > > >> > > > > > >> easy > > >> > > > > > >> > > way > > >> > > > > > >> > > > to get list of failed tests that don’t fall in the > > >> master > > >> > > > > branch. > > >> > > > > > >> These > > >> > > > > > >> > > > tests should: > > >> > > > > > >> > > > * fail in the PR > > >> > > > > > >> > > > * not fail in the master > > >> > > > > > >> > > > * not be flaky. > > >> > > > > > >> > > > > > >> > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, > > >> which > > >> > > will > > >> > > > > > >> notify PR > > >> > > > > > >> > > if > > >> > > > > > >> > > > it has such tests. PR will have a marker, which > > >> > > > allows/prohibits > > >> > > > > > >> merge. > > >> > > > > > >> > > > This marker will be shown near PR conflicts. > > >> > > > > > >> > > > > > >> > > > > > >> > > > Allowing marker will be shown in case: > > >> > > > > > >> > > > * no new fails. > > >> > > > > > >> > > > > > >> > > > > > >> > > > Prohibiting marker will be shown in cases: > > >> > > > > > >> > > > * new fails – tests must be fixed. > > >> > > > > > >> > > > * new timed out test suite – suite should be > > restarted > > >> or > > >> > > > tests > > >> > > > > > >> must be > > >> > > > > > >> > > > fixed. > > >> > > > > > >> > > > * runAll wasn’t launched – tests must be launched. > > >> > > > > > >> > > > > > >> > > > > > >> > > > This will make test checks much faster and easier. > > >> Also, > > >> > > this > > >> > > > > will > > >> > > > > > >> > > decrease > > >> > > > > > >> > > > the number of merges with new failed tests made by > > >> > > inattention > > >> > > > > to > > >> > > > > > >> the > > >> > > > > > >> > > > tests. > > >> > > > > > >> > > > > > >> > > > > > >> > > > Further, we can expand the plugin by adding new > > checks, > > >> > > > showing > > >> > > > > PR > > >> > > > > > >> > > quality. > > >> > > > > > >> > > > > > >> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > > |
«
Return to Apache Ignite Developers - Legacy Mail Archive
|
1 view|%1 views
| Free forum by Nabble | Edit this page |