Interesting statistics for our tests

[Vladimir Ozerov]

Igniters,

I collected several thought-provoking numbers for our TeamCity
(config-variations tests are excluded):

* Number of Java suites: 78
* Total execution time without build overhead: 38.5 hours
* Total test classes: 3376
* Total tests: 60681

* 25% of time is consumed by 881 tests (41 test class)
* 50% of time is consumed by 2755 tests (153 test class)
* 75% of time is consumed by 15911 tests (443 test class)

Or to be more demonstrative:
* 1.5% of tests consume 25% of time
* 4.5% of tests consume 50% of time
* 26.2% tests consume 75% of time

TOP 3 test classes:
* CacheSerializableTransactionsTest - 46 minutes
* IgniteCacheCrossCacheTxFailoverTest - 30 minutes
* GridCachePartitionedNodeRestartTest - 25 minutes

Morale:
1) We have lots of poor failover/concurrency tests requiring
disproportional time to run;
2) If we split all tests into "core" and "failover/concurrency" parts
properly, then *core* part will take *~15 mins* to answer you with high
level of confidence whether your change broken anything or not.

Thoughts?

Vladimir.

[Vladimir Ozerov]

And one more thing - how much time is spent on duplicated "Binary" suites? *15
hours*, or almost *40%* of total "Run All" execution time. It means, that
if move suites for *OptimizedMarshaller *(which we consider deprecated)
into a separate circuit (which will be executed not per-branch, but e.g.
nightly) we will almost double peak TeamCity throughput.

On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Igniters,
>
> I collected several thought-provoking numbers for our TeamCity
> (config-variations tests are excluded):
>
> * Number of Java suites: 78
> * Total execution time without build overhead: 38.5 hours
> * Total test classes: 3376
> * Total tests: 60681
>
> * 25% of time is consumed by 881 tests (41 test class)
> * 50% of time is consumed by 2755 tests (153 test class)
> * 75% of time is consumed by 15911 tests (443 test class)
>
> Or to be more demonstrative:
> * 1.5% of tests consume 25% of time
> * 4.5% of tests consume 50% of time
> * 26.2% tests consume 75% of time
>
> TOP 3 test classes:
> * CacheSerializableTransactionsTest - 46 minutes
> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> * GridCachePartitionedNodeRestartTest - 25 minutes
>
> Morale:
> 1) We have lots of poor failover/concurrency tests requiring
> disproportional time to run;
> 2) If we split all tests into "core" and "failover/concurrency" parts
> properly, then *core* part will take *~15 mins* to answer you with high
> level of confidence whether your change broken anything or not.
>
> Thoughts?
>
> Vladimir.
>

[dsetrakyan]

In addition to optimizing the test suites, should we add more TC agents as
well?

On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <[hidden email]>
wrote:

> And one more thing - how much time is spent on duplicated "Binary" suites?
> *15
> hours*, or almost *40%* of total "Run All" execution time. It means, that
> if move suites for *OptimizedMarshaller *(which we consider deprecated)
> into a separate circuit (which will be executed not per-branch, but e.g.
> nightly) we will almost double peak TeamCity throughput.
>
> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > I collected several thought-provoking numbers for our TeamCity
> > (config-variations tests are excluded):
> >
> > * Number of Java suites: 78
> > * Total execution time without build overhead: 38.5 hours
> > * Total test classes: 3376
> > * Total tests: 60681
> >
> > * 25% of time is consumed by 881 tests (41 test class)
> > * 50% of time is consumed by 2755 tests (153 test class)
> > * 75% of time is consumed by 15911 tests (443 test class)
> >
> > Or to be more demonstrative:
> > * 1.5% of tests consume 25% of time
> > * 4.5% of tests consume 50% of time
> > * 26.2% tests consume 75% of time
> >
> > TOP 3 test classes:
> > * CacheSerializableTransactionsTest - 46 minutes
> > * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> > * GridCachePartitionedNodeRestartTest - 25 minutes
> >
> > Morale:
> > 1) We have lots of poor failover/concurrency tests requiring
> > disproportional time to run;
> > 2) If we split all tests into "core" and "failover/concurrency" parts
> > properly, then *core* part will take *~15 mins* to answer you with high
> > level of confidence whether your change broken anything or not.
> >
> > Thoughts?
> >
> > Vladimir.
> >
>

[Rohit Mohta]

Interesting numbers, to say the least :)

Separating test suites, as suggested by Vladimir sounds good.

Probably off-placed question:
   As Ignite provides multiple features and is suitable for multiple use case- should we categorize test suites based on features?

------
Rohit Mohta

> On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <[hidden email]> wrote:
>
> In addition to optimizing the test suites, should we add more TC agents as
> well?
>
> On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
>> And one more thing - how much time is spent on duplicated "Binary" suites?
>> *15
>> hours*, or almost *40%* of total "Run All" execution time. It means, that
>> if move suites for *OptimizedMarshaller *(which we consider deprecated)
>> into a separate circuit (which will be executed not per-branch, but e.g.
>> nightly) we will almost double peak TeamCity throughput.
>>
>> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <[hidden email]>
>> wrote:
>>
>>> Igniters,
>>>
>>> I collected several thought-provoking numbers for our TeamCity
>>> (config-variations tests are excluded):
>>>
>>> * Number of Java suites: 78
>>> * Total execution time without build overhead: 38.5 hours
>>> * Total test classes: 3376
>>> * Total tests: 60681
>>>
>>> * 25% of time is consumed by 881 tests (41 test class)
>>> * 50% of time is consumed by 2755 tests (153 test class)
>>> * 75% of time is consumed by 15911 tests (443 test class)
>>>
>>> Or to be more demonstrative:
>>> * 1.5% of tests consume 25% of time
>>> * 4.5% of tests consume 50% of time
>>> * 26.2% tests consume 75% of time
>>>
>>> TOP 3 test classes:
>>> * CacheSerializableTransactionsTest - 46 minutes
>>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
>>> * GridCachePartitionedNodeRestartTest - 25 minutes
>>>
>>> Morale:
>>> 1) We have lots of poor failover/concurrency tests requiring
>>> disproportional time to run;
>>> 2) If we split all tests into "core" and "failover/concurrency" parts
>>> properly, then *core* part will take *~15 mins* to answer you with high
>>> level of confidence whether your change broken anything or not.
>>>
>>> Thoughts?
>>>
>>> Vladimir.
>>>
>>

[Vladimir Ozerov]

Dima,
I do not think we really need more hardware for now. 40 min test will not
run faster if we add more agents. We have a problem with tests quality, not
quantity.

Rohit,
Yes, Ignite is ver advanced framework, so we have about 80-90 different
suites, testing different parts of API.

Vladimir.

On Sat, Dec 17, 2016 at 2:47 AM, Rohit Mohta <[hidden email]> wrote:

> Interesting numbers, to say the least :)
>
> Separating test suites, as suggested by Vladimir sounds good.
>
> Probably off-placed question:
>    As Ignite provides multiple features and is suitable for multiple use
> case- should we categorize test suites based on features?
>
> ------
> Rohit Mohta
>
> > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <[hidden email]>
> wrote:
> >
> > In addition to optimizing the test suites, should we add more TC agents
> as
> > well?
> >
> > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> >> And one more thing - how much time is spent on duplicated "Binary"
> suites?
> >> *15
> >> hours*, or almost *40%* of total "Run All" execution time. It means,
> that
> >> if move suites for *OptimizedMarshaller *(which we consider deprecated)
> >> into a separate circuit (which will be executed not per-branch, but e.g.
> >> nightly) we will almost double peak TeamCity throughput.
> >>
> >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <[hidden email]>
> >> wrote:
> >>
> >>> Igniters,
> >>>
> >>> I collected several thought-provoking numbers for our TeamCity
> >>> (config-variations tests are excluded):
> >>>
> >>> * Number of Java suites: 78
> >>> * Total execution time without build overhead: 38.5 hours
> >>> * Total test classes: 3376
> >>> * Total tests: 60681
> >>>
> >>> * 25% of time is consumed by 881 tests (41 test class)
> >>> * 50% of time is consumed by 2755 tests (153 test class)
> >>> * 75% of time is consumed by 15911 tests (443 test class)
> >>>
> >>> Or to be more demonstrative:
> >>> * 1.5% of tests consume 25% of time
> >>> * 4.5% of tests consume 50% of time
> >>> * 26.2% tests consume 75% of time
> >>>
> >>> TOP 3 test classes:
> >>> * CacheSerializableTransactionsTest - 46 minutes
> >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> >>> * GridCachePartitionedNodeRestartTest - 25 minutes
> >>>
> >>> Morale:
> >>> 1) We have lots of poor failover/concurrency tests requiring
> >>> disproportional time to run;
> >>> 2) If we split all tests into "core" and "failover/concurrency" parts
> >>> properly, then *core* part will take *~15 mins* to answer you with high
> >>> level of confidence whether your change broken anything or not.
> >>>
> >>> Thoughts?
> >>>
> >>> Vladimir.
> >>>
> >>
>

[yzhdanov]

Agree with Vladimir's suggestions. Great analysis!

--Yakov

2016-12-17 11:36 GMT+03:00 Vladimir Ozerov <[hidden email]>:

> Dima,
> I do not think we really need more hardware for now. 40 min test will not
> run faster if we add more agents. We have a problem with tests quality, not
> quantity.
>
> Rohit,
> Yes, Ignite is ver advanced framework, so we have about 80-90 different
> suites, testing different parts of API.
>
> Vladimir.
>
> On Sat, Dec 17, 2016 at 2:47 AM, Rohit Mohta <[hidden email]>
> wrote:
>
> > Interesting numbers, to say the least :)
> >
> > Separating test suites, as suggested by Vladimir sounds good.
> >
> > Probably off-placed question:
> >    As Ignite provides multiple features and is suitable for multiple use
> > case- should we categorize test suites based on features?
> >
> > ------
> > Rohit Mohta
> >
> > > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <[hidden email]>
> > wrote:
> > >
> > > In addition to optimizing the test suites, should we add more TC agents
> > as
> > > well?
> > >
> > > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <[hidden email]
> >
> > > wrote:
> > >
> > >> And one more thing - how much time is spent on duplicated "Binary"
> > suites?
> > >> *15
> > >> hours*, or almost *40%* of total "Run All" execution time. It means,
> > that
> > >> if move suites for *OptimizedMarshaller *(which we consider
> deprecated)
> > >> into a separate circuit (which will be executed not per-branch, but
> e.g.
> > >> nightly) we will almost double peak TeamCity throughput.
> > >>
> > >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <
> [hidden email]>
> > >> wrote:
> > >>
> > >>> Igniters,
> > >>>
> > >>> I collected several thought-provoking numbers for our TeamCity
> > >>> (config-variations tests are excluded):
> > >>>
> > >>> * Number of Java suites: 78
> > >>> * Total execution time without build overhead: 38.5 hours
> > >>> * Total test classes: 3376
> > >>> * Total tests: 60681
> > >>>
> > >>> * 25% of time is consumed by 881 tests (41 test class)
> > >>> * 50% of time is consumed by 2755 tests (153 test class)
> > >>> * 75% of time is consumed by 15911 tests (443 test class)
> > >>>
> > >>> Or to be more demonstrative:
> > >>> * 1.5% of tests consume 25% of time
> > >>> * 4.5% of tests consume 50% of time
> > >>> * 26.2% tests consume 75% of time
> > >>>
> > >>> TOP 3 test classes:
> > >>> * CacheSerializableTransactionsTest - 46 minutes
> > >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> > >>> * GridCachePartitionedNodeRestartTest - 25 minutes
> > >>>
> > >>> Morale:
> > >>> 1) We have lots of poor failover/concurrency tests requiring
> > >>> disproportional time to run;
> > >>> 2) If we split all tests into "core" and "failover/concurrency" parts
> > >>> properly, then *core* part will take *~15 mins* to answer you with
> high
> > >>> level of confidence whether your change broken anything or not.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> Vladimir.
> > >>>
> > >>
> >
>