Merge small test suites into one big suite.

[Vladimir Ozerov]

Igniters,

Currently we have about 60-70 test suites on Team City. When I look at
results I see this:

Ignite 150 Clients -> 1 test -> 6m
Ignite AWS -> 10 tests -> 11m
Ignite GCE -> 1 test -> 3m
Ignite Geospacial Indexing -> 11 tests -> 4m
Ignite IGFS Examples -> 1 test -> 3m
Ignite Logging -> 1 test -> 3m

Etc.

It is natural to split tests by their modules. But in this case we spend
most of the time on compiling the project over and over again just to run
several tests. It consumes lots of TeamCity resources. Both CPU for
compilation, and disk for logs.

I think we can safely merge all these tine suites into one single suite and
call it "Ignite Integrations". This will save us a lot of time.

Note that I mean not each and every integration module, but only modules
where we have no more than a 1-2 dozens of tests.

Thoughts?

Vladimir.

[Alexey Kuznetsov-2]

I would merge them only if they ALL are "green".

or

May be better solution - build ONCE and REUSE binaries for test?



On Tue, Sep 27, 2016 at 7:25 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Igniters,
>
> Currently we have about 60-70 test suites on Team City. When I look at
> results I see this:
>
> Ignite 150 Clients -> 1 test -> 6m
> Ignite AWS -> 10 tests -> 11m
> Ignite GCE -> 1 test -> 3m
> Ignite Geospacial Indexing -> 11 tests -> 4m
> Ignite IGFS Examples -> 1 test -> 3m
> Ignite Logging -> 1 test -> 3m
>
> Etc.
>
> It is natural to split tests by their modules. But in this case we spend
> most of the time on compiling the project over and over again just to run
> several tests. It consumes lots of TeamCity resources. Both CPU for
> compilation, and disk for logs.
>
> I think we can safely merge all these tine suites into one single suite and
> call it "Ignite Integrations". This will save us a lot of time.
>
> Note that I mean not each and every integration module, but only modules
> where we have no more than a 1-2 dozens of tests.
>
> Thoughts?
>
> Vladimir.
>

--
Alexey Kuznetsov

[Pavel Tupitsyn]

+1 for reusing built binaries.

* Even minimal Java build (with excluded Scala, Javadoc, etc) takes over 5
minutes
* There are almost 100 suites currently

So we waste around 1 hour of agents time for each full TC run.

Joining small suites also makes sense, even with reused binaries, because
of other kinds of overhead (checkout).

On Tue, Sep 27, 2016 at 3:38 PM, Alexey Kuznetsov <[hidden email]>
wrote:

> I would merge them only if they ALL are "green".
>
> or
>
> May be better solution - build ONCE and REUSE binaries for test?
>
>
>
> On Tue, Sep 27, 2016 at 7:25 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > Currently we have about 60-70 test suites on Team City. When I look at
> > results I see this:
> >
> > Ignite 150 Clients -> 1 test -> 6m
> > Ignite AWS -> 10 tests -> 11m
> > Ignite GCE -> 1 test -> 3m
> > Ignite Geospacial Indexing -> 11 tests -> 4m
> > Ignite IGFS Examples -> 1 test -> 3m
> > Ignite Logging -> 1 test -> 3m
> >
> > Etc.
> >
> > It is natural to split tests by their modules. But in this case we spend
> > most of the time on compiling the project over and over again just to run
> > several tests. It consumes lots of TeamCity resources. Both CPU for
> > compilation, and disk for logs.
> >
> > I think we can safely merge all these tine suites into one single suite
> and
> > call it "Ignite Integrations". This will save us a lot of time.
> >
> > Note that I mean not each and every integration module, but only modules
> > where we have no more than a 1-2 dozens of tests.
> >
> > Thoughts?
> >
> > Vladimir.
> >
>
>
>
> --
> Alexey Kuznetsov
>

[Anton Vinogradov]

First of all we should undestand how many time we spend on compilation.
Im not sure that compilation required for test takes more than 20 secons.

Even minimal Java build (with excluded Scala, Javadoc, etc) takes over 5

Don't mix up project build and compilation required for tests.

But I see that each test spends time on this:
Step 3/5: Build P2P classes (Maven) (33s)
Is it necessary ?

On Tue, Sep 27, 2016 at 3:50 PM, Pavel Tupitsyn <[hidden email]>
wrote:

> +1 for reusing built binaries.
>
> * Even minimal Java build (with excluded Scala, Javadoc, etc) takes over 5
> minutes
> * There are almost 100 suites currently
>
> So we waste around 1 hour of agents time for each full TC run.
>
> Joining small suites also makes sense, even with reused binaries, because
> of other kinds of overhead (checkout).
>
> On Tue, Sep 27, 2016 at 3:38 PM, Alexey Kuznetsov <[hidden email]
> >
> wrote:
>
> > I would merge them only if they ALL are "green".
> >
> > or
> >
> > May be better solution - build ONCE and REUSE binaries for test?
> >
> >
> >
> > On Tue, Sep 27, 2016 at 7:25 PM, Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Igniters,
> > >
> > > Currently we have about 60-70 test suites on Team City. When I look at
> > > results I see this:
> > >
> > > Ignite 150 Clients -> 1 test -> 6m
> > > Ignite AWS -> 10 tests -> 11m
> > > Ignite GCE -> 1 test -> 3m
> > > Ignite Geospacial Indexing -> 11 tests -> 4m
> > > Ignite IGFS Examples -> 1 test -> 3m
> > > Ignite Logging -> 1 test -> 3m
> > >
> > > Etc.
> > >
> > > It is natural to split tests by their modules. But in this case we
> spend
> > > most of the time on compiling the project over and over again just to
> run
> > > several tests. It consumes lots of TeamCity resources. Both CPU for
> > > compilation, and disk for logs.
> > >
> > > I think we can safely merge all these tine suites into one single suite
> > and
> > > call it "Ignite Integrations". This will save us a lot of time.
> > >
> > > Note that I mean not each and every integration module, but only
> modules
> > > where we have no more than a 1-2 dozens of tests.
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> >
> >
> >
> > --
> > Alexey Kuznetsov
> >
>

[Alexey Kuznetsov-2]

I see following possible ways of optimize compilation:

1. Build ALL once and after that run tests (may be complicated to
implement).

2. We could tune test compilation steps by specifying only needed modules
in maven command.
If we running test for ignite-core, lets compile only ignite-core and do
not compile scala.
This will require to take a look to all suites build steps.

On Tue, Sep 27, 2016 at 7:54 PM, Anton Vinogradov <[hidden email]>
wrote:

> First of all we should undestand how many time we spend on compilation.
> Im not sure that compilation required for test takes more than 20 secons.
>
> Even minimal Java build (with excluded Scala, Javadoc, etc) takes over 5
>
> Don't mix up project build and compilation required for tests.
>
> But I see that each test spends time on this:
> Step 3/5: Build P2P classes (Maven) (33s)
> Is it necessary ?
>
> On Tue, Sep 27, 2016 at 3:50 PM, Pavel Tupitsyn <[hidden email]>
> wrote:
>
> > +1 for reusing built binaries.
> >
> > * Even minimal Java build (with excluded Scala, Javadoc, etc) takes over
> 5
> > minutes
> > * There are almost 100 suites currently
> >
> > So we waste around 1 hour of agents time for each full TC run.
> >
> > Joining small suites also makes sense, even with reused binaries, because
> > of other kinds of overhead (checkout).
> >
> > On Tue, Sep 27, 2016 at 3:38 PM, Alexey Kuznetsov <
> [hidden email]
> > >
> > wrote:
> >
> > > I would merge them only if they ALL are "green".
> > >
> > > or
> > >
> > > May be better solution - build ONCE and REUSE binaries for test?
> > >
> > >
> > >
> > > On Tue, Sep 27, 2016 at 7:25 PM, Vladimir Ozerov <[hidden email]
> >
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > Currently we have about 60-70 test suites on Team City. When I look
> at
> > > > results I see this:
> > > >
> > > > Ignite 150 Clients -> 1 test -> 6m
> > > > Ignite AWS -> 10 tests -> 11m
> > > > Ignite GCE -> 1 test -> 3m
> > > > Ignite Geospacial Indexing -> 11 tests -> 4m
> > > > Ignite IGFS Examples -> 1 test -> 3m
> > > > Ignite Logging -> 1 test -> 3m
> > > >
> > > > Etc.
> > > >
> > > > It is natural to split tests by their modules. But in this case we
> > spend
> > > > most of the time on compiling the project over and over again just to
> > run
> > > > several tests. It consumes lots of TeamCity resources. Both CPU for
> > > > compilation, and disk for logs.
> > > >
> > > > I think we can safely merge all these tine suites into one single
> suite
> > > and
> > > > call it "Ignite Integrations". This will save us a lot of time.
> > > >
> > > > Note that I mean not each and every integration module, but only
> > modules
> > > > where we have no more than a 1-2 dozens of tests.
> > > >
> > > > Thoughts?
> > > >
> > > > Vladimir.
> > > >
> > >
> > >
> > >
> > > --
> > > Alexey Kuznetsov
> > >
> >
>

--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

[Anton Vinogradov]

Guys, lets continue discussion after we gain metrics.
Simplest check is to create accumulated TC task (which will run all test
suites) and check how fast is it.

On Tue, Sep 27, 2016 at 4:09 PM, Alexey Kuznetsov <[hidden email]>
wrote:

> I see following possible ways of optimize compilation:
>
> 1. Build ALL once and after that run tests (may be complicated to
> implement).
>
> 2. We could tune test compilation steps by specifying only needed modules
> in maven command.
> If we running test for ignite-core, lets compile only ignite-core and do
> not compile scala.
> This will require to take a look to all suites build steps.
>
> On Tue, Sep 27, 2016 at 7:54 PM, Anton Vinogradov <
> [hidden email]>
> wrote:
>
> > First of all we should undestand how many time we spend on compilation.
> > Im not sure that compilation required for test takes more than 20 secons.
> >
> > Even minimal Java build (with excluded Scala, Javadoc, etc) takes over 5
> >
> > Don't mix up project build and compilation required for tests.
> >
> > But I see that each test spends time on this:
> > Step 3/5: Build P2P classes (Maven) (33s)
> > Is it necessary ?
> >
> > On Tue, Sep 27, 2016 at 3:50 PM, Pavel Tupitsyn <[hidden email]>
> > wrote:
> >
> > > +1 for reusing built binaries.
> > >
> > > * Even minimal Java build (with excluded Scala, Javadoc, etc) takes
> over
> > 5
> > > minutes
> > > * There are almost 100 suites currently
> > >
> > > So we waste around 1 hour of agents time for each full TC run.
> > >
> > > Joining small suites also makes sense, even with reused binaries,
> because
> > > of other kinds of overhead (checkout).
> > >
> > > On Tue, Sep 27, 2016 at 3:38 PM, Alexey Kuznetsov <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > I would merge them only if they ALL are "green".
> > > >
> > > > or
> > > >
> > > > May be better solution - build ONCE and REUSE binaries for test?
> > > >
> > > >
> > > >
> > > > On Tue, Sep 27, 2016 at 7:25 PM, Vladimir Ozerov <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > Currently we have about 60-70 test suites on Team City. When I look
> > at
> > > > > results I see this:
> > > > >
> > > > > Ignite 150 Clients -> 1 test -> 6m
> > > > > Ignite AWS -> 10 tests -> 11m
> > > > > Ignite GCE -> 1 test -> 3m
> > > > > Ignite Geospacial Indexing -> 11 tests -> 4m
> > > > > Ignite IGFS Examples -> 1 test -> 3m
> > > > > Ignite Logging -> 1 test -> 3m
> > > > >
> > > > > Etc.
> > > > >
> > > > > It is natural to split tests by their modules. But in this case we
> > > spend
> > > > > most of the time on compiling the project over and over again just
> to
> > > run
> > > > > several tests. It consumes lots of TeamCity resources. Both CPU for
> > > > > compilation, and disk for logs.
> > > > >
> > > > > I think we can safely merge all these tine suites into one single
> > suite
> > > > and
> > > > > call it "Ignite Integrations". This will save us a lot of time.
> > > > >
> > > > > Note that I mean not each and every integration module, but only
> > > modules
> > > > > where we have no more than a 1-2 dozens of tests.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vladimir.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alexey Kuznetsov
> > > >
> > >
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>

[Vladimir Ozerov]

Anton,

What metrics do you need? Most probably everything already can be found in
logs.

E.g.:
http://172.25.1.150:8111/viewLog.html?buildId=323412&buildTypeId=IgniteTests_Gce&tab=buildLog#_focus=2450

Highlights:
- p2p stuff build took 40s
- ignite-core took ~1m to compile


On Tue, Sep 27, 2016 at 4:30 PM, Anton Vinogradov <[hidden email]>
wrote:

> Guys, lets continue discussion after we gain metrics.
> Simplest check is to create accumulated TC task (which will run all test
> suites) and check how fast is it.
>
> On Tue, Sep 27, 2016 at 4:09 PM, Alexey Kuznetsov <[hidden email]
> >
> wrote:
>
> > I see following possible ways of optimize compilation:
> >
> > 1. Build ALL once and after that run tests (may be complicated to
> > implement).
> >
> > 2. We could tune test compilation steps by specifying only needed modules
> > in maven command.
> > If we running test for ignite-core, lets compile only ignite-core and do
> > not compile scala.
> > This will require to take a look to all suites build steps.
> >
> > On Tue, Sep 27, 2016 at 7:54 PM, Anton Vinogradov <
> > [hidden email]>
> > wrote:
> >
> > > First of all we should undestand how many time we spend on compilation.
> > > Im not sure that compilation required for test takes more than 20
> secons.
> > >
> > > Even minimal Java build (with excluded Scala, Javadoc, etc) takes over
> 5
> > >
> > > Don't mix up project build and compilation required for tests.
> > >
> > > But I see that each test spends time on this:
> > > Step 3/5: Build P2P classes (Maven) (33s)
> > > Is it necessary ?
> > >
> > > On Tue, Sep 27, 2016 at 3:50 PM, Pavel Tupitsyn <[hidden email]>
> > > wrote:
> > >
> > > > +1 for reusing built binaries.
> > > >
> > > > * Even minimal Java build (with excluded Scala, Javadoc, etc) takes
> > over
> > > 5
> > > > minutes
> > > > * There are almost 100 suites currently
> > > >
> > > > So we waste around 1 hour of agents time for each full TC run.
> > > >
> > > > Joining small suites also makes sense, even with reused binaries,
> > because
> > > > of other kinds of overhead (checkout).
> > > >
> > > > On Tue, Sep 27, 2016 at 3:38 PM, Alexey Kuznetsov <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > I would merge them only if they ALL are "green".
> > > > >
> > > > > or
> > > > >
> > > > > May be better solution - build ONCE and REUSE binaries for test?
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Sep 27, 2016 at 7:25 PM, Vladimir Ozerov <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > Currently we have about 60-70 test suites on Team City. When I
> look
> > > at
> > > > > > results I see this:
> > > > > >
> > > > > > Ignite 150 Clients -> 1 test -> 6m
> > > > > > Ignite AWS -> 10 tests -> 11m
> > > > > > Ignite GCE -> 1 test -> 3m
> > > > > > Ignite Geospacial Indexing -> 11 tests -> 4m
> > > > > > Ignite IGFS Examples -> 1 test -> 3m
> > > > > > Ignite Logging -> 1 test -> 3m
> > > > > >
> > > > > > Etc.
> > > > > >
> > > > > > It is natural to split tests by their modules. But in this case
> we
> > > > spend
> > > > > > most of the time on compiling the project over and over again
> just
> > to
> > > > run
> > > > > > several tests. It consumes lots of TeamCity resources. Both CPU
> for
> > > > > > compilation, and disk for logs.
> > > > > >
> > > > > > I think we can safely merge all these tine suites into one single
> > > suite
> > > > > and
> > > > > > call it "Ignite Integrations". This will save us a lot of time.
> > > > > >
> > > > > > Note that I mean not each and every integration module, but only
> > > > modules
> > > > > > where we have no more than a 1-2 dozens of tests.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Vladimir.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alexey Kuznetsov
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Alexey Kuznetsov
> > GridGain Systems
> > www.gridgain.com
> >
>

[Anton Vinogradov]

Correct, 1+ minute seems to be a problem.
We can add special TC suite which will compile sources and use it as
dependency for each another TC suite.
It can be more suitable than combine suites.

On Tue, Sep 27, 2016 at 4:33 PM, Vladimir Ozerov <[hidden email]>
wrote:

> Anton,
>
> What metrics do you need? Most probably everything already can be found in
> logs.
>
> E.g.:
> http://172.25.1.150:8111/viewLog.html?buildId=323412&
> buildTypeId=IgniteTests_Gce&tab=buildLog#_focus=2450
>
> Highlights:
> - p2p stuff build took 40s
> - ignite-core took ~1m to compile
>
>
> On Tue, Sep 27, 2016 at 4:30 PM, Anton Vinogradov <
> [hidden email]>
> wrote:
>
> > Guys, lets continue discussion after we gain metrics.
> > Simplest check is to create accumulated TC task (which will run all test
> > suites) and check how fast is it.
> >
> > On Tue, Sep 27, 2016 at 4:09 PM, Alexey Kuznetsov <
> [hidden email]
> > >
> > wrote:
> >
> > > I see following possible ways of optimize compilation:
> > >
> > > 1. Build ALL once and after that run tests (may be complicated to
> > > implement).
> > >
> > > 2. We could tune test compilation steps by specifying only needed
> modules
> > > in maven command.
> > > If we running test for ignite-core, lets compile only ignite-core and
> do
> > > not compile scala.
> > > This will require to take a look to all suites build steps.
> > >
> > > On Tue, Sep 27, 2016 at 7:54 PM, Anton Vinogradov <
> > > [hidden email]>
> > > wrote:
> > >
> > > > First of all we should undestand how many time we spend on
> compilation.
> > > > Im not sure that compilation required for test takes more than 20
> > secons.
> > > >
> > > > Even minimal Java build (with excluded Scala, Javadoc, etc) takes
> over
> > 5
> > > >
> > > > Don't mix up project build and compilation required for tests.
> > > >
> > > > But I see that each test spends time on this:
> > > > Step 3/5: Build P2P classes (Maven) (33s)
> > > > Is it necessary ?
> > > >
> > > > On Tue, Sep 27, 2016 at 3:50 PM, Pavel Tupitsyn <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > +1 for reusing built binaries.
> > > > >
> > > > > * Even minimal Java build (with excluded Scala, Javadoc, etc) takes
> > > over
> > > > 5
> > > > > minutes
> > > > > * There are almost 100 suites currently
> > > > >
> > > > > So we waste around 1 hour of agents time for each full TC run.
> > > > >
> > > > > Joining small suites also makes sense, even with reused binaries,
> > > because
> > > > > of other kinds of overhead (checkout).
> > > > >
> > > > > On Tue, Sep 27, 2016 at 3:38 PM, Alexey Kuznetsov <
> > > > [hidden email]
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I would merge them only if they ALL are "green".
> > > > > >
> > > > > > or
> > > > > >
> > > > > > May be better solution - build ONCE and REUSE binaries for test?
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 27, 2016 at 7:25 PM, Vladimir Ozerov <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > Currently we have about 60-70 test suites on Team City. When I
> > look
> > > > at
> > > > > > > results I see this:
> > > > > > >
> > > > > > > Ignite 150 Clients -> 1 test -> 6m
> > > > > > > Ignite AWS -> 10 tests -> 11m
> > > > > > > Ignite GCE -> 1 test -> 3m
> > > > > > > Ignite Geospacial Indexing -> 11 tests -> 4m
> > > > > > > Ignite IGFS Examples -> 1 test -> 3m
> > > > > > > Ignite Logging -> 1 test -> 3m
> > > > > > >
> > > > > > > Etc.
> > > > > > >
> > > > > > > It is natural to split tests by their modules. But in this case
> > we
> > > > > spend
> > > > > > > most of the time on compiling the project over and over again
> > just
> > > to
> > > > > run
> > > > > > > several tests. It consumes lots of TeamCity resources. Both CPU
> > for
> > > > > > > compilation, and disk for logs.
> > > > > > >
> > > > > > > I think we can safely merge all these tine suites into one
> single
> > > > suite
> > > > > > and
> > > > > > > call it "Ignite Integrations". This will save us a lot of time.
> > > > > > >
> > > > > > > Note that I mean not each and every integration module, but
> only
> > > > > modules
> > > > > > > where we have no more than a 1-2 dozens of tests.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Vladimir.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Alexey Kuznetsov
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Alexey Kuznetsov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>