Apache Ignite Developers - Legacy Mail Archive

Nodes starts in tests

Classic

List

Threaded

10 messages Options

Sergi

Nodes starts in tests

Guys,

A little observation. We have really many tests which start and stop nodes
for each test method.
Example is IgniteCacheAbstractQuerySelfTest, it contains about 30 test
methods and 3 subclasses.
In beforeTest method it starts nodes in afterTest it stops them, while
cache setup is the same.

What I'm trying to say is that each test method itself takes milliseconds
to run, but grid start/stop takes* more than 5 seconds* for each test
method. So from the standpoint of time we are testing grid start/stop all
the time. It is not surprising that our tests take very long time to finish.

Since we already support dynamic cache start/destroy which are much more
fast than grid start/stop,
I think we should go through a refactoring and use existing nodes within a
suite as much as possible.
I believe this can reduce run time of cache related tests like 10 times.

Sergi

yzhdanov

Re: Nodes starts in tests

Very cool idea! However this will not be working in 100% cases since
sometimes we need custom grid configurations, SPIs, etc, but for
significant number of tests this will work.

--Yakov

2015-09-20 11:00 GMT+03:00 Sergi Vladykin <[hidden email]>:

> Guys,
>
> A little observation. We have really many tests which start and stop nodes
> for each test method.
> Example is IgniteCacheAbstractQuerySelfTest, it contains about 30 test
> methods and 3 subclasses.
> In beforeTest method it starts nodes in afterTest it stops them, while
> cache setup is the same.
>
> What I'm trying to say is that each test method itself takes milliseconds
> to run, but grid start/stop takes* more than 5 seconds* for each test
> method. So from the standpoint of time we are testing grid start/stop all
> the time. It is not surprising that our tests take very long time to
> finish.
>
> Since we already support dynamic cache start/destroy which are much more
> fast than grid start/stop,
> I think we should go through a refactoring and use existing nodes within a
> suite as much as possible.
> I believe this can reduce run time of cache related tests like 10 times.
>
> Sergi
>

Branko Čibej

Re: Nodes starts in tests

On 20.09.2015 10:11, Yakov Zhdanov wrote:
> Very cool idea! However this will not be working in 100% cases since
> sometimes we need custom grid configurations, SPIs, etc, but for
> significant number of tests this will work.

This is actually, IMO, a very bad idea. One of the most important
principles when designing tests is that each test case should be
independent of others. In practice that means that if you run test cases
in random order, their results should always be the same. If a test case
depends on a particular way the grid is started up, then moving the grid
startup outside the test case will introduce dependencies between test
cases that could hide bugs.

The primary goal of a test suite is not to be efficient and fast but to
be accurate.

-- Brane

> 2015-09-20 11:00 GMT+03:00 Sergi Vladykin <[hidden email]>:
>
>> Guys,
>>
>> A little observation. We have really many tests which start and stop nodes
>> for each test method.
>> Example is IgniteCacheAbstractQuerySelfTest, it contains about 30 test
>> methods and 3 subclasses.
>> In beforeTest method it starts nodes in afterTest it stops them, while
>> cache setup is the same.
>>
>> What I'm trying to say is that each test method itself takes milliseconds
>> to run, but grid start/stop takes* more than 5 seconds* for each test
>> method. So from the standpoint of time we are testing grid start/stop all
>> the time. It is not surprising that our tests take very long time to
>> finish.
>>
>> Since we already support dynamic cache start/destroy which are much more
>> fast than grid start/stop,
>> I think we should go through a refactoring and use existing nodes within a
>> suite as much as possible.
>> I believe this can reduce run time of cache related tests like 10 times.
>>
>> Sergi
>>

yzhdanov

Re: Nodes starts in tests

Brane, to be fully accurate we should restart the machine and login to the
OS to start each new test, but this is an overhead.

In terms of Ignite in most cases Serj's idea will be fine, I think.

I absolutely agree on accuracy point with you. It should never suffer.

--Yakov

2015-09-20 13:34 GMT+03:00 Branko Čibej <[hidden email]>:

> On 20.09.2015 10:11, Yakov Zhdanov wrote:
> > Very cool idea! However this will not be working in 100% cases since
> > sometimes we need custom grid configurations, SPIs, etc, but for
> > significant number of tests this will work.
>
> This is actually, IMO, a very bad idea. One of the most important
> principles when designing tests is that each test case should be
> independent of others. In practice that means that if you run test cases
> in random order, their results should always be the same. If a test case
> depends on a particular way the grid is started up, then moving the grid
> startup outside the test case will introduce dependencies between test
> cases that could hide bugs.
>
> The primary goal of a test suite is not to be efficient and fast but to
> be accurate.
>
> -- Brane
>
>
> > 2015-09-20 11:00 GMT+03:00 Sergi Vladykin <[hidden email]>:
> >
> >> Guys,
> >>
> >> A little observation. We have really many tests which start and stop
> nodes
> >> for each test method.
> >> Example is IgniteCacheAbstractQuerySelfTest, it contains about 30 test
> >> methods and 3 subclasses.
> >> In beforeTest method it starts nodes in afterTest it stops them, while
> >> cache setup is the same.
> >>
> >> What I'm trying to say is that each test method itself takes
> milliseconds
> >> to run, but grid start/stop takes* more than 5 seconds* for each test
> >> method. So from the standpoint of time we are testing grid start/stop
> all
> >> the time. It is not surprising that our tests take very long time to
> >> finish.
> >>
> >> Since we already support dynamic cache start/destroy which are much more
> >> fast than grid start/stop,
> >> I think we should go through a refactoring and use existing nodes
> within a
> >> suite as much as possible.
> >> I believe this can reduce run time of cache related tests like 10 times.
> >>
> >> Sergi
> >>
>
>

dsetrakyan

Re: Nodes starts in tests

In reply to this post by Branko Čibej

On Sun, Sep 20, 2015 at 1:34 PM, Branko Čibej <[hidden email]> wrote:

Brane, if we test cache behavior (which we do a lot), it is enough to
create/destroy caches using the same node in each test, rather than
start/stop the node itself. Every test should use its own instance of
cache, thus not depending on other tests or test order.

I believe that this will speed up test execution in many cases.

D.

>
> -- Brane
>
>
> > 2015-09-20 11:00 GMT+03:00 Sergi Vladykin <[hidden email]>:
> >
> >> Guys,
> >>
> >> A little observation. We have really many tests which start and stop
> nodes
> >> for each test method.
> >> Example is IgniteCacheAbstractQuerySelfTest, it contains about 30 test
> >> methods and 3 subclasses.
> >> In beforeTest method it starts nodes in afterTest it stops them, while
> >> cache setup is the same.
> >>
> >> What I'm trying to say is that each test method itself takes
> milliseconds
> >> to run, but grid start/stop takes* more than 5 seconds* for each test
> >> method. So from the standpoint of time we are testing grid start/stop
> all
> >> the time. It is not surprising that our tests take very long time to
> >> finish.
> >>
> >> Since we already support dynamic cache start/destroy which are much more
> >> fast than grid start/stop,
> >> I think we should go through a refactoring and use existing nodes
> within a
> >> suite as much as possible.
> >> I believe this can reduce run time of cache related tests like 10 times.
> >>
> >> Sergi
> >>
>
>

Vladimir Ozerov

Re: Nodes starts in tests

This Friday I ran unit tests which starts about 10 nodes in a single JVM.
It took about a second or so on Windows (!!!) when localhost is set to
127.0.0.1 and IP finder is shared.

I believe with can already gain signicant speedup with only two steps:
1) Fix tests where nodes misatkenly start in "beforeTest" instead of
'beforeTestsStarted".
2) Fix test configurations where it is possible to use locahost and shared
IP finder.

On Sun, Sep 20, 2015 at 1:46 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Sun, Sep 20, 2015 at 1:34 PM, Branko Čibej <[hidden email]> wrote:
>
> > On 20.09.2015 10:11, Yakov Zhdanov wrote:
> > > Very cool idea! However this will not be working in 100% cases since
> > > sometimes we need custom grid configurations, SPIs, etc, but for
> > > significant number of tests this will work.
> >
> > This is actually, IMO, a very bad idea. One of the most important
> > principles when designing tests is that each test case should be
> > independent of others. In practice that means that if you run test cases
> > in random order, their results should always be the same. If a test case
> > depends on a particular way the grid is started up, then moving the grid
> > startup outside the test case will introduce dependencies between test
> > cases that could hide bugs.
> >
> > The primary goal of a test suite is not to be efficient and fast but to
> > be accurate.
> >
>
> Brane, if we test cache behavior (which we do a lot), it is enough to
> create/destroy caches using the same node in each test, rather than
> start/stop the node itself. Every test should use its own instance of
> cache, thus not depending on other tests or test order.
>
> I believe that this will speed up test execution in many cases.
>
> D.
>
>
> >
> > -- Brane
> >
> >
> > > 2015-09-20 11:00 GMT+03:00 Sergi Vladykin <[hidden email]>:
> > >
> > >> Guys,
> > >>
> > >> A little observation. We have really many tests which start and stop
> > nodes
> > >> for each test method.
> > >> Example is IgniteCacheAbstractQuerySelfTest, it contains about 30 test
> > >> methods and 3 subclasses.
> > >> In beforeTest method it starts nodes in afterTest it stops them, while
> > >> cache setup is the same.
> > >>
> > >> What I'm trying to say is that each test method itself takes
> > milliseconds
> > >> to run, but grid start/stop takes* more than 5 seconds* for each test
> > >> method. So from the standpoint of time we are testing grid start/stop
> > all
> > >> the time. It is not surprising that our tests take very long time to
> > >> finish.
> > >>
> > >> Since we already support dynamic cache start/destroy which are much
> more
> > >> fast than grid start/stop,
> > >> I think we should go through a refactoring and use existing nodes
> > within a
> > >> suite as much as possible.
> > >> I believe this can reduce run time of cache related tests like 10
> times.
> > >>
> > >> Sergi
> > >>
> >
> >
>

yzhdanov

Re: Nodes starts in tests

We can analyze how much time is spent in nodes start/stop via build logs. I
think this will be quiet significant amount.

I would try Sergi's approach in new tests we ads and see what tests can be
reapproached and what we can gain from it.

Yakov
On Sep 20, 2015 14:18, "Vladimir Ozerov" <[hidden email]> wrote:

> This Friday I ran unit tests which starts about 10 nodes in a single JVM.
> It took about a second or so on Windows (!!!) when localhost is set to
> 127.0.0.1 and IP finder is shared.
>
> I believe with can already gain signicant speedup with only two steps:
> 1) Fix tests where nodes misatkenly start in "beforeTest" instead of
> 'beforeTestsStarted".
> 2) Fix test configurations where it is possible to use locahost and shared
> IP finder.
>
> On Sun, Sep 20, 2015 at 1:46 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > On Sun, Sep 20, 2015 at 1:34 PM, Branko Čibej <[hidden email]> wrote:
> >
> > > On 20.09.2015 10:11, Yakov Zhdanov wrote:
> > > > Very cool idea! However this will not be working in 100% cases since
> > > > sometimes we need custom grid configurations, SPIs, etc, but for
> > > > significant number of tests this will work.
> > >
> > > This is actually, IMO, a very bad idea. One of the most important
> > > principles when designing tests is that each test case should be
> > > independent of others. In practice that means that if you run test
> cases
> > > in random order, their results should always be the same. If a test
> case
> > > depends on a particular way the grid is started up, then moving the
> grid
> > > startup outside the test case will introduce dependencies between test
> > > cases that could hide bugs.
> > >
> > > The primary goal of a test suite is not to be efficient and fast but to
> > > be accurate.
> > >
> >
> > Brane, if we test cache behavior (which we do a lot), it is enough to
> > create/destroy caches using the same node in each test, rather than
> > start/stop the node itself. Every test should use its own instance of
> > cache, thus not depending on other tests or test order.
> >
> > I believe that this will speed up test execution in many cases.
> >
> > D.
> >
> >
> > >
> > > -- Brane
> > >
> > >
> > > > 2015-09-20 11:00 GMT+03:00 Sergi Vladykin <[hidden email]
> >:
> > > >
> > > >> Guys,
> > > >>
> > > >> A little observation. We have really many tests which start and stop
> > > nodes
> > > >> for each test method.
> > > >> Example is IgniteCacheAbstractQuerySelfTest, it contains about 30
> test
> > > >> methods and 3 subclasses.
> > > >> In beforeTest method it starts nodes in afterTest it stops them,
> while
> > > >> cache setup is the same.
> > > >>
> > > >> What I'm trying to say is that each test method itself takes
> > > milliseconds
> > > >> to run, but grid start/stop takes* more than 5 seconds* for each
> test
> > > >> method. So from the standpoint of time we are testing grid
> start/stop
> > > all
> > > >> the time. It is not surprising that our tests take very long time to
> > > >> finish.
> > > >>
> > > >> Since we already support dynamic cache start/destroy which are much
> > more
> > > >> fast than grid start/stop,
> > > >> I think we should go through a refactoring and use existing nodes
> > > within a
> > > >> suite as much as possible.
> > > >> I believe this can reduce run time of cache related tests like 10
> > times.
> > > >>
> > > >> Sergi
> > > >>
> > >
> > >
> >
>

Sergey Kozlov

Fwd: Nodes starts in tests

In reply to this post by dsetrakyan

+1 Sergi idea

Create/destroy cache with unique names should be enough to isolated test
logic.
If failed test case depends which tests were executed (or weren't
executed) earlier the it means a bug or non-determenistic test code.

---------- Forwarded message ----------
From: Dmitriy Setrakyan <[hidden email]>
Date: Sun, Sep 20, 2015 at 1:46 PM
Subject: Re: Nodes starts in tests
To: [hidden email]

On Sun, Sep 20, 2015 at 1:34 PM, Branko Čibej <[hidden email]> wrote:

more
> >> fast than grid start/stop,
> >> I think we should go through a refactoring and use existing nodes
> within a
> >> suite as much as possible.
> >> I believe this can reduce run time of cache related tests like 10
times.
> >>
> >> Sergi
> >>
>
>

--
Sergey Kozlov

Sergi

Re: Nodes starts in tests

Vladimir,

Setting setLocalHost to 127.0.0.1 with shared ipfinder did not help me much
on my mac, at least on tests with awaitPartitionMapExchange.

As far as I understand we don't need to do awaitPartitionMapExchange when
starting cache dynamically on stable topology?

Sergi

2015-09-20 14:25 GMT+03:00 Sergey Kozlov <[hidden email]>:

> +1 Sergi idea
>
> Create/destroy cache with unique names should be enough to isolated test
> logic.
> If failed test case depends which tests were executed (or weren't
> executed) earlier the it means a bug or non-determenistic test code.
>
> ---------- Forwarded message ----------
> From: Dmitriy Setrakyan <[hidden email]>
> Date: Sun, Sep 20, 2015 at 1:46 PM
> Subject: Re: Nodes starts in tests
> To: [hidden email]
>
>
> On Sun, Sep 20, 2015 at 1:34 PM, Branko Čibej <[hidden email]> wrote:
>
> > On 20.09.2015 10:11, Yakov Zhdanov wrote:
> > > Very cool idea! However this will not be working in 100% cases since
> > > sometimes we need custom grid configurations, SPIs, etc, but for
> > > significant number of tests this will work.
> >
> > This is actually, IMO, a very bad idea. One of the most important
> > principles when designing tests is that each test case should be
> > independent of others. In practice that means that if you run test cases
> > in random order, their results should always be the same. If a test case
> > depends on a particular way the grid is started up, then moving the grid
> > startup outside the test case will introduce dependencies between test
> > cases that could hide bugs.
> >
> > The primary goal of a test suite is not to be efficient and fast but to
> > be accurate.
> >
>
> Brane, if we test cache behavior (which we do a lot), it is enough to
> create/destroy caches using the same node in each test, rather than
> start/stop the node itself. Every test should use its own instance of
> cache, thus not depending on other tests or test order.
>
> I believe that this will speed up test execution in many cases.
>
> D.
>
>
> >
> > -- Brane
> >
> >
> > > 2015-09-20 11:00 GMT+03:00 Sergi Vladykin <[hidden email]>:
> > >
> > >> Guys,
> > >>
> > >> A little observation. We have really many tests which start and stop
> > nodes
> > >> for each test method.
> > >> Example is IgniteCacheAbstractQuerySelfTest, it contains about 30 test
> > >> methods and 3 subclasses.
> > >> In beforeTest method it starts nodes in afterTest it stops them, while
> > >> cache setup is the same.
> > >>
> > >> What I'm trying to say is that each test method itself takes
> > milliseconds
> > >> to run, but grid start/stop takes* more than 5 seconds* for each test
> > >> method. So from the standpoint of time we are testing grid start/stop
> > all
> > >> the time. It is not surprising that our tests take very long time to
> > >> finish.
> > >>
> > >> Since we already support dynamic cache start/destroy which are much
> more
> > >> fast than grid start/stop,
> > >> I think we should go through a refactoring and use existing nodes
> > within a
> > >> suite as much as possible.
> > >> I believe this can reduce run time of cache related tests like 10
> times.
> > >>
> > >> Sergi
> > >>
> >
> >
>
>
>
> --
> Sergey Kozlov
>

yzhdanov

Re: Nodes starts in tests

We still should. But in this case it will be faster.

--Yakov

2015-09-20 20:51 GMT+03:00 Sergi Vladykin <[hidden email]>:

> Vladimir,
>
> Setting setLocalHost to 127.0.0.1 with shared ipfinder did not help me much
> on my mac, at least on tests with awaitPartitionMapExchange.
>
> As far as I understand we don't need to do awaitPartitionMapExchange when
> starting cache dynamically on stable topology?
>
> Sergi
>
> 2015-09-20 14:25 GMT+03:00 Sergey Kozlov <[hidden email]>:
>
> > +1 Sergi idea
> >
> > Create/destroy cache with unique names should be enough to isolated test
> > logic.
> > If failed test case depends which tests were executed (or weren't
> > executed) earlier the it means a bug or non-determenistic test code.
> >
> > ---------- Forwarded message ----------
> > From: Dmitriy Setrakyan <[hidden email]>
> > Date: Sun, Sep 20, 2015 at 1:46 PM
> > Subject: Re: Nodes starts in tests
> > To: [hidden email]
> >
> >
> > On Sun, Sep 20, 2015 at 1:34 PM, Branko Čibej <[hidden email]> wrote:
> >
> > > On 20.09.2015 10:11, Yakov Zhdanov wrote:
> > > > Very cool idea! However this will not be working in 100% cases since
> > > > sometimes we need custom grid configurations, SPIs, etc, but for
> > > > significant number of tests this will work.
> > >
> > > This is actually, IMO, a very bad idea. One of the most important
> > > principles when designing tests is that each test case should be
> > > independent of others. In practice that means that if you run test
> cases
> > > in random order, their results should always be the same. If a test
> case
> > > depends on a particular way the grid is started up, then moving the
> grid
> > > startup outside the test case will introduce dependencies between test
> > > cases that could hide bugs.
> > >
> > > The primary goal of a test suite is not to be efficient and fast but to
> > > be accurate.
> > >
> >
> > Brane, if we test cache behavior (which we do a lot), it is enough to
> > create/destroy caches using the same node in each test, rather than
> > start/stop the node itself. Every test should use its own instance of
> > cache, thus not depending on other tests or test order.
> >
> > I believe that this will speed up test execution in many cases.
> >
> > D.
> >
> >
> > >
> > > -- Brane
> > >
> > >
> > > > 2015-09-20 11:00 GMT+03:00 Sergi Vladykin <[hidden email]
> >:
> > > >
> > > >> Guys,
> > > >>
> > > >> A little observation. We have really many tests which start and stop
> > > nodes
> > > >> for each test method.
> > > >> Example is IgniteCacheAbstractQuerySelfTest, it contains about 30
> test
> > > >> methods and 3 subclasses.
> > > >> In beforeTest method it starts nodes in afterTest it stops them,
> while
> > > >> cache setup is the same.
> > > >>
> > > >> What I'm trying to say is that each test method itself takes
> > > milliseconds
> > > >> to run, but grid start/stop takes* more than 5 seconds* for each
> test
> > > >> method. So from the standpoint of time we are testing grid
> start/stop
> > > all
> > > >> the time. It is not surprising that our tests take very long time to
> > > >> finish.
> > > >>
> > > >> Since we already support dynamic cache start/destroy which are much
> > more
> > > >> fast than grid start/stop,
> > > >> I think we should go through a refactoring and use existing nodes
> > > within a
> > > >> suite as much as possible.
> > > >> I believe this can reduce run time of cache related tests like 10
> > times.
> > > >>
> > > >> Sergi
> > > >>
> > >
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> >
>