Full API coverage enhancement

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Full API coverage enhancement

Artem Shutak
Igniters,

I'm working on an enhancement of Full API coverage [1] [2].

Ignite already has Full API test, but currently it's hard to test all
configuration combinations.

Feel free to add comments in the jira if you have any thought.

[1] https://issues.apache.org/jira/browse/IGNITE-2521
[2] https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests

Thanks,
-- Artem --
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

dsetrakyan
Artem,

This is great. I have noticed from the ticket that you have created some
initial suite already. Is there a branch I can look at it?

D.

On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]> wrote:

> Igniters,
>
> I'm working on an enhancement of Full API coverage [1] [2].
>
> Ignite already has Full API test, but currently it's hard to test all
> configuration combinations.
>
> Feel free to add comments in the jira if you have any thought.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-2521
> [2] https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
>
> Thanks,
> -- Artem --
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Artem Shutak
Dmitriy,

There is a branch at my fork and a pull request at Ignite. See comment
about pull request at the ticket (PR-446).

But I have to notice that the branch under hard development and you it can
not work (have compilation or test errors) at some moments.

Good luck!

-- Artem --

On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Artem,
>
> This is great. I have noticed from the ticket that you have created some
> initial suite already. Is there a branch I can look at it?
>
> D.
>
> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]>
> wrote:
>
> > Igniters,
> >
> > I'm working on an enhancement of Full API coverage [1] [2].
> >
> > Ignite already has Full API test, but currently it's hard to test all
> > configuration combinations.
> >
> > Feel free to add comments in the jira if you have any thought.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > [2]
> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> >
> > Thanks,
> > -- Artem --
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Artem Shutak
Igniters,

I thought a little bit more and think we need to add a support for the
following permutations too (I've added these to the jira description):
- We should also test with Serializable, Externalizable, and plain Pojos
for keys and values.
- The Pojo in the above test should contain an enum value
- We should also test Enums as keys and Enums as values
- All operations should have single-key and multi-key operations

Maybe someone see any other permutation to be supported?

-- Artem --

On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]> wrote:

> Dmitriy,
>
> There is a branch at my fork and a pull request at Ignite. See comment
> about pull request at the ticket (PR-446).
>
> But I have to notice that the branch under hard development and you it can
> not work (have compilation or test errors) at some moments.
>
> Good luck!
>
> -- Artem --
>
> On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
>> Artem,
>>
>> This is great. I have noticed from the ticket that you have created some
>> initial suite already. Is there a branch I can look at it?
>>
>> D.
>>
>> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]>
>> wrote:
>>
>> > Igniters,
>> >
>> > I'm working on an enhancement of Full API coverage [1] [2].
>> >
>> > Ignite already has Full API test, but currently it's hard to test all
>> > configuration combinations.
>> >
>> > Feel free to add comments in the jira if you have any thought.
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
>> > [2]
>> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
>> >
>> > Thanks,
>> > -- Artem --
>> >
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

dsetrakyan
Artem, I think it is best to specify all the permutations here, so others
can make additional suggestions. Otherwise, we cannot get a full picture.

Thanks,
D.

On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> wrote:

> Igniters,
>
> I thought a little bit more and think we need to add a support for the
> following permutations too (I've added these to the jira description):
> - We should also test with Serializable, Externalizable, and plain Pojos
> for keys and values.
> - The Pojo in the above test should contain an enum value
> - We should also test Enums as keys and Enums as values
> - All operations should have single-key and multi-key operations
>
> Maybe someone see any other permutation to be supported?
>
> -- Artem --
>
> On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]>
> wrote:
>
> > Dmitriy,
> >
> > There is a branch at my fork and a pull request at Ignite. See comment
> > about pull request at the ticket (PR-446).
> >
> > But I have to notice that the branch under hard development and you it
> can
> > not work (have compilation or test errors) at some moments.
> >
> > Good luck!
> >
> > -- Artem --
> >
> > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> >> Artem,
> >>
> >> This is great. I have noticed from the ticket that you have created some
> >> initial suite already. Is there a branch I can look at it?
> >>
> >> D.
> >>
> >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]>
> >> wrote:
> >>
> >> > Igniters,
> >> >
> >> > I'm working on an enhancement of Full API coverage [1] [2].
> >> >
> >> > Ignite already has Full API test, but currently it's hard to test all
> >> > configuration combinations.
> >> >
> >> > Feel free to add comments in the jira if you have any thought.
> >> >
> >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> >> > [2]
> >> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> >> >
> >> > Thanks,
> >> > -- Artem --
> >> >
> >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Artem Shutak
Dmitriy,

Actually, I don't have a list with all the permutations.

At first, we need to split in our discussion test cases and Ignite
configuration which should be covered.

For example, new Full Api test cases for cache are based on old Full Api
test cases. So, it need to think what the test cases was not covered before.

About Ignite configurations, I'm going to add permutation for each
IgniteConfiguration and CacheConfiguration property.

By the way, the jira contains the following list of permutation (feel free
to add something):

The following tests should be added (for functional blocks):

   1. Interceptor
   2. Queries: continuous, scan, SQL, fields and text queries.
   3. cache events
   4. We should also test with Serializable, Externalizable, and plain
   Pojos for keys and values.
   5. The Pojo in the above test should contain an enum value
   6. We should also test Enums as keys and Enums as values
   7. All operations should have single-key and multi-key operations

New tests should cover all combinations for following properties:

   1. cache modes
   2. operation from client nodes and server nodes
   3. store enabled/disabled
   4. evicts sycn/non-sync
   5. eviction policies
   6. near on/off
   7. marshallers (+ Binary marshaller with different mappers)
   8. keys and values - externalizable, serializable, binaryzable, "none of
   previous"
   9. classes available on servers: true/false
   10. Peer loading on/off
   11. Affinity functions
   12. expiry policies


Thanks,
-- Artem --

On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Artem, I think it is best to specify all the permutations here, so others
> can make additional suggestions. Otherwise, we cannot get a full picture.
>
> Thanks,
> D.
>
> On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> wrote:
>
> > Igniters,
> >
> > I thought a little bit more and think we need to add a support for the
> > following permutations too (I've added these to the jira description):
> > - We should also test with Serializable, Externalizable, and plain Pojos
> > for keys and values.
> > - The Pojo in the above test should contain an enum value
> > - We should also test Enums as keys and Enums as values
> > - All operations should have single-key and multi-key operations
> >
> > Maybe someone see any other permutation to be supported?
> >
> > -- Artem --
> >
> > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > There is a branch at my fork and a pull request at Ignite. See comment
> > > about pull request at the ticket (PR-446).
> > >
> > > But I have to notice that the branch under hard development and you it
> > can
> > > not work (have compilation or test errors) at some moments.
> > >
> > > Good luck!
> > >
> > > -- Artem --
> > >
> > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > > wrote:
> > >
> > >> Artem,
> > >>
> > >> This is great. I have noticed from the ticket that you have created
> some
> > >> initial suite already. Is there a branch I can look at it?
> > >>
> > >> D.
> > >>
> > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]>
> > >> wrote:
> > >>
> > >> > Igniters,
> > >> >
> > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > >> >
> > >> > Ignite already has Full API test, but currently it's hard to test
> all
> > >> > configuration combinations.
> > >> >
> > >> > Feel free to add comments in the jira if you have any thought.
> > >> >
> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > >> > [2]
> > >> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > >> >
> > >> > Thanks,
> > >> > -- Artem --
> > >> >
> > >>
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Alexey Kuznetsov-2
Artem, could you describe (in general) how are you going to describe
permutations?

Are you going to run tests with all possible combinations of all possible
properties?

Along time ago I implemented smth. similar (not in java)  and
 I created framework where I could describe nested rules of what I want to
investigate and
 engine that read that rules and iterate over all possible combination of
rules and execute some action.

Maybe this will be useful to you.


On Wed, Feb 3, 2016 at 10:40 PM, Artem Shutak <[hidden email]> wrote:

> Dmitriy,
>
> Actually, I don't have a list with all the permutations.
>
> At first, we need to split in our discussion test cases and Ignite
> configuration which should be covered.
>
> For example, new Full Api test cases for cache are based on old Full Api
> test cases. So, it need to think what the test cases was not covered
> before.
>
> About Ignite configurations, I'm going to add permutation for each
> IgniteConfiguration and CacheConfiguration property.
>
> By the way, the jira contains the following list of permutation (feel free
> to add something):
>
> The following tests should be added (for functional blocks):
>
>    1. Interceptor
>    2. Queries: continuous, scan, SQL, fields and text queries.
>    3. cache events
>    4. We should also test with Serializable, Externalizable, and plain
>    Pojos for keys and values.
>    5. The Pojo in the above test should contain an enum value
>    6. We should also test Enums as keys and Enums as values
>    7. All operations should have single-key and multi-key operations
>
> New tests should cover all combinations for following properties:
>
>    1. cache modes
>    2. operation from client nodes and server nodes
>    3. store enabled/disabled
>    4. evicts sycn/non-sync
>    5. eviction policies
>    6. near on/off
>    7. marshallers (+ Binary marshaller with different mappers)
>    8. keys and values - externalizable, serializable, binaryzable, "none of
>    previous"
>    9. classes available on servers: true/false
>    10. Peer loading on/off
>    11. Affinity functions
>    12. expiry policies
>
>
> Thanks,
> -- Artem --
>
> On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Artem, I think it is best to specify all the permutations here, so others
> > can make additional suggestions. Otherwise, we cannot get a full picture.
> >
> > Thanks,
> > D.
> >
> > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]>
> wrote:
> >
> > > Igniters,
> > >
> > > I thought a little bit more and think we need to add a support for the
> > > following permutations too (I've added these to the jira description):
> > > - We should also test with Serializable, Externalizable, and plain
> Pojos
> > > for keys and values.
> > > - The Pojo in the above test should contain an enum value
> > > - We should also test Enums as keys and Enums as values
> > > - All operations should have single-key and multi-key operations
> > >
> > > Maybe someone see any other permutation to be supported?
> > >
> > > -- Artem --
> > >
> > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]>
> > > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > There is a branch at my fork and a pull request at Ignite. See
> comment
> > > > about pull request at the ticket (PR-446).
> > > >
> > > > But I have to notice that the branch under hard development and you
> it
> > > can
> > > > not work (have compilation or test errors) at some moments.
> > > >
> > > > Good luck!
> > > >
> > > > -- Artem --
> > > >
> > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > >> Artem,
> > > >>
> > > >> This is great. I have noticed from the ticket that you have created
> > some
> > > >> initial suite already. Is there a branch I can look at it?
> > > >>
> > > >> D.
> > > >>
> > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]
> >
> > > >> wrote:
> > > >>
> > > >> > Igniters,
> > > >> >
> > > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > > >> >
> > > >> > Ignite already has Full API test, but currently it's hard to test
> > all
> > > >> > configuration combinations.
> > > >> >
> > > >> > Feel free to add comments in the jira if you have any thought.
> > > >> >
> > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > >> > [2]
> > > >>
> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > >> >
> > > >> > Thanks,
> > > >> > -- Artem --
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>



--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

dsetrakyan
In reply to this post by Artem Shutak
Thanks Artem! In your view, how many combinations of the configuration
properties will be tested together. I don’t think we can afford to test
every possible combination.

On Wed, Feb 3, 2016 at 7:40 AM, Artem Shutak <[hidden email]> wrote:

> Dmitriy,
>
> Actually, I don't have a list with all the permutations.
>
> At first, we need to split in our discussion test cases and Ignite
> configuration which should be covered.
>
> For example, new Full Api test cases for cache are based on old Full Api
> test cases. So, it need to think what the test cases was not covered
> before.
>
> About Ignite configurations, I'm going to add permutation for each
> IgniteConfiguration and CacheConfiguration property.
>
> By the way, the jira contains the following list of permutation (feel free
> to add something):
>
> The following tests should be added (for functional blocks):
>
>    1. Interceptor
>    2. Queries: continuous, scan, SQL, fields and text queries.
>    3. cache events
>    4. We should also test with Serializable, Externalizable, and plain
>    Pojos for keys and values.
>    5. The Pojo in the above test should contain an enum value
>    6. We should also test Enums as keys and Enums as values
>    7. All operations should have single-key and multi-key operations
>
> New tests should cover all combinations for following properties:
>
>    1. cache modes
>    2. operation from client nodes and server nodes
>    3. store enabled/disabled
>    4. evicts sycn/non-sync
>    5. eviction policies
>    6. near on/off
>    7. marshallers (+ Binary marshaller with different mappers)
>    8. keys and values - externalizable, serializable, binaryzable, "none of
>    previous"
>    9. classes available on servers: true/false
>    10. Peer loading on/off
>    11. Affinity functions
>    12. expiry policies
>
>
> Thanks,
> -- Artem --
>
> On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Artem, I think it is best to specify all the permutations here, so others
> > can make additional suggestions. Otherwise, we cannot get a full picture.
> >
> > Thanks,
> > D.
> >
> > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]>
> wrote:
> >
> > > Igniters,
> > >
> > > I thought a little bit more and think we need to add a support for the
> > > following permutations too (I've added these to the jira description):
> > > - We should also test with Serializable, Externalizable, and plain
> Pojos
> > > for keys and values.
> > > - The Pojo in the above test should contain an enum value
> > > - We should also test Enums as keys and Enums as values
> > > - All operations should have single-key and multi-key operations
> > >
> > > Maybe someone see any other permutation to be supported?
> > >
> > > -- Artem --
> > >
> > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]>
> > > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > There is a branch at my fork and a pull request at Ignite. See
> comment
> > > > about pull request at the ticket (PR-446).
> > > >
> > > > But I have to notice that the branch under hard development and you
> it
> > > can
> > > > not work (have compilation or test errors) at some moments.
> > > >
> > > > Good luck!
> > > >
> > > > -- Artem --
> > > >
> > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > >> Artem,
> > > >>
> > > >> This is great. I have noticed from the ticket that you have created
> > some
> > > >> initial suite already. Is there a branch I can look at it?
> > > >>
> > > >> D.
> > > >>
> > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]
> >
> > > >> wrote:
> > > >>
> > > >> > Igniters,
> > > >> >
> > > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > > >> >
> > > >> > Ignite already has Full API test, but currently it's hard to test
> > all
> > > >> > configuration combinations.
> > > >> >
> > > >> > Feel free to add comments in the jira if you have any thought.
> > > >> >
> > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > >> > [2]
> > > >>
> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > >> >
> > > >> > Thanks,
> > > >> > -- Artem --
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Semyon Boikov
In reply to this post by Artem Shutak
Artem,

One more thing for new tests: I think test should start both server and
client nodes and use Ignite API from all nodes.

On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]> wrote:

> Dmitriy,
>
> Actually, I don't have a list with all the permutations.
>
> At first, we need to split in our discussion test cases and Ignite
> configuration which should be covered.
>
> For example, new Full Api test cases for cache are based on old Full Api
> test cases. So, it need to think what the test cases was not covered
> before.
>
> About Ignite configurations, I'm going to add permutation for each
> IgniteConfiguration and CacheConfiguration property.
>
> By the way, the jira contains the following list of permutation (feel free
> to add something):
>
> The following tests should be added (for functional blocks):
>
>    1. Interceptor
>    2. Queries: continuous, scan, SQL, fields and text queries.
>    3. cache events
>    4. We should also test with Serializable, Externalizable, and plain
>    Pojos for keys and values.
>    5. The Pojo in the above test should contain an enum value
>    6. We should also test Enums as keys and Enums as values
>    7. All operations should have single-key and multi-key operations
>
> New tests should cover all combinations for following properties:
>
>    1. cache modes
>    2. operation from client nodes and server nodes
>    3. store enabled/disabled
>    4. evicts sycn/non-sync
>    5. eviction policies
>    6. near on/off
>    7. marshallers (+ Binary marshaller with different mappers)
>    8. keys and values - externalizable, serializable, binaryzable, "none of
>    previous"
>    9. classes available on servers: true/false
>    10. Peer loading on/off
>    11. Affinity functions
>    12. expiry policies
>
>
> Thanks,
> -- Artem --
>
> On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Artem, I think it is best to specify all the permutations here, so others
> > can make additional suggestions. Otherwise, we cannot get a full picture.
> >
> > Thanks,
> > D.
> >
> > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]>
> wrote:
> >
> > > Igniters,
> > >
> > > I thought a little bit more and think we need to add a support for the
> > > following permutations too (I've added these to the jira description):
> > > - We should also test with Serializable, Externalizable, and plain
> Pojos
> > > for keys and values.
> > > - The Pojo in the above test should contain an enum value
> > > - We should also test Enums as keys and Enums as values
> > > - All operations should have single-key and multi-key operations
> > >
> > > Maybe someone see any other permutation to be supported?
> > >
> > > -- Artem --
> > >
> > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]>
> > > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > There is a branch at my fork and a pull request at Ignite. See
> comment
> > > > about pull request at the ticket (PR-446).
> > > >
> > > > But I have to notice that the branch under hard development and you
> it
> > > can
> > > > not work (have compilation or test errors) at some moments.
> > > >
> > > > Good luck!
> > > >
> > > > -- Artem --
> > > >
> > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > >> Artem,
> > > >>
> > > >> This is great. I have noticed from the ticket that you have created
> > some
> > > >> initial suite already. Is there a branch I can look at it?
> > > >>
> > > >> D.
> > > >>
> > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]
> >
> > > >> wrote:
> > > >>
> > > >> > Igniters,
> > > >> >
> > > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > > >> >
> > > >> > Ignite already has Full API test, but currently it's hard to test
> > all
> > > >> > configuration combinations.
> > > >> >
> > > >> > Feel free to add comments in the jira if you have any thought.
> > > >> >
> > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > >> > [2]
> > > >>
> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > >> >
> > > >> > Thanks,
> > > >> > -- Artem --
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Artem Shutak
Hi all,

I have an update.

I've started from *CacheConfiguration* permutations. I wrote out a list
with all CacheConfiguration setters and filtered it with Alexey G.

Finally we have:

   1. CacheMode - 3 variants
   2. CacheAtomicityMode - 2 variants
   3. CacheMemoryMode - 3 variants
   4. setLoadPreviousValue - 2 variants
   5. setReadFromBackup - 2 variants
   6. setStoreKeepBinary - 2 variants
   7. setRebalanceMode - SYNC and ASYNC (2 variants)
   8. setSwapEnabled - 2 variants
   9. setCopyOnRead - 2 variants
   10. NearConfiguration disabled / default NearConfiguration / custom
   NearConfiguration - 3 variants
   11. With and without a complex parameter. The complex parameter defines
   not-default Eviction policy and filter, cache store configuration
   (storeFactory and storeSessionListenerFactory), rebalancing configuration,
   affinity function, offHeapMaxMemory, interceptor, topology validator and
   CacheEntryListener.

I've run 123 Cache Full Api test cases for all permutations of parameters
1-9 and got 256896 test cases (1152 configuration variants * 123 test
cases). All these tests take 4 hours 40 minutes. Not all tests pass, so MAY
BE when all tests will pass it will take less time (3,5 hours for example).

As we can see the tests take a lot of time.

The following permutation should be supported too:

   1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1
   backups, 4 nodes and 2 backups - 3 variants
   2. Client and Server nodes - 2 variants
   3. Indexing enabled and disabled for cache - 2 variants
   4. IgniteConfiguration permutations - how many variants? I see at least
   4 (2 Marshallers, P2P).

Plus we need add new test cases to test different key and value types and
etc.

So, we need multiply more then 3,5-4,5 hours on ~250. If we will split all
tests on 250 suites and run on all 30 TC agents it will take about 30-40
hours. Ok, we can do it during weekends.

I think it will take too much time.

As an option we can start a cache for each configuration and run tests
concurrently. But we need to implement this opportunity in our test
framework.

Any other thoughts how we can decrease time for tests?

Thanks,
-- Artem --

On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]> wrote:

> Artem,
>
> One more thing for new tests: I think test should start both server and
> client nodes and use Ignite API from all nodes.
>
> On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]> wrote:
>
> > Dmitriy,
> >
> > Actually, I don't have a list with all the permutations.
> >
> > At first, we need to split in our discussion test cases and Ignite
> > configuration which should be covered.
> >
> > For example, new Full Api test cases for cache are based on old Full Api
> > test cases. So, it need to think what the test cases was not covered
> > before.
> >
> > About Ignite configurations, I'm going to add permutation for each
> > IgniteConfiguration and CacheConfiguration property.
> >
> > By the way, the jira contains the following list of permutation (feel
> free
> > to add something):
> >
> > The following tests should be added (for functional blocks):
> >
> >    1. Interceptor
> >    2. Queries: continuous, scan, SQL, fields and text queries.
> >    3. cache events
> >    4. We should also test with Serializable, Externalizable, and plain
> >    Pojos for keys and values.
> >    5. The Pojo in the above test should contain an enum value
> >    6. We should also test Enums as keys and Enums as values
> >    7. All operations should have single-key and multi-key operations
> >
> > New tests should cover all combinations for following properties:
> >
> >    1. cache modes
> >    2. operation from client nodes and server nodes
> >    3. store enabled/disabled
> >    4. evicts sycn/non-sync
> >    5. eviction policies
> >    6. near on/off
> >    7. marshallers (+ Binary marshaller with different mappers)
> >    8. keys and values - externalizable, serializable, binaryzable, "none
> of
> >    previous"
> >    9. classes available on servers: true/false
> >    10. Peer loading on/off
> >    11. Affinity functions
> >    12. expiry policies
> >
> >
> > Thanks,
> > -- Artem --
> >
> > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> > > Artem, I think it is best to specify all the permutations here, so
> others
> > > can make additional suggestions. Otherwise, we cannot get a full
> picture.
> > >
> > > Thanks,
> > > D.
> > >
> > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]>
> > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I thought a little bit more and think we need to add a support for
> the
> > > > following permutations too (I've added these to the jira
> description):
> > > > - We should also test with Serializable, Externalizable, and plain
> > Pojos
> > > > for keys and values.
> > > > - The Pojo in the above test should contain an enum value
> > > > - We should also test Enums as keys and Enums as values
> > > > - All operations should have single-key and multi-key operations
> > > >
> > > > Maybe someone see any other permutation to be supported?
> > > >
> > > > -- Artem --
> > > >
> > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]>
> > > > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > There is a branch at my fork and a pull request at Ignite. See
> > comment
> > > > > about pull request at the ticket (PR-446).
> > > > >
> > > > > But I have to notice that the branch under hard development and you
> > it
> > > > can
> > > > > not work (have compilation or test errors) at some moments.
> > > > >
> > > > > Good luck!
> > > > >
> > > > > -- Artem --
> > > > >
> > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Artem,
> > > > >>
> > > > >> This is great. I have noticed from the ticket that you have
> created
> > > some
> > > > >> initial suite already. Is there a branch I can look at it?
> > > > >>
> > > > >> D.
> > > > >>
> > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> [hidden email]
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > Igniters,
> > > > >> >
> > > > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > > > >> >
> > > > >> > Ignite already has Full API test, but currently it's hard to
> test
> > > all
> > > > >> > configuration combinations.
> > > > >> >
> > > > >> > Feel free to add comments in the jira if you have any thought.
> > > > >> >
> > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > >> > [2]
> > > > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > >> >
> > > > >> > Thanks,
> > > > >> > -- Artem --
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Sergey Kozlov
Hi Artem

It's good idea to create 20-30 cache configurations at once and then to
iterate tests over those caches in parallel (but make sure that cache names
are unique).
Another point that some combinations make no sense like *backups *and
*REPLICATED
*cache


On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> wrote:

> Hi all,
>
> I have an update.
>
> I've started from *CacheConfiguration* permutations. I wrote out a list
> with all CacheConfiguration setters and filtered it with Alexey G.
>
> Finally we have:
>
>    1. CacheMode - 3 variants
>    2. CacheAtomicityMode - 2 variants
>    3. CacheMemoryMode - 3 variants
>    4. setLoadPreviousValue - 2 variants
>    5. setReadFromBackup - 2 variants
>    6. setStoreKeepBinary - 2 variants
>    7. setRebalanceMode - SYNC and ASYNC (2 variants)
>    8. setSwapEnabled - 2 variants
>    9. setCopyOnRead - 2 variants
>    10. NearConfiguration disabled / default NearConfiguration / custom
>    NearConfiguration - 3 variants
>    11. With and without a complex parameter. The complex parameter defines
>    not-default Eviction policy and filter, cache store configuration
>    (storeFactory and storeSessionListenerFactory), rebalancing
> configuration,
>    affinity function, offHeapMaxMemory, interceptor, topology validator and
>    CacheEntryListener.
>
> I've run 123 Cache Full Api test cases for all permutations of parameters
> 1-9 and got 256896 test cases (1152 configuration variants * 123 test
> cases). All these tests take 4 hours 40 minutes. Not all tests pass, so MAY
> BE when all tests will pass it will take less time (3,5 hours for example).
>
> As we can see the tests take a lot of time.
>
> The following permutation should be supported too:
>
>    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1
>    backups, 4 nodes and 2 backups - 3 variants
>    2. Client and Server nodes - 2 variants
>    3. Indexing enabled and disabled for cache - 2 variants
>    4. IgniteConfiguration permutations - how many variants? I see at least
>    4 (2 Marshallers, P2P).
>
> Plus we need add new test cases to test different key and value types and
> etc.
>
> So, we need multiply more then 3,5-4,5 hours on ~250. If we will split all
> tests on 250 suites and run on all 30 TC agents it will take about 30-40
> hours. Ok, we can do it during weekends.
>
> I think it will take too much time.
>
> As an option we can start a cache for each configuration and run tests
> concurrently. But we need to implement this opportunity in our test
> framework.
>
> Any other thoughts how we can decrease time for tests?
>
> Thanks,
> -- Artem --
>
> On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]>
> wrote:
>
> > Artem,
> >
> > One more thing for new tests: I think test should start both server and
> > client nodes and use Ignite API from all nodes.
> >
> > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]>
> wrote:
> >
> > > Dmitriy,
> > >
> > > Actually, I don't have a list with all the permutations.
> > >
> > > At first, we need to split in our discussion test cases and Ignite
> > > configuration which should be covered.
> > >
> > > For example, new Full Api test cases for cache are based on old Full
> Api
> > > test cases. So, it need to think what the test cases was not covered
> > > before.
> > >
> > > About Ignite configurations, I'm going to add permutation for each
> > > IgniteConfiguration and CacheConfiguration property.
> > >
> > > By the way, the jira contains the following list of permutation (feel
> > free
> > > to add something):
> > >
> > > The following tests should be added (for functional blocks):
> > >
> > >    1. Interceptor
> > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > >    3. cache events
> > >    4. We should also test with Serializable, Externalizable, and plain
> > >    Pojos for keys and values.
> > >    5. The Pojo in the above test should contain an enum value
> > >    6. We should also test Enums as keys and Enums as values
> > >    7. All operations should have single-key and multi-key operations
> > >
> > > New tests should cover all combinations for following properties:
> > >
> > >    1. cache modes
> > >    2. operation from client nodes and server nodes
> > >    3. store enabled/disabled
> > >    4. evicts sycn/non-sync
> > >    5. eviction policies
> > >    6. near on/off
> > >    7. marshallers (+ Binary marshaller with different mappers)
> > >    8. keys and values - externalizable, serializable, binaryzable,
> "none
> > of
> > >    previous"
> > >    9. classes available on servers: true/false
> > >    10. Peer loading on/off
> > >    11. Affinity functions
> > >    12. expiry policies
> > >
> > >
> > > Thanks,
> > > -- Artem --
> > >
> > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Artem, I think it is best to specify all the permutations here, so
> > others
> > > > can make additional suggestions. Otherwise, we cannot get a full
> > picture.
> > > >
> > > > Thanks,
> > > > D.
> > > >
> > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]>
> > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > I thought a little bit more and think we need to add a support for
> > the
> > > > > following permutations too (I've added these to the jira
> > description):
> > > > > - We should also test with Serializable, Externalizable, and plain
> > > Pojos
> > > > > for keys and values.
> > > > > - The Pojo in the above test should contain an enum value
> > > > > - We should also test Enums as keys and Enums as values
> > > > > - All operations should have single-key and multi-key operations
> > > > >
> > > > > Maybe someone see any other permutation to be supported?
> > > > >
> > > > > -- Artem --
> > > > >
> > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Dmitriy,
> > > > > >
> > > > > > There is a branch at my fork and a pull request at Ignite. See
> > > comment
> > > > > > about pull request at the ticket (PR-446).
> > > > > >
> > > > > > But I have to notice that the branch under hard development and
> you
> > > it
> > > > > can
> > > > > > not work (have compilation or test errors) at some moments.
> > > > > >
> > > > > > Good luck!
> > > > > >
> > > > > > -- Artem --
> > > > > >
> > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Artem,
> > > > > >>
> > > > > >> This is great. I have noticed from the ticket that you have
> > created
> > > > some
> > > > > >> initial suite already. Is there a branch I can look at it?
> > > > > >>
> > > > > >> D.
> > > > > >>
> > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> > [hidden email]
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Igniters,
> > > > > >> >
> > > > > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > > > > >> >
> > > > > >> > Ignite already has Full API test, but currently it's hard to
> > test
> > > > all
> > > > > >> > configuration combinations.
> > > > > >> >
> > > > > >> > Feel free to add comments in the jira if you have any thought.
> > > > > >> >
> > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > >> > [2]
> > > > > >>
> > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > -- Artem --
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



--
Sergey Kozlov
GridGain Systems
www.gridgain.com
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Artem Shutak
Sergey,

I think we should start more caches, like 1000 in one time. But we have to
have enough memory on our TC agents. As I know, empty cache is require
about 50 mb (without indexing), am I right?

You are right, I keep in mind that *backups* and *REPLICATED* mode make no
sense together, but we still have to test it in one node and multi node
casesю

Any other *no sense* combinations?

I forgot about custom BinaryConfiguration at IgniteConfiguration for
BinaryMarshaller. So, at least 6 IgniteConfigurations.

-- Artem --

On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]> wrote:

> Hi Artem
>
> It's good idea to create 20-30 cache configurations at once and then to
> iterate tests over those caches in parallel (but make sure that cache names
> are unique).
> Another point that some combinations make no sense like *backups *and
> *REPLICATED
> *cache
>
>
> On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> wrote:
>
> > Hi all,
> >
> > I have an update.
> >
> > I've started from *CacheConfiguration* permutations. I wrote out a list
> > with all CacheConfiguration setters and filtered it with Alexey G.
> >
> > Finally we have:
> >
> >    1. CacheMode - 3 variants
> >    2. CacheAtomicityMode - 2 variants
> >    3. CacheMemoryMode - 3 variants
> >    4. setLoadPreviousValue - 2 variants
> >    5. setReadFromBackup - 2 variants
> >    6. setStoreKeepBinary - 2 variants
> >    7. setRebalanceMode - SYNC and ASYNC (2 variants)
> >    8. setSwapEnabled - 2 variants
> >    9. setCopyOnRead - 2 variants
> >    10. NearConfiguration disabled / default NearConfiguration / custom
> >    NearConfiguration - 3 variants
> >    11. With and without a complex parameter. The complex parameter
> defines
> >    not-default Eviction policy and filter, cache store configuration
> >    (storeFactory and storeSessionListenerFactory), rebalancing
> > configuration,
> >    affinity function, offHeapMaxMemory, interceptor, topology validator
> and
> >    CacheEntryListener.
> >
> > I've run 123 Cache Full Api test cases for all permutations of parameters
> > 1-9 and got 256896 test cases (1152 configuration variants * 123 test
> > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so
> MAY
> > BE when all tests will pass it will take less time (3,5 hours for
> example).
> >
> > As we can see the tests take a lot of time.
> >
> > The following permutation should be supported too:
> >
> >    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1
> >    backups, 4 nodes and 2 backups - 3 variants
> >    2. Client and Server nodes - 2 variants
> >    3. Indexing enabled and disabled for cache - 2 variants
> >    4. IgniteConfiguration permutations - how many variants? I see at
> least
> >    4 (2 Marshallers, P2P).
> >
> > Plus we need add new test cases to test different key and value types and
> > etc.
> >
> > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split
> all
> > tests on 250 suites and run on all 30 TC agents it will take about 30-40
> > hours. Ok, we can do it during weekends.
> >
> > I think it will take too much time.
> >
> > As an option we can start a cache for each configuration and run tests
> > concurrently. But we need to implement this opportunity in our test
> > framework.
> >
> > Any other thoughts how we can decrease time for tests?
> >
> > Thanks,
> > -- Artem --
> >
> > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]>
> > wrote:
> >
> > > Artem,
> > >
> > > One more thing for new tests: I think test should start both server and
> > > client nodes and use Ignite API from all nodes.
> > >
> > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]>
> > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > Actually, I don't have a list with all the permutations.
> > > >
> > > > At first, we need to split in our discussion test cases and Ignite
> > > > configuration which should be covered.
> > > >
> > > > For example, new Full Api test cases for cache are based on old Full
> > Api
> > > > test cases. So, it need to think what the test cases was not covered
> > > > before.
> > > >
> > > > About Ignite configurations, I'm going to add permutation for each
> > > > IgniteConfiguration and CacheConfiguration property.
> > > >
> > > > By the way, the jira contains the following list of permutation (feel
> > > free
> > > > to add something):
> > > >
> > > > The following tests should be added (for functional blocks):
> > > >
> > > >    1. Interceptor
> > > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > > >    3. cache events
> > > >    4. We should also test with Serializable, Externalizable, and
> plain
> > > >    Pojos for keys and values.
> > > >    5. The Pojo in the above test should contain an enum value
> > > >    6. We should also test Enums as keys and Enums as values
> > > >    7. All operations should have single-key and multi-key operations
> > > >
> > > > New tests should cover all combinations for following properties:
> > > >
> > > >    1. cache modes
> > > >    2. operation from client nodes and server nodes
> > > >    3. store enabled/disabled
> > > >    4. evicts sycn/non-sync
> > > >    5. eviction policies
> > > >    6. near on/off
> > > >    7. marshallers (+ Binary marshaller with different mappers)
> > > >    8. keys and values - externalizable, serializable, binaryzable,
> > "none
> > > of
> > > >    previous"
> > > >    9. classes available on servers: true/false
> > > >    10. Peer loading on/off
> > > >    11. Affinity functions
> > > >    12. expiry policies
> > > >
> > > >
> > > > Thanks,
> > > > -- Artem --
> > > >
> > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Artem, I think it is best to specify all the permutations here, so
> > > others
> > > > > can make additional suggestions. Otherwise, we cannot get a full
> > > picture.
> > > > >
> > > > > Thanks,
> > > > > D.
> > > > >
> > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]
> >
> > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > I thought a little bit more and think we need to add a support
> for
> > > the
> > > > > > following permutations too (I've added these to the jira
> > > description):
> > > > > > - We should also test with Serializable, Externalizable, and
> plain
> > > > Pojos
> > > > > > for keys and values.
> > > > > > - The Pojo in the above test should contain an enum value
> > > > > > - We should also test Enums as keys and Enums as values
> > > > > > - All operations should have single-key and multi-key operations
> > > > > >
> > > > > > Maybe someone see any other permutation to be supported?
> > > > > >
> > > > > > -- Artem --
> > > > > >
> > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <
> > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Dmitriy,
> > > > > > >
> > > > > > > There is a branch at my fork and a pull request at Ignite. See
> > > > comment
> > > > > > > about pull request at the ticket (PR-446).
> > > > > > >
> > > > > > > But I have to notice that the branch under hard development and
> > you
> > > > it
> > > > > > can
> > > > > > > not work (have compilation or test errors) at some moments.
> > > > > > >
> > > > > > > Good luck!
> > > > > > >
> > > > > > > -- Artem --
> > > > > > >
> > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > > > [hidden email]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Artem,
> > > > > > >>
> > > > > > >> This is great. I have noticed from the ticket that you have
> > > created
> > > > > some
> > > > > > >> initial suite already. Is there a branch I can look at it?
> > > > > > >>
> > > > > > >> D.
> > > > > > >>
> > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> > > [hidden email]
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Igniters,
> > > > > > >> >
> > > > > > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > > > > > >> >
> > > > > > >> > Ignite already has Full API test, but currently it's hard to
> > > test
> > > > > all
> > > > > > >> > configuration combinations.
> > > > > > >> >
> > > > > > >> > Feel free to add comments in the jira if you have any
> thought.
> > > > > > >> >
> > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > > >> > [2]
> > > > > > >>
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > -- Artem --
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Sergey Kozlov
1000 caches x 50MB = 50GB heap. Do we really have >50GB RAM on each agents?

On Mon, Feb 8, 2016 at 5:45 PM, Artem Shutak <[hidden email]> wrote:

> Sergey,
>
> I think we should start more caches, like 1000 in one time. But we have to
> have enough memory on our TC agents. As I know, empty cache is require
> about 50 mb (without indexing), am I right?
>
> You are right, I keep in mind that *backups* and *REPLICATED* mode make no
> sense together, but we still have to test it in one node and multi node
> casesю
>
> Any other *no sense* combinations?
>
> I forgot about custom BinaryConfiguration at IgniteConfiguration for
> BinaryMarshaller. So, at least 6 IgniteConfigurations.
>
> -- Artem --
>
> On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]>
> wrote:
>
> > Hi Artem
> >
> > It's good idea to create 20-30 cache configurations at once and then to
> > iterate tests over those caches in parallel (but make sure that cache
> names
> > are unique).
> > Another point that some combinations make no sense like *backups *and
> > *REPLICATED
> > *cache
> >
> >
> > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]>
> wrote:
> >
> > > Hi all,
> > >
> > > I have an update.
> > >
> > > I've started from *CacheConfiguration* permutations. I wrote out a list
> > > with all CacheConfiguration setters and filtered it with Alexey G.
> > >
> > > Finally we have:
> > >
> > >    1. CacheMode - 3 variants
> > >    2. CacheAtomicityMode - 2 variants
> > >    3. CacheMemoryMode - 3 variants
> > >    4. setLoadPreviousValue - 2 variants
> > >    5. setReadFromBackup - 2 variants
> > >    6. setStoreKeepBinary - 2 variants
> > >    7. setRebalanceMode - SYNC and ASYNC (2 variants)
> > >    8. setSwapEnabled - 2 variants
> > >    9. setCopyOnRead - 2 variants
> > >    10. NearConfiguration disabled / default NearConfiguration / custom
> > >    NearConfiguration - 3 variants
> > >    11. With and without a complex parameter. The complex parameter
> > defines
> > >    not-default Eviction policy and filter, cache store configuration
> > >    (storeFactory and storeSessionListenerFactory), rebalancing
> > > configuration,
> > >    affinity function, offHeapMaxMemory, interceptor, topology validator
> > and
> > >    CacheEntryListener.
> > >
> > > I've run 123 Cache Full Api test cases for all permutations of
> parameters
> > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test
> > > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so
> > MAY
> > > BE when all tests will pass it will take less time (3,5 hours for
> > example).
> > >
> > > As we can see the tests take a lot of time.
> > >
> > > The following permutation should be supported too:
> > >
> > >    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1
> > >    backups, 4 nodes and 2 backups - 3 variants
> > >    2. Client and Server nodes - 2 variants
> > >    3. Indexing enabled and disabled for cache - 2 variants
> > >    4. IgniteConfiguration permutations - how many variants? I see at
> > least
> > >    4 (2 Marshallers, P2P).
> > >
> > > Plus we need add new test cases to test different key and value types
> and
> > > etc.
> > >
> > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split
> > all
> > > tests on 250 suites and run on all 30 TC agents it will take about
> 30-40
> > > hours. Ok, we can do it during weekends.
> > >
> > > I think it will take too much time.
> > >
> > > As an option we can start a cache for each configuration and run tests
> > > concurrently. But we need to implement this opportunity in our test
> > > framework.
> > >
> > > Any other thoughts how we can decrease time for tests?
> > >
> > > Thanks,
> > > -- Artem --
> > >
> > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]>
> > > wrote:
> > >
> > > > Artem,
> > > >
> > > > One more thing for new tests: I think test should start both server
> and
> > > > client nodes and use Ignite API from all nodes.
> > > >
> > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]>
> > > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > Actually, I don't have a list with all the permutations.
> > > > >
> > > > > At first, we need to split in our discussion test cases and Ignite
> > > > > configuration which should be covered.
> > > > >
> > > > > For example, new Full Api test cases for cache are based on old
> Full
> > > Api
> > > > > test cases. So, it need to think what the test cases was not
> covered
> > > > > before.
> > > > >
> > > > > About Ignite configurations, I'm going to add permutation for each
> > > > > IgniteConfiguration and CacheConfiguration property.
> > > > >
> > > > > By the way, the jira contains the following list of permutation
> (feel
> > > > free
> > > > > to add something):
> > > > >
> > > > > The following tests should be added (for functional blocks):
> > > > >
> > > > >    1. Interceptor
> > > > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > > > >    3. cache events
> > > > >    4. We should also test with Serializable, Externalizable, and
> > plain
> > > > >    Pojos for keys and values.
> > > > >    5. The Pojo in the above test should contain an enum value
> > > > >    6. We should also test Enums as keys and Enums as values
> > > > >    7. All operations should have single-key and multi-key
> operations
> > > > >
> > > > > New tests should cover all combinations for following properties:
> > > > >
> > > > >    1. cache modes
> > > > >    2. operation from client nodes and server nodes
> > > > >    3. store enabled/disabled
> > > > >    4. evicts sycn/non-sync
> > > > >    5. eviction policies
> > > > >    6. near on/off
> > > > >    7. marshallers (+ Binary marshaller with different mappers)
> > > > >    8. keys and values - externalizable, serializable, binaryzable,
> > > "none
> > > > of
> > > > >    previous"
> > > > >    9. classes available on servers: true/false
> > > > >    10. Peer loading on/off
> > > > >    11. Affinity functions
> > > > >    12. expiry policies
> > > > >
> > > > >
> > > > > Thanks,
> > > > > -- Artem --
> > > > >
> > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Artem, I think it is best to specify all the permutations here,
> so
> > > > others
> > > > > > can make additional suggestions. Otherwise, we cannot get a full
> > > > picture.
> > > > > >
> > > > > > Thanks,
> > > > > > D.
> > > > > >
> > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <
> [hidden email]
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > I thought a little bit more and think we need to add a support
> > for
> > > > the
> > > > > > > following permutations too (I've added these to the jira
> > > > description):
> > > > > > > - We should also test with Serializable, Externalizable, and
> > plain
> > > > > Pojos
> > > > > > > for keys and values.
> > > > > > > - The Pojo in the above test should contain an enum value
> > > > > > > - We should also test Enums as keys and Enums as values
> > > > > > > - All operations should have single-key and multi-key
> operations
> > > > > > >
> > > > > > > Maybe someone see any other permutation to be supported?
> > > > > > >
> > > > > > > -- Artem --
> > > > > > >
> > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Dmitriy,
> > > > > > > >
> > > > > > > > There is a branch at my fork and a pull request at Ignite.
> See
> > > > > comment
> > > > > > > > about pull request at the ticket (PR-446).
> > > > > > > >
> > > > > > > > But I have to notice that the branch under hard development
> and
> > > you
> > > > > it
> > > > > > > can
> > > > > > > > not work (have compilation or test errors) at some moments.
> > > > > > > >
> > > > > > > > Good luck!
> > > > > > > >
> > > > > > > > -- Artem --
> > > > > > > >
> > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Artem,
> > > > > > > >>
> > > > > > > >> This is great. I have noticed from the ticket that you have
> > > > created
> > > > > > some
> > > > > > > >> initial suite already. Is there a branch I can look at it?
> > > > > > > >>
> > > > > > > >> D.
> > > > > > > >>
> > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> > > > [hidden email]
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Igniters,
> > > > > > > >> >
> > > > > > > >> > I'm working on an enhancement of Full API coverage [1]
> [2].
> > > > > > > >> >
> > > > > > > >> > Ignite already has Full API test, but currently it's hard
> to
> > > > test
> > > > > > all
> > > > > > > >> > configuration combinations.
> > > > > > > >> >
> > > > > > > >> > Feel free to add comments in the jira if you have any
> > thought.
> > > > > > > >> >
> > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > > > >> > [2]
> > > > > > > >>
> > > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > -- Artem --
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>



--
Sergey Kozlov
GridGain Systems
www.gridgain.com
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Alexey Kuznetsov-2
How about such approach:

Generate all permutation descriptors and save them to database.
On each agent start some process that will grab from DB some descriptors
(lets say 100) mark them as "in progress" and execute them.
Other agents will grab remaining "not in progress" descriptors and do the
same.
So we could run in parallel all permutations even if we have 1M of them.

What do you think?


On Mon, Feb 8, 2016 at 9:52 PM, Sergey Kozlov <[hidden email]> wrote:

> 1000 caches x 50MB = 50GB heap. Do we really have >50GB RAM on each agents?
>
> On Mon, Feb 8, 2016 at 5:45 PM, Artem Shutak <[hidden email]> wrote:
>
> > Sergey,
> >
> > I think we should start more caches, like 1000 in one time. But we have
> to
> > have enough memory on our TC agents. As I know, empty cache is require
> > about 50 mb (without indexing), am I right?
> >
> > You are right, I keep in mind that *backups* and *REPLICATED* mode make
> no
> > sense together, but we still have to test it in one node and multi node
> > casesю
> >
> > Any other *no sense* combinations?
> >
> > I forgot about custom BinaryConfiguration at IgniteConfiguration for
> > BinaryMarshaller. So, at least 6 IgniteConfigurations.
> >
> > -- Artem --
> >
> > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]>
> > wrote:
> >
> > > Hi Artem
> > >
> > > It's good idea to create 20-30 cache configurations at once and then to
> > > iterate tests over those caches in parallel (but make sure that cache
> > names
> > > are unique).
> > > Another point that some combinations make no sense like *backups *and
> > > *REPLICATED
> > > *cache
> > >
> > >
> > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have an update.
> > > >
> > > > I've started from *CacheConfiguration* permutations. I wrote out a
> list
> > > > with all CacheConfiguration setters and filtered it with Alexey G.
> > > >
> > > > Finally we have:
> > > >
> > > >    1. CacheMode - 3 variants
> > > >    2. CacheAtomicityMode - 2 variants
> > > >    3. CacheMemoryMode - 3 variants
> > > >    4. setLoadPreviousValue - 2 variants
> > > >    5. setReadFromBackup - 2 variants
> > > >    6. setStoreKeepBinary - 2 variants
> > > >    7. setRebalanceMode - SYNC and ASYNC (2 variants)
> > > >    8. setSwapEnabled - 2 variants
> > > >    9. setCopyOnRead - 2 variants
> > > >    10. NearConfiguration disabled / default NearConfiguration /
> custom
> > > >    NearConfiguration - 3 variants
> > > >    11. With and without a complex parameter. The complex parameter
> > > defines
> > > >    not-default Eviction policy and filter, cache store configuration
> > > >    (storeFactory and storeSessionListenerFactory), rebalancing
> > > > configuration,
> > > >    affinity function, offHeapMaxMemory, interceptor, topology
> validator
> > > and
> > > >    CacheEntryListener.
> > > >
> > > > I've run 123 Cache Full Api test cases for all permutations of
> > parameters
> > > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test
> > > > cases). All these tests take 4 hours 40 minutes. Not all tests pass,
> so
> > > MAY
> > > > BE when all tests will pass it will take less time (3,5 hours for
> > > example).
> > > >
> > > > As we can see the tests take a lot of time.
> > > >
> > > > The following permutation should be supported too:
> > > >
> > > >    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes
> and 1
> > > >    backups, 4 nodes and 2 backups - 3 variants
> > > >    2. Client and Server nodes - 2 variants
> > > >    3. Indexing enabled and disabled for cache - 2 variants
> > > >    4. IgniteConfiguration permutations - how many variants? I see at
> > > least
> > > >    4 (2 Marshallers, P2P).
> > > >
> > > > Plus we need add new test cases to test different key and value types
> > and
> > > > etc.
> > > >
> > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will
> split
> > > all
> > > > tests on 250 suites and run on all 30 TC agents it will take about
> > 30-40
> > > > hours. Ok, we can do it during weekends.
> > > >
> > > > I think it will take too much time.
> > > >
> > > > As an option we can start a cache for each configuration and run
> tests
> > > > concurrently. But we need to implement this opportunity in our test
> > > > framework.
> > > >
> > > > Any other thoughts how we can decrease time for tests?
> > > >
> > > > Thanks,
> > > > -- Artem --
> > > >
> > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]>
> > > > wrote:
> > > >
> > > > > Artem,
> > > > >
> > > > > One more thing for new tests: I think test should start both server
> > and
> > > > > client nodes and use Ignite API from all nodes.
> > > > >
> > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]
> >
> > > > wrote:
> > > > >
> > > > > > Dmitriy,
> > > > > >
> > > > > > Actually, I don't have a list with all the permutations.
> > > > > >
> > > > > > At first, we need to split in our discussion test cases and
> Ignite
> > > > > > configuration which should be covered.
> > > > > >
> > > > > > For example, new Full Api test cases for cache are based on old
> > Full
> > > > Api
> > > > > > test cases. So, it need to think what the test cases was not
> > covered
> > > > > > before.
> > > > > >
> > > > > > About Ignite configurations, I'm going to add permutation for
> each
> > > > > > IgniteConfiguration and CacheConfiguration property.
> > > > > >
> > > > > > By the way, the jira contains the following list of permutation
> > (feel
> > > > > free
> > > > > > to add something):
> > > > > >
> > > > > > The following tests should be added (for functional blocks):
> > > > > >
> > > > > >    1. Interceptor
> > > > > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > > > > >    3. cache events
> > > > > >    4. We should also test with Serializable, Externalizable, and
> > > plain
> > > > > >    Pojos for keys and values.
> > > > > >    5. The Pojo in the above test should contain an enum value
> > > > > >    6. We should also test Enums as keys and Enums as values
> > > > > >    7. All operations should have single-key and multi-key
> > operations
> > > > > >
> > > > > > New tests should cover all combinations for following properties:
> > > > > >
> > > > > >    1. cache modes
> > > > > >    2. operation from client nodes and server nodes
> > > > > >    3. store enabled/disabled
> > > > > >    4. evicts sycn/non-sync
> > > > > >    5. eviction policies
> > > > > >    6. near on/off
> > > > > >    7. marshallers (+ Binary marshaller with different mappers)
> > > > > >    8. keys and values - externalizable, serializable,
> binaryzable,
> > > > "none
> > > > > of
> > > > > >    previous"
> > > > > >    9. classes available on servers: true/false
> > > > > >    10. Peer loading on/off
> > > > > >    11. Affinity functions
> > > > > >    12. expiry policies
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > -- Artem --
> > > > > >
> > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Artem, I think it is best to specify all the permutations here,
> > so
> > > > > others
> > > > > > > can make additional suggestions. Otherwise, we cannot get a
> full
> > > > > picture.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > D.
> > > > > > >
> > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > I thought a little bit more and think we need to add a
> support
> > > for
> > > > > the
> > > > > > > > following permutations too (I've added these to the jira
> > > > > description):
> > > > > > > > - We should also test with Serializable, Externalizable, and
> > > plain
> > > > > > Pojos
> > > > > > > > for keys and values.
> > > > > > > > - The Pojo in the above test should contain an enum value
> > > > > > > > - We should also test Enums as keys and Enums as values
> > > > > > > > - All operations should have single-key and multi-key
> > operations
> > > > > > > >
> > > > > > > > Maybe someone see any other permutation to be supported?
> > > > > > > >
> > > > > > > > -- Artem --
> > > > > > > >
> > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Dmitriy,
> > > > > > > > >
> > > > > > > > > There is a branch at my fork and a pull request at Ignite.
> > See
> > > > > > comment
> > > > > > > > > about pull request at the ticket (PR-446).
> > > > > > > > >
> > > > > > > > > But I have to notice that the branch under hard development
> > and
> > > > you
> > > > > > it
> > > > > > > > can
> > > > > > > > > not work (have compilation or test errors) at some moments.
> > > > > > > > >
> > > > > > > > > Good luck!
> > > > > > > > >
> > > > > > > > > -- Artem --
> > > > > > > > >
> > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Artem,
> > > > > > > > >>
> > > > > > > > >> This is great. I have noticed from the ticket that you
> have
> > > > > created
> > > > > > > some
> > > > > > > > >> initial suite already. Is there a branch I can look at it?
> > > > > > > > >>
> > > > > > > > >> D.
> > > > > > > > >>
> > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Igniters,
> > > > > > > > >> >
> > > > > > > > >> > I'm working on an enhancement of Full API coverage [1]
> > [2].
> > > > > > > > >> >
> > > > > > > > >> > Ignite already has Full API test, but currently it's
> hard
> > to
> > > > > test
> > > > > > > all
> > > > > > > > >> > configuration combinations.
> > > > > > > > >> >
> > > > > > > > >> > Feel free to add comments in the jira if you have any
> > > thought.
> > > > > > > > >> >
> > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > > > > >> > [2]
> > > > > > > > >>
> > > > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > -- Artem --
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sergey Kozlov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>



--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Artem Shutak
Alexey,

First of all, it's strange to have disk-dependent tests for in-memory data
fabric ;) I'm sure we can solve this issue without bottleneck. I see many
technical issues with that approach.

Another bad thing about the approach is we don't know how to find test
result by configuration.

I think, if we will split tests it should be done manually: by marshaller,
enable/disable indexing, enabled/disabled near configuration and etc. It
will be really simple when I will finish with the test framework.

In my view, it's not really hard to run tests concurrently. My question is
in approach. May be we can another options or we cannot do it at all by any
reason.

-- Artem --

On Mon, Feb 8, 2016 at 6:38 PM, Alexey Kuznetsov <[hidden email]>
wrote:

> How about such approach:
>
> Generate all permutation descriptors and save them to database.
> On each agent start some process that will grab from DB some descriptors
> (lets say 100) mark them as "in progress" and execute them.
> Other agents will grab remaining "not in progress" descriptors and do the
> same.
> So we could run in parallel all permutations even if we have 1M of them.
>
> What do you think?
>
>
> On Mon, Feb 8, 2016 at 9:52 PM, Sergey Kozlov <[hidden email]>
> wrote:
>
> > 1000 caches x 50MB = 50GB heap. Do we really have >50GB RAM on each
> agents?
> >
> > On Mon, Feb 8, 2016 at 5:45 PM, Artem Shutak <[hidden email]>
> wrote:
> >
> > > Sergey,
> > >
> > > I think we should start more caches, like 1000 in one time. But we have
> > to
> > > have enough memory on our TC agents. As I know, empty cache is require
> > > about 50 mb (without indexing), am I right?
> > >
> > > You are right, I keep in mind that *backups* and *REPLICATED* mode make
> > no
> > > sense together, but we still have to test it in one node and multi node
> > > casesю
> > >
> > > Any other *no sense* combinations?
> > >
> > > I forgot about custom BinaryConfiguration at IgniteConfiguration for
> > > BinaryMarshaller. So, at least 6 IgniteConfigurations.
> > >
> > > -- Artem --
> > >
> > > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]>
> > > wrote:
> > >
> > > > Hi Artem
> > > >
> > > > It's good idea to create 20-30 cache configurations at once and then
> to
> > > > iterate tests over those caches in parallel (but make sure that cache
> > > names
> > > > are unique).
> > > > Another point that some combinations make no sense like *backups *and
> > > > *REPLICATED
> > > > *cache
> > > >
> > > >
> > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have an update.
> > > > >
> > > > > I've started from *CacheConfiguration* permutations. I wrote out a
> > list
> > > > > with all CacheConfiguration setters and filtered it with Alexey G.
> > > > >
> > > > > Finally we have:
> > > > >
> > > > >    1. CacheMode - 3 variants
> > > > >    2. CacheAtomicityMode - 2 variants
> > > > >    3. CacheMemoryMode - 3 variants
> > > > >    4. setLoadPreviousValue - 2 variants
> > > > >    5. setReadFromBackup - 2 variants
> > > > >    6. setStoreKeepBinary - 2 variants
> > > > >    7. setRebalanceMode - SYNC and ASYNC (2 variants)
> > > > >    8. setSwapEnabled - 2 variants
> > > > >    9. setCopyOnRead - 2 variants
> > > > >    10. NearConfiguration disabled / default NearConfiguration /
> > custom
> > > > >    NearConfiguration - 3 variants
> > > > >    11. With and without a complex parameter. The complex parameter
> > > > defines
> > > > >    not-default Eviction policy and filter, cache store
> configuration
> > > > >    (storeFactory and storeSessionListenerFactory), rebalancing
> > > > > configuration,
> > > > >    affinity function, offHeapMaxMemory, interceptor, topology
> > validator
> > > > and
> > > > >    CacheEntryListener.
> > > > >
> > > > > I've run 123 Cache Full Api test cases for all permutations of
> > > parameters
> > > > > 1-9 and got 256896 test cases (1152 configuration variants * 123
> test
> > > > > cases). All these tests take 4 hours 40 minutes. Not all tests
> pass,
> > so
> > > > MAY
> > > > > BE when all tests will pass it will take less time (3,5 hours for
> > > > example).
> > > > >
> > > > > As we can see the tests take a lot of time.
> > > > >
> > > > > The following permutation should be supported too:
> > > > >
> > > > >    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes
> > and 1
> > > > >    backups, 4 nodes and 2 backups - 3 variants
> > > > >    2. Client and Server nodes - 2 variants
> > > > >    3. Indexing enabled and disabled for cache - 2 variants
> > > > >    4. IgniteConfiguration permutations - how many variants? I see
> at
> > > > least
> > > > >    4 (2 Marshallers, P2P).
> > > > >
> > > > > Plus we need add new test cases to test different key and value
> types
> > > and
> > > > > etc.
> > > > >
> > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will
> > split
> > > > all
> > > > > tests on 250 suites and run on all 30 TC agents it will take about
> > > 30-40
> > > > > hours. Ok, we can do it during weekends.
> > > > >
> > > > > I think it will take too much time.
> > > > >
> > > > > As an option we can start a cache for each configuration and run
> > tests
> > > > > concurrently. But we need to implement this opportunity in our test
> > > > > framework.
> > > > >
> > > > > Any other thoughts how we can decrease time for tests?
> > > > >
> > > > > Thanks,
> > > > > -- Artem --
> > > > >
> > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Artem,
> > > > > >
> > > > > > One more thing for new tests: I think test should start both
> server
> > > and
> > > > > > client nodes and use Ignite API from all nodes.
> > > > > >
> > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <
> [hidden email]
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Dmitriy,
> > > > > > >
> > > > > > > Actually, I don't have a list with all the permutations.
> > > > > > >
> > > > > > > At first, we need to split in our discussion test cases and
> > Ignite
> > > > > > > configuration which should be covered.
> > > > > > >
> > > > > > > For example, new Full Api test cases for cache are based on old
> > > Full
> > > > > Api
> > > > > > > test cases. So, it need to think what the test cases was not
> > > covered
> > > > > > > before.
> > > > > > >
> > > > > > > About Ignite configurations, I'm going to add permutation for
> > each
> > > > > > > IgniteConfiguration and CacheConfiguration property.
> > > > > > >
> > > > > > > By the way, the jira contains the following list of permutation
> > > (feel
> > > > > > free
> > > > > > > to add something):
> > > > > > >
> > > > > > > The following tests should be added (for functional blocks):
> > > > > > >
> > > > > > >    1. Interceptor
> > > > > > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > > > > > >    3. cache events
> > > > > > >    4. We should also test with Serializable, Externalizable,
> and
> > > > plain
> > > > > > >    Pojos for keys and values.
> > > > > > >    5. The Pojo in the above test should contain an enum value
> > > > > > >    6. We should also test Enums as keys and Enums as values
> > > > > > >    7. All operations should have single-key and multi-key
> > > operations
> > > > > > >
> > > > > > > New tests should cover all combinations for following
> properties:
> > > > > > >
> > > > > > >    1. cache modes
> > > > > > >    2. operation from client nodes and server nodes
> > > > > > >    3. store enabled/disabled
> > > > > > >    4. evicts sycn/non-sync
> > > > > > >    5. eviction policies
> > > > > > >    6. near on/off
> > > > > > >    7. marshallers (+ Binary marshaller with different mappers)
> > > > > > >    8. keys and values - externalizable, serializable,
> > binaryzable,
> > > > > "none
> > > > > > of
> > > > > > >    previous"
> > > > > > >    9. classes available on servers: true/false
> > > > > > >    10. Peer loading on/off
> > > > > > >    11. Affinity functions
> > > > > > >    12. expiry policies
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -- Artem --
> > > > > > >
> > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> > > > > [hidden email]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Artem, I think it is best to specify all the permutations
> here,
> > > so
> > > > > > others
> > > > > > > > can make additional suggestions. Otherwise, we cannot get a
> > full
> > > > > > picture.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Igniters,
> > > > > > > > >
> > > > > > > > > I thought a little bit more and think we need to add a
> > support
> > > > for
> > > > > > the
> > > > > > > > > following permutations too (I've added these to the jira
> > > > > > description):
> > > > > > > > > - We should also test with Serializable, Externalizable,
> and
> > > > plain
> > > > > > > Pojos
> > > > > > > > > for keys and values.
> > > > > > > > > - The Pojo in the above test should contain an enum value
> > > > > > > > > - We should also test Enums as keys and Enums as values
> > > > > > > > > - All operations should have single-key and multi-key
> > > operations
> > > > > > > > >
> > > > > > > > > Maybe someone see any other permutation to be supported?
> > > > > > > > >
> > > > > > > > > -- Artem --
> > > > > > > > >
> > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Dmitriy,
> > > > > > > > > >
> > > > > > > > > > There is a branch at my fork and a pull request at
> Ignite.
> > > See
> > > > > > > comment
> > > > > > > > > > about pull request at the ticket (PR-446).
> > > > > > > > > >
> > > > > > > > > > But I have to notice that the branch under hard
> development
> > > and
> > > > > you
> > > > > > > it
> > > > > > > > > can
> > > > > > > > > > not work (have compilation or test errors) at some
> moments.
> > > > > > > > > >
> > > > > > > > > > Good luck!
> > > > > > > > > >
> > > > > > > > > > -- Artem --
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Artem,
> > > > > > > > > >>
> > > > > > > > > >> This is great. I have noticed from the ticket that you
> > have
> > > > > > created
> > > > > > > > some
> > > > > > > > > >> initial suite already. Is there a branch I can look at
> it?
> > > > > > > > > >>
> > > > > > > > > >> D.
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Igniters,
> > > > > > > > > >> >
> > > > > > > > > >> > I'm working on an enhancement of Full API coverage [1]
> > > [2].
> > > > > > > > > >> >
> > > > > > > > > >> > Ignite already has Full API test, but currently it's
> > hard
> > > to
> > > > > > test
> > > > > > > > all
> > > > > > > > > >> > configuration combinations.
> > > > > > > > > >> >
> > > > > > > > > >> > Feel free to add comments in the jira if you have any
> > > > thought.
> > > > > > > > > >> >
> > > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > > > > > >> > [2]
> > > > > > > > > >>
> > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > -- Artem --
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sergey Kozlov
> > > > GridGain Systems
> > > > www.gridgain.com
> > > >
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>
Reply | Threaded
Open this post in threaded view
|

RE: Full API coverage enhancement

Andrey Kornev
In reply to this post by Artem Shutak
50MB for an empty cache! That's a lot of emptiness!

The Ignite team's recommendation to have one key/value type per cache now seems a little more dubious and difficult to follow in practice.

Just curious, what is the reason for such waste? Is it because by default the initial cache size is set to 1.5M entries?

Thanks
Andrey

> Date: Mon, 8 Feb 2016 17:45:33 +0300
> Subject: Re: Full API coverage enhancement
> From: [hidden email]
> To: [hidden email]
>
> Sergey,
>
> I think we should start more caches, like 1000 in one time. But we have to
> have enough memory on our TC agents. As I know, empty cache is require
> about 50 mb (without indexing), am I right?
>
> You are right, I keep in mind that *backups* and *REPLICATED* mode make no
> sense together, but we still have to test it in one node and multi node
> casesю
>
> Any other *no sense* combinations?
>
> I forgot about custom BinaryConfiguration at IgniteConfiguration for
> BinaryMarshaller. So, at least 6 IgniteConfigurations.
>
> -- Artem --
>
> On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]> wrote:
>
> > Hi Artem
> >
> > It's good idea to create 20-30 cache configurations at once and then to
> > iterate tests over those caches in parallel (but make sure that cache names
> > are unique).
> > Another point that some combinations make no sense like *backups *and
> > *REPLICATED
> > *cache
> >
> >
> > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> wrote:
> >
> > > Hi all,
> > >
> > > I have an update.
> > >
> > > I've started from *CacheConfiguration* permutations. I wrote out a list
> > > with all CacheConfiguration setters and filtered it with Alexey G.
> > >
> > > Finally we have:
> > >
> > >    1. CacheMode - 3 variants
> > >    2. CacheAtomicityMode - 2 variants
> > >    3. CacheMemoryMode - 3 variants
> > >    4. setLoadPreviousValue - 2 variants
> > >    5. setReadFromBackup - 2 variants
> > >    6. setStoreKeepBinary - 2 variants
> > >    7. setRebalanceMode - SYNC and ASYNC (2 variants)
> > >    8. setSwapEnabled - 2 variants
> > >    9. setCopyOnRead - 2 variants
> > >    10. NearConfiguration disabled / default NearConfiguration / custom
> > >    NearConfiguration - 3 variants
> > >    11. With and without a complex parameter. The complex parameter
> > defines
> > >    not-default Eviction policy and filter, cache store configuration
> > >    (storeFactory and storeSessionListenerFactory), rebalancing
> > > configuration,
> > >    affinity function, offHeapMaxMemory, interceptor, topology validator
> > and
> > >    CacheEntryListener.
> > >
> > > I've run 123 Cache Full Api test cases for all permutations of parameters
> > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test
> > > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so
> > MAY
> > > BE when all tests will pass it will take less time (3,5 hours for
> > example).
> > >
> > > As we can see the tests take a lot of time.
> > >
> > > The following permutation should be supported too:
> > >
> > >    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1
> > >    backups, 4 nodes and 2 backups - 3 variants
> > >    2. Client and Server nodes - 2 variants
> > >    3. Indexing enabled and disabled for cache - 2 variants
> > >    4. IgniteConfiguration permutations - how many variants? I see at
> > least
> > >    4 (2 Marshallers, P2P).
> > >
> > > Plus we need add new test cases to test different key and value types and
> > > etc.
> > >
> > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split
> > all
> > > tests on 250 suites and run on all 30 TC agents it will take about 30-40
> > > hours. Ok, we can do it during weekends.
> > >
> > > I think it will take too much time.
> > >
> > > As an option we can start a cache for each configuration and run tests
> > > concurrently. But we need to implement this opportunity in our test
> > > framework.
> > >
> > > Any other thoughts how we can decrease time for tests?
> > >
> > > Thanks,
> > > -- Artem --
> > >
> > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]>
> > > wrote:
> > >
> > > > Artem,
> > > >
> > > > One more thing for new tests: I think test should start both server and
> > > > client nodes and use Ignite API from all nodes.
> > > >
> > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]>
> > > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > Actually, I don't have a list with all the permutations.
> > > > >
> > > > > At first, we need to split in our discussion test cases and Ignite
> > > > > configuration which should be covered.
> > > > >
> > > > > For example, new Full Api test cases for cache are based on old Full
> > > Api
> > > > > test cases. So, it need to think what the test cases was not covered
> > > > > before.
> > > > >
> > > > > About Ignite configurations, I'm going to add permutation for each
> > > > > IgniteConfiguration and CacheConfiguration property.
> > > > >
> > > > > By the way, the jira contains the following list of permutation (feel
> > > > free
> > > > > to add something):
> > > > >
> > > > > The following tests should be added (for functional blocks):
> > > > >
> > > > >    1. Interceptor
> > > > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > > > >    3. cache events
> > > > >    4. We should also test with Serializable, Externalizable, and
> > plain
> > > > >    Pojos for keys and values.
> > > > >    5. The Pojo in the above test should contain an enum value
> > > > >    6. We should also test Enums as keys and Enums as values
> > > > >    7. All operations should have single-key and multi-key operations
> > > > >
> > > > > New tests should cover all combinations for following properties:
> > > > >
> > > > >    1. cache modes
> > > > >    2. operation from client nodes and server nodes
> > > > >    3. store enabled/disabled
> > > > >    4. evicts sycn/non-sync
> > > > >    5. eviction policies
> > > > >    6. near on/off
> > > > >    7. marshallers (+ Binary marshaller with different mappers)
> > > > >    8. keys and values - externalizable, serializable, binaryzable,
> > > "none
> > > > of
> > > > >    previous"
> > > > >    9. classes available on servers: true/false
> > > > >    10. Peer loading on/off
> > > > >    11. Affinity functions
> > > > >    12. expiry policies
> > > > >
> > > > >
> > > > > Thanks,
> > > > > -- Artem --
> > > > >
> > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Artem, I think it is best to specify all the permutations here, so
> > > > others
> > > > > > can make additional suggestions. Otherwise, we cannot get a full
> > > > picture.
> > > > > >
> > > > > > Thanks,
> > > > > > D.
> > > > > >
> > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > I thought a little bit more and think we need to add a support
> > for
> > > > the
> > > > > > > following permutations too (I've added these to the jira
> > > > description):
> > > > > > > - We should also test with Serializable, Externalizable, and
> > plain
> > > > > Pojos
> > > > > > > for keys and values.
> > > > > > > - The Pojo in the above test should contain an enum value
> > > > > > > - We should also test Enums as keys and Enums as values
> > > > > > > - All operations should have single-key and multi-key operations
> > > > > > >
> > > > > > > Maybe someone see any other permutation to be supported?
> > > > > > >
> > > > > > > -- Artem --
> > > > > > >
> > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Dmitriy,
> > > > > > > >
> > > > > > > > There is a branch at my fork and a pull request at Ignite. See
> > > > > comment
> > > > > > > > about pull request at the ticket (PR-446).
> > > > > > > >
> > > > > > > > But I have to notice that the branch under hard development and
> > > you
> > > > > it
> > > > > > > can
> > > > > > > > not work (have compilation or test errors) at some moments.
> > > > > > > >
> > > > > > > > Good luck!
> > > > > > > >
> > > > > > > > -- Artem --
> > > > > > > >
> > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Artem,
> > > > > > > >>
> > > > > > > >> This is great. I have noticed from the ticket that you have
> > > > created
> > > > > > some
> > > > > > > >> initial suite already. Is there a branch I can look at it?
> > > > > > > >>
> > > > > > > >> D.
> > > > > > > >>
> > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> > > > [hidden email]
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Igniters,
> > > > > > > >> >
> > > > > > > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > > > > > > >> >
> > > > > > > >> > Ignite already has Full API test, but currently it's hard to
> > > > test
> > > > > > all
> > > > > > > >> > configuration combinations.
> > > > > > > >> >
> > > > > > > >> > Feel free to add comments in the jira if you have any
> > thought.
> > > > > > > >> >
> > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > > > >> > [2]
> > > > > > > >>
> > > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > -- Artem --
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
     
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

dsetrakyan
Andrey, I think the reason for the waste is some sort of internal
ring-buffer allocation which can be turned off.

Alexey G, can you please comment?

D.

On Mon, Feb 8, 2016 at 10:21 AM, Andrey Kornev <[hidden email]>
wrote:

> 50MB for an empty cache! That's a lot of emptiness!
>
> The Ignite team's recommendation to have one key/value type per cache now
> seems a little more dubious and difficult to follow in practice.
>
> Just curious, what is the reason for such waste? Is it because by default
> the initial cache size is set to 1.5M entries?
>
> Thanks
> Andrey
>
> > Date: Mon, 8 Feb 2016 17:45:33 +0300
> > Subject: Re: Full API coverage enhancement
> > From: [hidden email]
> > To: [hidden email]
> >
> > Sergey,
> >
> > I think we should start more caches, like 1000 in one time. But we have
> to
> > have enough memory on our TC agents. As I know, empty cache is require
> > about 50 mb (without indexing), am I right?
> >
> > You are right, I keep in mind that *backups* and *REPLICATED* mode make
> no
> > sense together, but we still have to test it in one node and multi node
> > casesю
> >
> > Any other *no sense* combinations?
> >
> > I forgot about custom BinaryConfiguration at IgniteConfiguration for
> > BinaryMarshaller. So, at least 6 IgniteConfigurations.
> >
> > -- Artem --
> >
> > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]>
> wrote:
> >
> > > Hi Artem
> > >
> > > It's good idea to create 20-30 cache configurations at once and then to
> > > iterate tests over those caches in parallel (but make sure that cache
> names
> > > are unique).
> > > Another point that some combinations make no sense like *backups *and
> > > *REPLICATED
> > > *cache
> > >
> > >
> > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have an update.
> > > >
> > > > I've started from *CacheConfiguration* permutations. I wrote out a
> list
> > > > with all CacheConfiguration setters and filtered it with Alexey G.
> > > >
> > > > Finally we have:
> > > >
> > > >    1. CacheMode - 3 variants
> > > >    2. CacheAtomicityMode - 2 variants
> > > >    3. CacheMemoryMode - 3 variants
> > > >    4. setLoadPreviousValue - 2 variants
> > > >    5. setReadFromBackup - 2 variants
> > > >    6. setStoreKeepBinary - 2 variants
> > > >    7. setRebalanceMode - SYNC and ASYNC (2 variants)
> > > >    8. setSwapEnabled - 2 variants
> > > >    9. setCopyOnRead - 2 variants
> > > >    10. NearConfiguration disabled / default NearConfiguration /
> custom
> > > >    NearConfiguration - 3 variants
> > > >    11. With and without a complex parameter. The complex parameter
> > > defines
> > > >    not-default Eviction policy and filter, cache store configuration
> > > >    (storeFactory and storeSessionListenerFactory), rebalancing
> > > > configuration,
> > > >    affinity function, offHeapMaxMemory, interceptor, topology
> validator
> > > and
> > > >    CacheEntryListener.
> > > >
> > > > I've run 123 Cache Full Api test cases for all permutations of
> parameters
> > > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test
> > > > cases). All these tests take 4 hours 40 minutes. Not all tests pass,
> so
> > > MAY
> > > > BE when all tests will pass it will take less time (3,5 hours for
> > > example).
> > > >
> > > > As we can see the tests take a lot of time.
> > > >
> > > > The following permutation should be supported too:
> > > >
> > > >    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes
> and 1
> > > >    backups, 4 nodes and 2 backups - 3 variants
> > > >    2. Client and Server nodes - 2 variants
> > > >    3. Indexing enabled and disabled for cache - 2 variants
> > > >    4. IgniteConfiguration permutations - how many variants? I see at
> > > least
> > > >    4 (2 Marshallers, P2P).
> > > >
> > > > Plus we need add new test cases to test different key and value
> types and
> > > > etc.
> > > >
> > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will
> split
> > > all
> > > > tests on 250 suites and run on all 30 TC agents it will take about
> 30-40
> > > > hours. Ok, we can do it during weekends.
> > > >
> > > > I think it will take too much time.
> > > >
> > > > As an option we can start a cache for each configuration and run
> tests
> > > > concurrently. But we need to implement this opportunity in our test
> > > > framework.
> > > >
> > > > Any other thoughts how we can decrease time for tests?
> > > >
> > > > Thanks,
> > > > -- Artem --
> > > >
> > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]>
> > > > wrote:
> > > >
> > > > > Artem,
> > > > >
> > > > > One more thing for new tests: I think test should start both
> server and
> > > > > client nodes and use Ignite API from all nodes.
> > > > >
> > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]
> >
> > > > wrote:
> > > > >
> > > > > > Dmitriy,
> > > > > >
> > > > > > Actually, I don't have a list with all the permutations.
> > > > > >
> > > > > > At first, we need to split in our discussion test cases and
> Ignite
> > > > > > configuration which should be covered.
> > > > > >
> > > > > > For example, new Full Api test cases for cache are based on old
> Full
> > > > Api
> > > > > > test cases. So, it need to think what the test cases was not
> covered
> > > > > > before.
> > > > > >
> > > > > > About Ignite configurations, I'm going to add permutation for
> each
> > > > > > IgniteConfiguration and CacheConfiguration property.
> > > > > >
> > > > > > By the way, the jira contains the following list of permutation
> (feel
> > > > > free
> > > > > > to add something):
> > > > > >
> > > > > > The following tests should be added (for functional blocks):
> > > > > >
> > > > > >    1. Interceptor
> > > > > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > > > > >    3. cache events
> > > > > >    4. We should also test with Serializable, Externalizable, and
> > > plain
> > > > > >    Pojos for keys and values.
> > > > > >    5. The Pojo in the above test should contain an enum value
> > > > > >    6. We should also test Enums as keys and Enums as values
> > > > > >    7. All operations should have single-key and multi-key
> operations
> > > > > >
> > > > > > New tests should cover all combinations for following properties:
> > > > > >
> > > > > >    1. cache modes
> > > > > >    2. operation from client nodes and server nodes
> > > > > >    3. store enabled/disabled
> > > > > >    4. evicts sycn/non-sync
> > > > > >    5. eviction policies
> > > > > >    6. near on/off
> > > > > >    7. marshallers (+ Binary marshaller with different mappers)
> > > > > >    8. keys and values - externalizable, serializable,
> binaryzable,
> > > > "none
> > > > > of
> > > > > >    previous"
> > > > > >    9. classes available on servers: true/false
> > > > > >    10. Peer loading on/off
> > > > > >    11. Affinity functions
> > > > > >    12. expiry policies
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > -- Artem --
> > > > > >
> > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> > > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Artem, I think it is best to specify all the permutations
> here, so
> > > > > others
> > > > > > > can make additional suggestions. Otherwise, we cannot get a
> full
> > > > > picture.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > D.
> > > > > > >
> > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <
> [hidden email]
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > I thought a little bit more and think we need to add a
> support
> > > for
> > > > > the
> > > > > > > > following permutations too (I've added these to the jira
> > > > > description):
> > > > > > > > - We should also test with Serializable, Externalizable, and
> > > plain
> > > > > > Pojos
> > > > > > > > for keys and values.
> > > > > > > > - The Pojo in the above test should contain an enum value
> > > > > > > > - We should also test Enums as keys and Enums as values
> > > > > > > > - All operations should have single-key and multi-key
> operations
> > > > > > > >
> > > > > > > > Maybe someone see any other permutation to be supported?
> > > > > > > >
> > > > > > > > -- Artem --
> > > > > > > >
> > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Dmitriy,
> > > > > > > > >
> > > > > > > > > There is a branch at my fork and a pull request at Ignite.
> See
> > > > > > comment
> > > > > > > > > about pull request at the ticket (PR-446).
> > > > > > > > >
> > > > > > > > > But I have to notice that the branch under hard
> development and
> > > > you
> > > > > > it
> > > > > > > > can
> > > > > > > > > not work (have compilation or test errors) at some moments.
> > > > > > > > >
> > > > > > > > > Good luck!
> > > > > > > > >
> > > > > > > > > -- Artem --
> > > > > > > > >
> > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Artem,
> > > > > > > > >>
> > > > > > > > >> This is great. I have noticed from the ticket that you
> have
> > > > > created
> > > > > > > some
> > > > > > > > >> initial suite already. Is there a branch I can look at it?
> > > > > > > > >>
> > > > > > > > >> D.
> > > > > > > > >>
> > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Igniters,
> > > > > > > > >> >
> > > > > > > > >> > I'm working on an enhancement of Full API coverage [1]
> [2].
> > > > > > > > >> >
> > > > > > > > >> > Ignite already has Full API test, but currently it's
> hard to
> > > > > test
> > > > > > > all
> > > > > > > > >> > configuration combinations.
> > > > > > > > >> >
> > > > > > > > >> > Feel free to add comments in the jira if you have any
> > > thought.
> > > > > > > > >> >
> > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > > > > >> > [2]
> > > > > > > > >>
> > > > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > -- Artem --
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sergey Kozlov
> > > GridGain Systems
> > > www.gridgain.com
> > >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Alexey Goncharuk
Dmitriy, the size of the circular buffer can be decreased by setting
IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE system property.

I was wondering what our next steps will be after implementing the
framework. From the initial discussion I thought we want to convert some
older tests to this new framework and run all new tests using this
framework. However, from what Artem already has written, this sounds
unrealistic because adding even a simple test case will induce hours of run
time. I think we still need to separate more important and less important
configurations, run important ones on a daily basis, and use all others for
after-code-freeze runs, for example.

Thoughts?
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

dsetrakyan
Agree about separation. I think we need to define some internal
permutations that keep coming up with bugs. I can start the list here:

   1. Serializable, Externalizable, neither.
   1. We should run the whole suite 3 times, once for each serialization
      mode. Having 2 or 3 nodes in the cluster here should be enough.
No need to
      test it on the changing cluster.
   2. On-heap and off-heap with entry processor and peer-deployment
   1. on-heap, entry-processor, peer-deployment=true/false
      2. off-heap, entry-processor, peer-deployment=true/false
   3. On-heap and off-heap with eviction policies
      1. eviction policy, memory-mode=on-heap
      2. eviction policy, memory-mode=off-heap
      3. eviction policy, memory-mode=off-heap-values
   4. On-heap and off-heap with continuous queries and
   peer-deployment=true/false
   5. …

I think we can come up with about 20 most important permutations. Thoughts?

D.


On Wed, Feb 10, 2016 at 9:24 AM, Alexey Goncharuk <
[hidden email]> wrote:

> Dmitriy, the size of the circular buffer can be decreased by setting
> IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE system property.
>
> I was wondering what our next steps will be after implementing the
> framework. From the initial discussion I thought we want to convert some
> older tests to this new framework and run all new tests using this
> framework. However, from what Artem already has written, this sounds
> unrealistic because adding even a simple test case will induce hours of run
> time. I think we still need to separate more important and less important
> configurations, run important ones on a daily basis, and use all others for
> after-code-freeze runs, for example.
>
> Thoughts?
>
Reply | Threaded
Open this post in threaded view
|

Re: Full API coverage enhancement

Artem Shutak
Igniters,

I'm glad to announce that Configuration Variations test framework has been
merged to master and is ready to use.

The framework provides an opportunity to write a test-case once and to run
it in many Ignite and Cache configuration variations, for different Ignite
topologies (with and without clients) and execute the same test scenario
from different Ignite nodes (server and client) automatically.

Another useful feature is "runInAllDataModes" functionality that allows to
run test-case and run it in all supported by framework data modes (like
Serializable, Externalizable, plane, mixed and etc. objects).

I've added the framework description on "Implementing Tests" [1] page.
Please, feel free to comment.

As proof-of-concept I've
added IgniteCacheBasicConfigVariationsFullApiTestSuite and add it on TC.

It does not mean that the work on IGNITE-2521
<https://issues.apache.org/jira/browse/IGNITE-2521> is completed and we
have coverage for full Ignite API, but I think it is a good improvement for
Ignite's test framework and Igniters can test functionality better and
simpler.

[1] https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
(see "Configuration variations test framework" section).

Thanks,
-- Artem --

On Thu, Feb 11, 2016 at 12:29 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Agree about separation. I think we need to define some internal
> permutations that keep coming up with bugs. I can start the list here:
>
>    1. Serializable, Externalizable, neither.
>    1. We should run the whole suite 3 times, once for each serialization
>       mode. Having 2 or 3 nodes in the cluster here should be enough.
> No need to
>       test it on the changing cluster.
>    2. On-heap and off-heap with entry processor and peer-deployment
>    1. on-heap, entry-processor, peer-deployment=true/false
>       2. off-heap, entry-processor, peer-deployment=true/false
>    3. On-heap and off-heap with eviction policies
>       1. eviction policy, memory-mode=on-heap
>       2. eviction policy, memory-mode=off-heap
>       3. eviction policy, memory-mode=off-heap-values
>    4. On-heap and off-heap with continuous queries and
>    peer-deployment=true/false
>    5. …
>
> I think we can come up with about 20 most important permutations. Thoughts?
>
> D.
>
>
> On Wed, Feb 10, 2016 at 9:24 AM, Alexey Goncharuk <
> [hidden email]> wrote:
>
> > Dmitriy, the size of the circular buffer can be decreased by setting
> > IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE system property.
> >
> > I was wondering what our next steps will be after implementing the
> > framework. From the initial discussion I thought we want to convert some
> > older tests to this new framework and run all new tests using this
> > framework. However, from what Artem already has written, this sounds
> > unrealistic because adding even a simple test case will induce hours of
> run
> > time. I think we still need to separate more important and less important
> > configurations, run important ones on a daily basis, and use all others
> for
> > after-code-freeze runs, for example.
> >
> > Thoughts?
> >
>
12