Igniters,
I'm working on an enhancement of Full API coverage [1] [2]. Ignite already has Full API test, but currently it's hard to test all configuration combinations. Feel free to add comments in the jira if you have any thought. [1] https://issues.apache.org/jira/browse/IGNITE-2521 [2] https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests Thanks, -- Artem -- |
Artem,
This is great. I have noticed from the ticket that you have created some initial suite already. Is there a branch I can look at it? D. On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]> wrote: > Igniters, > > I'm working on an enhancement of Full API coverage [1] [2]. > > Ignite already has Full API test, but currently it's hard to test all > configuration combinations. > > Feel free to add comments in the jira if you have any thought. > > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > [2] https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > Thanks, > -- Artem -- > |
Dmitriy,
There is a branch at my fork and a pull request at Ignite. See comment about pull request at the ticket (PR-446). But I have to notice that the branch under hard development and you it can not work (have compilation or test errors) at some moments. Good luck! -- Artem -- On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <[hidden email]> wrote: > Artem, > > This is great. I have noticed from the ticket that you have created some > initial suite already. Is there a branch I can look at it? > > D. > > On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]> > wrote: > > > Igniters, > > > > I'm working on an enhancement of Full API coverage [1] [2]. > > > > Ignite already has Full API test, but currently it's hard to test all > > configuration combinations. > > > > Feel free to add comments in the jira if you have any thought. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > [2] > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > Thanks, > > -- Artem -- > > > |
Igniters,
I thought a little bit more and think we need to add a support for the following permutations too (I've added these to the jira description): - We should also test with Serializable, Externalizable, and plain Pojos for keys and values. - The Pojo in the above test should contain an enum value - We should also test Enums as keys and Enums as values - All operations should have single-key and multi-key operations Maybe someone see any other permutation to be supported? -- Artem -- On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]> wrote: > Dmitriy, > > There is a branch at my fork and a pull request at Ignite. See comment > about pull request at the ticket (PR-446). > > But I have to notice that the branch under hard development and you it can > not work (have compilation or test errors) at some moments. > > Good luck! > > -- Artem -- > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > >> Artem, >> >> This is great. I have noticed from the ticket that you have created some >> initial suite already. Is there a branch I can look at it? >> >> D. >> >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]> >> wrote: >> >> > Igniters, >> > >> > I'm working on an enhancement of Full API coverage [1] [2]. >> > >> > Ignite already has Full API test, but currently it's hard to test all >> > configuration combinations. >> > >> > Feel free to add comments in the jira if you have any thought. >> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 >> > [2] >> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests >> > >> > Thanks, >> > -- Artem -- >> > >> > > |
Artem, I think it is best to specify all the permutations here, so others
can make additional suggestions. Otherwise, we cannot get a full picture. Thanks, D. On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> wrote: > Igniters, > > I thought a little bit more and think we need to add a support for the > following permutations too (I've added these to the jira description): > - We should also test with Serializable, Externalizable, and plain Pojos > for keys and values. > - The Pojo in the above test should contain an enum value > - We should also test Enums as keys and Enums as values > - All operations should have single-key and multi-key operations > > Maybe someone see any other permutation to be supported? > > -- Artem -- > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]> > wrote: > > > Dmitriy, > > > > There is a branch at my fork and a pull request at Ignite. See comment > > about pull request at the ticket (PR-446). > > > > But I have to notice that the branch under hard development and you it > can > > not work (have compilation or test errors) at some moments. > > > > Good luck! > > > > -- Artem -- > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <[hidden email] > > > > wrote: > > > >> Artem, > >> > >> This is great. I have noticed from the ticket that you have created some > >> initial suite already. Is there a branch I can look at it? > >> > >> D. > >> > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]> > >> wrote: > >> > >> > Igniters, > >> > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > >> > > >> > Ignite already has Full API test, but currently it's hard to test all > >> > configuration combinations. > >> > > >> > Feel free to add comments in the jira if you have any thought. > >> > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > >> > [2] > >> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > >> > > >> > Thanks, > >> > -- Artem -- > >> > > >> > > > > > |
Dmitriy,
Actually, I don't have a list with all the permutations. At first, we need to split in our discussion test cases and Ignite configuration which should be covered. For example, new Full Api test cases for cache are based on old Full Api test cases. So, it need to think what the test cases was not covered before. About Ignite configurations, I'm going to add permutation for each IgniteConfiguration and CacheConfiguration property. By the way, the jira contains the following list of permutation (feel free to add something): The following tests should be added (for functional blocks): 1. Interceptor 2. Queries: continuous, scan, SQL, fields and text queries. 3. cache events 4. We should also test with Serializable, Externalizable, and plain Pojos for keys and values. 5. The Pojo in the above test should contain an enum value 6. We should also test Enums as keys and Enums as values 7. All operations should have single-key and multi-key operations New tests should cover all combinations for following properties: 1. cache modes 2. operation from client nodes and server nodes 3. store enabled/disabled 4. evicts sycn/non-sync 5. eviction policies 6. near on/off 7. marshallers (+ Binary marshaller with different mappers) 8. keys and values - externalizable, serializable, binaryzable, "none of previous" 9. classes available on servers: true/false 10. Peer loading on/off 11. Affinity functions 12. expiry policies Thanks, -- Artem -- On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]> wrote: > Artem, I think it is best to specify all the permutations here, so others > can make additional suggestions. Otherwise, we cannot get a full picture. > > Thanks, > D. > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> wrote: > > > Igniters, > > > > I thought a little bit more and think we need to add a support for the > > following permutations too (I've added these to the jira description): > > - We should also test with Serializable, Externalizable, and plain Pojos > > for keys and values. > > - The Pojo in the above test should contain an enum value > > - We should also test Enums as keys and Enums as values > > - All operations should have single-key and multi-key operations > > > > Maybe someone see any other permutation to be supported? > > > > -- Artem -- > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]> > > wrote: > > > > > Dmitriy, > > > > > > There is a branch at my fork and a pull request at Ignite. See comment > > > about pull request at the ticket (PR-446). > > > > > > But I have to notice that the branch under hard development and you it > > can > > > not work (have compilation or test errors) at some moments. > > > > > > Good luck! > > > > > > -- Artem -- > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > [hidden email] > > > > > > wrote: > > > > > >> Artem, > > >> > > >> This is great. I have noticed from the ticket that you have created > some > > >> initial suite already. Is there a branch I can look at it? > > >> > > >> D. > > >> > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email]> > > >> wrote: > > >> > > >> > Igniters, > > >> > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > >> > > > >> > Ignite already has Full API test, but currently it's hard to test > all > > >> > configuration combinations. > > >> > > > >> > Feel free to add comments in the jira if you have any thought. > > >> > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > >> > [2] > > >> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > >> > > > >> > Thanks, > > >> > -- Artem -- > > >> > > > >> > > > > > > > > > |
Artem, could you describe (in general) how are you going to describe
permutations? Are you going to run tests with all possible combinations of all possible properties? Along time ago I implemented smth. similar (not in java) and I created framework where I could describe nested rules of what I want to investigate and engine that read that rules and iterate over all possible combination of rules and execute some action. Maybe this will be useful to you. On Wed, Feb 3, 2016 at 10:40 PM, Artem Shutak <[hidden email]> wrote: > Dmitriy, > > Actually, I don't have a list with all the permutations. > > At first, we need to split in our discussion test cases and Ignite > configuration which should be covered. > > For example, new Full Api test cases for cache are based on old Full Api > test cases. So, it need to think what the test cases was not covered > before. > > About Ignite configurations, I'm going to add permutation for each > IgniteConfiguration and CacheConfiguration property. > > By the way, the jira contains the following list of permutation (feel free > to add something): > > The following tests should be added (for functional blocks): > > 1. Interceptor > 2. Queries: continuous, scan, SQL, fields and text queries. > 3. cache events > 4. We should also test with Serializable, Externalizable, and plain > Pojos for keys and values. > 5. The Pojo in the above test should contain an enum value > 6. We should also test Enums as keys and Enums as values > 7. All operations should have single-key and multi-key operations > > New tests should cover all combinations for following properties: > > 1. cache modes > 2. operation from client nodes and server nodes > 3. store enabled/disabled > 4. evicts sycn/non-sync > 5. eviction policies > 6. near on/off > 7. marshallers (+ Binary marshaller with different mappers) > 8. keys and values - externalizable, serializable, binaryzable, "none of > previous" > 9. classes available on servers: true/false > 10. Peer loading on/off > 11. Affinity functions > 12. expiry policies > > > Thanks, > -- Artem -- > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > > > Artem, I think it is best to specify all the permutations here, so others > > can make additional suggestions. Otherwise, we cannot get a full picture. > > > > Thanks, > > D. > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> > wrote: > > > > > Igniters, > > > > > > I thought a little bit more and think we need to add a support for the > > > following permutations too (I've added these to the jira description): > > > - We should also test with Serializable, Externalizable, and plain > Pojos > > > for keys and values. > > > - The Pojo in the above test should contain an enum value > > > - We should also test Enums as keys and Enums as values > > > - All operations should have single-key and multi-key operations > > > > > > Maybe someone see any other permutation to be supported? > > > > > > -- Artem -- > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]> > > > wrote: > > > > > > > Dmitriy, > > > > > > > > There is a branch at my fork and a pull request at Ignite. See > comment > > > > about pull request at the ticket (PR-446). > > > > > > > > But I have to notice that the branch under hard development and you > it > > > can > > > > not work (have compilation or test errors) at some moments. > > > > > > > > Good luck! > > > > > > > > -- Artem -- > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > [hidden email] > > > > > > > > wrote: > > > > > > > >> Artem, > > > >> > > > >> This is great. I have noticed from the ticket that you have created > > some > > > >> initial suite already. Is there a branch I can look at it? > > > >> > > > >> D. > > > >> > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email] > > > > > >> wrote: > > > >> > > > >> > Igniters, > > > >> > > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > > >> > > > > >> > Ignite already has Full API test, but currently it's hard to test > > all > > > >> > configuration combinations. > > > >> > > > > >> > Feel free to add comments in the jira if you have any thought. > > > >> > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > >> > [2] > > > >> > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > >> > > > > >> > Thanks, > > > >> > -- Artem -- > > > >> > > > > >> > > > > > > > > > > > > > > -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
In reply to this post by Artem Shutak
Thanks Artem! In your view, how many combinations of the configuration
properties will be tested together. I don’t think we can afford to test every possible combination. On Wed, Feb 3, 2016 at 7:40 AM, Artem Shutak <[hidden email]> wrote: > Dmitriy, > > Actually, I don't have a list with all the permutations. > > At first, we need to split in our discussion test cases and Ignite > configuration which should be covered. > > For example, new Full Api test cases for cache are based on old Full Api > test cases. So, it need to think what the test cases was not covered > before. > > About Ignite configurations, I'm going to add permutation for each > IgniteConfiguration and CacheConfiguration property. > > By the way, the jira contains the following list of permutation (feel free > to add something): > > The following tests should be added (for functional blocks): > > 1. Interceptor > 2. Queries: continuous, scan, SQL, fields and text queries. > 3. cache events > 4. We should also test with Serializable, Externalizable, and plain > Pojos for keys and values. > 5. The Pojo in the above test should contain an enum value > 6. We should also test Enums as keys and Enums as values > 7. All operations should have single-key and multi-key operations > > New tests should cover all combinations for following properties: > > 1. cache modes > 2. operation from client nodes and server nodes > 3. store enabled/disabled > 4. evicts sycn/non-sync > 5. eviction policies > 6. near on/off > 7. marshallers (+ Binary marshaller with different mappers) > 8. keys and values - externalizable, serializable, binaryzable, "none of > previous" > 9. classes available on servers: true/false > 10. Peer loading on/off > 11. Affinity functions > 12. expiry policies > > > Thanks, > -- Artem -- > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > > > Artem, I think it is best to specify all the permutations here, so others > > can make additional suggestions. Otherwise, we cannot get a full picture. > > > > Thanks, > > D. > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> > wrote: > > > > > Igniters, > > > > > > I thought a little bit more and think we need to add a support for the > > > following permutations too (I've added these to the jira description): > > > - We should also test with Serializable, Externalizable, and plain > Pojos > > > for keys and values. > > > - The Pojo in the above test should contain an enum value > > > - We should also test Enums as keys and Enums as values > > > - All operations should have single-key and multi-key operations > > > > > > Maybe someone see any other permutation to be supported? > > > > > > -- Artem -- > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]> > > > wrote: > > > > > > > Dmitriy, > > > > > > > > There is a branch at my fork and a pull request at Ignite. See > comment > > > > about pull request at the ticket (PR-446). > > > > > > > > But I have to notice that the branch under hard development and you > it > > > can > > > > not work (have compilation or test errors) at some moments. > > > > > > > > Good luck! > > > > > > > > -- Artem -- > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > [hidden email] > > > > > > > > wrote: > > > > > > > >> Artem, > > > >> > > > >> This is great. I have noticed from the ticket that you have created > > some > > > >> initial suite already. Is there a branch I can look at it? > > > >> > > > >> D. > > > >> > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email] > > > > > >> wrote: > > > >> > > > >> > Igniters, > > > >> > > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > > >> > > > > >> > Ignite already has Full API test, but currently it's hard to test > > all > > > >> > configuration combinations. > > > >> > > > > >> > Feel free to add comments in the jira if you have any thought. > > > >> > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > >> > [2] > > > >> > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > >> > > > > >> > Thanks, > > > >> > -- Artem -- > > > >> > > > > >> > > > > > > > > > > > > > > |
In reply to this post by Artem Shutak
Artem,
One more thing for new tests: I think test should start both server and client nodes and use Ignite API from all nodes. On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]> wrote: > Dmitriy, > > Actually, I don't have a list with all the permutations. > > At first, we need to split in our discussion test cases and Ignite > configuration which should be covered. > > For example, new Full Api test cases for cache are based on old Full Api > test cases. So, it need to think what the test cases was not covered > before. > > About Ignite configurations, I'm going to add permutation for each > IgniteConfiguration and CacheConfiguration property. > > By the way, the jira contains the following list of permutation (feel free > to add something): > > The following tests should be added (for functional blocks): > > 1. Interceptor > 2. Queries: continuous, scan, SQL, fields and text queries. > 3. cache events > 4. We should also test with Serializable, Externalizable, and plain > Pojos for keys and values. > 5. The Pojo in the above test should contain an enum value > 6. We should also test Enums as keys and Enums as values > 7. All operations should have single-key and multi-key operations > > New tests should cover all combinations for following properties: > > 1. cache modes > 2. operation from client nodes and server nodes > 3. store enabled/disabled > 4. evicts sycn/non-sync > 5. eviction policies > 6. near on/off > 7. marshallers (+ Binary marshaller with different mappers) > 8. keys and values - externalizable, serializable, binaryzable, "none of > previous" > 9. classes available on servers: true/false > 10. Peer loading on/off > 11. Affinity functions > 12. expiry policies > > > Thanks, > -- Artem -- > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > > > Artem, I think it is best to specify all the permutations here, so others > > can make additional suggestions. Otherwise, we cannot get a full picture. > > > > Thanks, > > D. > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> > wrote: > > > > > Igniters, > > > > > > I thought a little bit more and think we need to add a support for the > > > following permutations too (I've added these to the jira description): > > > - We should also test with Serializable, Externalizable, and plain > Pojos > > > for keys and values. > > > - The Pojo in the above test should contain an enum value > > > - We should also test Enums as keys and Enums as values > > > - All operations should have single-key and multi-key operations > > > > > > Maybe someone see any other permutation to be supported? > > > > > > -- Artem -- > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]> > > > wrote: > > > > > > > Dmitriy, > > > > > > > > There is a branch at my fork and a pull request at Ignite. See > comment > > > > about pull request at the ticket (PR-446). > > > > > > > > But I have to notice that the branch under hard development and you > it > > > can > > > > not work (have compilation or test errors) at some moments. > > > > > > > > Good luck! > > > > > > > > -- Artem -- > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > [hidden email] > > > > > > > > wrote: > > > > > > > >> Artem, > > > >> > > > >> This is great. I have noticed from the ticket that you have created > > some > > > >> initial suite already. Is there a branch I can look at it? > > > >> > > > >> D. > > > >> > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <[hidden email] > > > > > >> wrote: > > > >> > > > >> > Igniters, > > > >> > > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > > >> > > > > >> > Ignite already has Full API test, but currently it's hard to test > > all > > > >> > configuration combinations. > > > >> > > > > >> > Feel free to add comments in the jira if you have any thought. > > > >> > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > >> > [2] > > > >> > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > >> > > > > >> > Thanks, > > > >> > -- Artem -- > > > >> > > > > >> > > > > > > > > > > > > > > |
Hi all,
I have an update. I've started from *CacheConfiguration* permutations. I wrote out a list with all CacheConfiguration setters and filtered it with Alexey G. Finally we have: 1. CacheMode - 3 variants 2. CacheAtomicityMode - 2 variants 3. CacheMemoryMode - 3 variants 4. setLoadPreviousValue - 2 variants 5. setReadFromBackup - 2 variants 6. setStoreKeepBinary - 2 variants 7. setRebalanceMode - SYNC and ASYNC (2 variants) 8. setSwapEnabled - 2 variants 9. setCopyOnRead - 2 variants 10. NearConfiguration disabled / default NearConfiguration / custom NearConfiguration - 3 variants 11. With and without a complex parameter. The complex parameter defines not-default Eviction policy and filter, cache store configuration (storeFactory and storeSessionListenerFactory), rebalancing configuration, affinity function, offHeapMaxMemory, interceptor, topology validator and CacheEntryListener. I've run 123 Cache Full Api test cases for all permutations of parameters 1-9 and got 256896 test cases (1152 configuration variants * 123 test cases). All these tests take 4 hours 40 minutes. Not all tests pass, so MAY BE when all tests will pass it will take less time (3,5 hours for example). As we can see the tests take a lot of time. The following permutation should be supported too: 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1 backups, 4 nodes and 2 backups - 3 variants 2. Client and Server nodes - 2 variants 3. Indexing enabled and disabled for cache - 2 variants 4. IgniteConfiguration permutations - how many variants? I see at least 4 (2 Marshallers, P2P). Plus we need add new test cases to test different key and value types and etc. So, we need multiply more then 3,5-4,5 hours on ~250. If we will split all tests on 250 suites and run on all 30 TC agents it will take about 30-40 hours. Ok, we can do it during weekends. I think it will take too much time. As an option we can start a cache for each configuration and run tests concurrently. But we need to implement this opportunity in our test framework. Any other thoughts how we can decrease time for tests? Thanks, -- Artem -- On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]> wrote: > Artem, > > One more thing for new tests: I think test should start both server and > client nodes and use Ignite API from all nodes. > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]> wrote: > > > Dmitriy, > > > > Actually, I don't have a list with all the permutations. > > > > At first, we need to split in our discussion test cases and Ignite > > configuration which should be covered. > > > > For example, new Full Api test cases for cache are based on old Full Api > > test cases. So, it need to think what the test cases was not covered > > before. > > > > About Ignite configurations, I'm going to add permutation for each > > IgniteConfiguration and CacheConfiguration property. > > > > By the way, the jira contains the following list of permutation (feel > free > > to add something): > > > > The following tests should be added (for functional blocks): > > > > 1. Interceptor > > 2. Queries: continuous, scan, SQL, fields and text queries. > > 3. cache events > > 4. We should also test with Serializable, Externalizable, and plain > > Pojos for keys and values. > > 5. The Pojo in the above test should contain an enum value > > 6. We should also test Enums as keys and Enums as values > > 7. All operations should have single-key and multi-key operations > > > > New tests should cover all combinations for following properties: > > > > 1. cache modes > > 2. operation from client nodes and server nodes > > 3. store enabled/disabled > > 4. evicts sycn/non-sync > > 5. eviction policies > > 6. near on/off > > 7. marshallers (+ Binary marshaller with different mappers) > > 8. keys and values - externalizable, serializable, binaryzable, "none > of > > previous" > > 9. classes available on servers: true/false > > 10. Peer loading on/off > > 11. Affinity functions > > 12. expiry policies > > > > > > Thanks, > > -- Artem -- > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <[hidden email] > > > > wrote: > > > > > Artem, I think it is best to specify all the permutations here, so > others > > > can make additional suggestions. Otherwise, we cannot get a full > picture. > > > > > > Thanks, > > > D. > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> > > wrote: > > > > > > > Igniters, > > > > > > > > I thought a little bit more and think we need to add a support for > the > > > > following permutations too (I've added these to the jira > description): > > > > - We should also test with Serializable, Externalizable, and plain > > Pojos > > > > for keys and values. > > > > - The Pojo in the above test should contain an enum value > > > > - We should also test Enums as keys and Enums as values > > > > - All operations should have single-key and multi-key operations > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > -- Artem -- > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <[hidden email]> > > > > wrote: > > > > > > > > > Dmitriy, > > > > > > > > > > There is a branch at my fork and a pull request at Ignite. See > > comment > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > But I have to notice that the branch under hard development and you > > it > > > > can > > > > > not work (have compilation or test errors) at some moments. > > > > > > > > > > Good luck! > > > > > > > > > > -- Artem -- > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > >> Artem, > > > > >> > > > > >> This is great. I have noticed from the ticket that you have > created > > > some > > > > >> initial suite already. Is there a branch I can look at it? > > > > >> > > > > >> D. > > > > >> > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > [hidden email] > > > > > > > >> wrote: > > > > >> > > > > >> > Igniters, > > > > >> > > > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > > > >> > > > > > >> > Ignite already has Full API test, but currently it's hard to > test > > > all > > > > >> > configuration combinations. > > > > >> > > > > > >> > Feel free to add comments in the jira if you have any thought. > > > > >> > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > >> > [2] > > > > >> > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > >> > > > > > >> > Thanks, > > > > >> > -- Artem -- > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > |
Hi Artem
It's good idea to create 20-30 cache configurations at once and then to iterate tests over those caches in parallel (but make sure that cache names are unique). Another point that some combinations make no sense like *backups *and *REPLICATED *cache On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> wrote: > Hi all, > > I have an update. > > I've started from *CacheConfiguration* permutations. I wrote out a list > with all CacheConfiguration setters and filtered it with Alexey G. > > Finally we have: > > 1. CacheMode - 3 variants > 2. CacheAtomicityMode - 2 variants > 3. CacheMemoryMode - 3 variants > 4. setLoadPreviousValue - 2 variants > 5. setReadFromBackup - 2 variants > 6. setStoreKeepBinary - 2 variants > 7. setRebalanceMode - SYNC and ASYNC (2 variants) > 8. setSwapEnabled - 2 variants > 9. setCopyOnRead - 2 variants > 10. NearConfiguration disabled / default NearConfiguration / custom > NearConfiguration - 3 variants > 11. With and without a complex parameter. The complex parameter defines > not-default Eviction policy and filter, cache store configuration > (storeFactory and storeSessionListenerFactory), rebalancing > configuration, > affinity function, offHeapMaxMemory, interceptor, topology validator and > CacheEntryListener. > > I've run 123 Cache Full Api test cases for all permutations of parameters > 1-9 and got 256896 test cases (1152 configuration variants * 123 test > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so MAY > BE when all tests will pass it will take less time (3,5 hours for example). > > As we can see the tests take a lot of time. > > The following permutation should be supported too: > > 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1 > backups, 4 nodes and 2 backups - 3 variants > 2. Client and Server nodes - 2 variants > 3. Indexing enabled and disabled for cache - 2 variants > 4. IgniteConfiguration permutations - how many variants? I see at least > 4 (2 Marshallers, P2P). > > Plus we need add new test cases to test different key and value types and > etc. > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split all > tests on 250 suites and run on all 30 TC agents it will take about 30-40 > hours. Ok, we can do it during weekends. > > I think it will take too much time. > > As an option we can start a cache for each configuration and run tests > concurrently. But we need to implement this opportunity in our test > framework. > > Any other thoughts how we can decrease time for tests? > > Thanks, > -- Artem -- > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]> > wrote: > > > Artem, > > > > One more thing for new tests: I think test should start both server and > > client nodes and use Ignite API from all nodes. > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]> > wrote: > > > > > Dmitriy, > > > > > > Actually, I don't have a list with all the permutations. > > > > > > At first, we need to split in our discussion test cases and Ignite > > > configuration which should be covered. > > > > > > For example, new Full Api test cases for cache are based on old Full > Api > > > test cases. So, it need to think what the test cases was not covered > > > before. > > > > > > About Ignite configurations, I'm going to add permutation for each > > > IgniteConfiguration and CacheConfiguration property. > > > > > > By the way, the jira contains the following list of permutation (feel > > free > > > to add something): > > > > > > The following tests should be added (for functional blocks): > > > > > > 1. Interceptor > > > 2. Queries: continuous, scan, SQL, fields and text queries. > > > 3. cache events > > > 4. We should also test with Serializable, Externalizable, and plain > > > Pojos for keys and values. > > > 5. The Pojo in the above test should contain an enum value > > > 6. We should also test Enums as keys and Enums as values > > > 7. All operations should have single-key and multi-key operations > > > > > > New tests should cover all combinations for following properties: > > > > > > 1. cache modes > > > 2. operation from client nodes and server nodes > > > 3. store enabled/disabled > > > 4. evicts sycn/non-sync > > > 5. eviction policies > > > 6. near on/off > > > 7. marshallers (+ Binary marshaller with different mappers) > > > 8. keys and values - externalizable, serializable, binaryzable, > "none > > of > > > previous" > > > 9. classes available on servers: true/false > > > 10. Peer loading on/off > > > 11. Affinity functions > > > 12. expiry policies > > > > > > > > > Thanks, > > > -- Artem -- > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan < > [hidden email] > > > > > > wrote: > > > > > > > Artem, I think it is best to specify all the permutations here, so > > others > > > > can make additional suggestions. Otherwise, we cannot get a full > > picture. > > > > > > > > Thanks, > > > > D. > > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email]> > > > wrote: > > > > > > > > > Igniters, > > > > > > > > > > I thought a little bit more and think we need to add a support for > > the > > > > > following permutations too (I've added these to the jira > > description): > > > > > - We should also test with Serializable, Externalizable, and plain > > > Pojos > > > > > for keys and values. > > > > > - The Pojo in the above test should contain an enum value > > > > > - We should also test Enums as keys and Enums as values > > > > > - All operations should have single-key and multi-key operations > > > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > > > -- Artem -- > > > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak < > [hidden email]> > > > > > wrote: > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > There is a branch at my fork and a pull request at Ignite. See > > > comment > > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > > > But I have to notice that the branch under hard development and > you > > > it > > > > > can > > > > > > not work (have compilation or test errors) at some moments. > > > > > > > > > > > > Good luck! > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > >> Artem, > > > > > >> > > > > > >> This is great. I have noticed from the ticket that you have > > created > > > > some > > > > > >> initial suite already. Is there a branch I can look at it? > > > > > >> > > > > > >> D. > > > > > >> > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > > [hidden email] > > > > > > > > > >> wrote: > > > > > >> > > > > > >> > Igniters, > > > > > >> > > > > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > > > > >> > > > > > > >> > Ignite already has Full API test, but currently it's hard to > > test > > > > all > > > > > >> > configuration combinations. > > > > > >> > > > > > > >> > Feel free to add comments in the jira if you have any thought. > > > > > >> > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > > >> > [2] > > > > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > > >> > > > > > > >> > Thanks, > > > > > >> > -- Artem -- > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Sergey Kozlov GridGain Systems www.gridgain.com |
Sergey,
I think we should start more caches, like 1000 in one time. But we have to have enough memory on our TC agents. As I know, empty cache is require about 50 mb (without indexing), am I right? You are right, I keep in mind that *backups* and *REPLICATED* mode make no sense together, but we still have to test it in one node and multi node casesю Any other *no sense* combinations? I forgot about custom BinaryConfiguration at IgniteConfiguration for BinaryMarshaller. So, at least 6 IgniteConfigurations. -- Artem -- On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]> wrote: > Hi Artem > > It's good idea to create 20-30 cache configurations at once and then to > iterate tests over those caches in parallel (but make sure that cache names > are unique). > Another point that some combinations make no sense like *backups *and > *REPLICATED > *cache > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> wrote: > > > Hi all, > > > > I have an update. > > > > I've started from *CacheConfiguration* permutations. I wrote out a list > > with all CacheConfiguration setters and filtered it with Alexey G. > > > > Finally we have: > > > > 1. CacheMode - 3 variants > > 2. CacheAtomicityMode - 2 variants > > 3. CacheMemoryMode - 3 variants > > 4. setLoadPreviousValue - 2 variants > > 5. setReadFromBackup - 2 variants > > 6. setStoreKeepBinary - 2 variants > > 7. setRebalanceMode - SYNC and ASYNC (2 variants) > > 8. setSwapEnabled - 2 variants > > 9. setCopyOnRead - 2 variants > > 10. NearConfiguration disabled / default NearConfiguration / custom > > NearConfiguration - 3 variants > > 11. With and without a complex parameter. The complex parameter > defines > > not-default Eviction policy and filter, cache store configuration > > (storeFactory and storeSessionListenerFactory), rebalancing > > configuration, > > affinity function, offHeapMaxMemory, interceptor, topology validator > and > > CacheEntryListener. > > > > I've run 123 Cache Full Api test cases for all permutations of parameters > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test > > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so > MAY > > BE when all tests will pass it will take less time (3,5 hours for > example). > > > > As we can see the tests take a lot of time. > > > > The following permutation should be supported too: > > > > 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1 > > backups, 4 nodes and 2 backups - 3 variants > > 2. Client and Server nodes - 2 variants > > 3. Indexing enabled and disabled for cache - 2 variants > > 4. IgniteConfiguration permutations - how many variants? I see at > least > > 4 (2 Marshallers, P2P). > > > > Plus we need add new test cases to test different key and value types and > > etc. > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split > all > > tests on 250 suites and run on all 30 TC agents it will take about 30-40 > > hours. Ok, we can do it during weekends. > > > > I think it will take too much time. > > > > As an option we can start a cache for each configuration and run tests > > concurrently. But we need to implement this opportunity in our test > > framework. > > > > Any other thoughts how we can decrease time for tests? > > > > Thanks, > > -- Artem -- > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]> > > wrote: > > > > > Artem, > > > > > > One more thing for new tests: I think test should start both server and > > > client nodes and use Ignite API from all nodes. > > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]> > > wrote: > > > > > > > Dmitriy, > > > > > > > > Actually, I don't have a list with all the permutations. > > > > > > > > At first, we need to split in our discussion test cases and Ignite > > > > configuration which should be covered. > > > > > > > > For example, new Full Api test cases for cache are based on old Full > > Api > > > > test cases. So, it need to think what the test cases was not covered > > > > before. > > > > > > > > About Ignite configurations, I'm going to add permutation for each > > > > IgniteConfiguration and CacheConfiguration property. > > > > > > > > By the way, the jira contains the following list of permutation (feel > > > free > > > > to add something): > > > > > > > > The following tests should be added (for functional blocks): > > > > > > > > 1. Interceptor > > > > 2. Queries: continuous, scan, SQL, fields and text queries. > > > > 3. cache events > > > > 4. We should also test with Serializable, Externalizable, and > plain > > > > Pojos for keys and values. > > > > 5. The Pojo in the above test should contain an enum value > > > > 6. We should also test Enums as keys and Enums as values > > > > 7. All operations should have single-key and multi-key operations > > > > > > > > New tests should cover all combinations for following properties: > > > > > > > > 1. cache modes > > > > 2. operation from client nodes and server nodes > > > > 3. store enabled/disabled > > > > 4. evicts sycn/non-sync > > > > 5. eviction policies > > > > 6. near on/off > > > > 7. marshallers (+ Binary marshaller with different mappers) > > > > 8. keys and values - externalizable, serializable, binaryzable, > > "none > > > of > > > > previous" > > > > 9. classes available on servers: true/false > > > > 10. Peer loading on/off > > > > 11. Affinity functions > > > > 12. expiry policies > > > > > > > > > > > > Thanks, > > > > -- Artem -- > > > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan < > > [hidden email] > > > > > > > > wrote: > > > > > > > > > Artem, I think it is best to specify all the permutations here, so > > > others > > > > > can make additional suggestions. Otherwise, we cannot get a full > > > picture. > > > > > > > > > > Thanks, > > > > > D. > > > > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email] > > > > > > wrote: > > > > > > > > > > > Igniters, > > > > > > > > > > > > I thought a little bit more and think we need to add a support > for > > > the > > > > > > following permutations too (I've added these to the jira > > > description): > > > > > > - We should also test with Serializable, Externalizable, and > plain > > > > Pojos > > > > > > for keys and values. > > > > > > - The Pojo in the above test should contain an enum value > > > > > > - We should also test Enums as keys and Enums as values > > > > > > - All operations should have single-key and multi-key operations > > > > > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak < > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > There is a branch at my fork and a pull request at Ignite. See > > > > comment > > > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > > > > > But I have to notice that the branch under hard development and > > you > > > > it > > > > > > can > > > > > > > not work (have compilation or test errors) at some moments. > > > > > > > > > > > > > > Good luck! > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> Artem, > > > > > > >> > > > > > > >> This is great. I have noticed from the ticket that you have > > > created > > > > > some > > > > > > >> initial suite already. Is there a branch I can look at it? > > > > > > >> > > > > > > >> D. > > > > > > >> > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > > > [hidden email] > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Igniters, > > > > > > >> > > > > > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > > > > > >> > > > > > > > >> > Ignite already has Full API test, but currently it's hard to > > > test > > > > > all > > > > > > >> > configuration combinations. > > > > > > >> > > > > > > > >> > Feel free to add comments in the jira if you have any > thought. > > > > > > >> > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > > > >> > [2] > > > > > > >> > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > > > >> > > > > > > > >> > Thanks, > > > > > > >> > -- Artem -- > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com > |
1000 caches x 50MB = 50GB heap. Do we really have >50GB RAM on each agents?
On Mon, Feb 8, 2016 at 5:45 PM, Artem Shutak <[hidden email]> wrote: > Sergey, > > I think we should start more caches, like 1000 in one time. But we have to > have enough memory on our TC agents. As I know, empty cache is require > about 50 mb (without indexing), am I right? > > You are right, I keep in mind that *backups* and *REPLICATED* mode make no > sense together, but we still have to test it in one node and multi node > casesю > > Any other *no sense* combinations? > > I forgot about custom BinaryConfiguration at IgniteConfiguration for > BinaryMarshaller. So, at least 6 IgniteConfigurations. > > -- Artem -- > > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]> > wrote: > > > Hi Artem > > > > It's good idea to create 20-30 cache configurations at once and then to > > iterate tests over those caches in parallel (but make sure that cache > names > > are unique). > > Another point that some combinations make no sense like *backups *and > > *REPLICATED > > *cache > > > > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> > wrote: > > > > > Hi all, > > > > > > I have an update. > > > > > > I've started from *CacheConfiguration* permutations. I wrote out a list > > > with all CacheConfiguration setters and filtered it with Alexey G. > > > > > > Finally we have: > > > > > > 1. CacheMode - 3 variants > > > 2. CacheAtomicityMode - 2 variants > > > 3. CacheMemoryMode - 3 variants > > > 4. setLoadPreviousValue - 2 variants > > > 5. setReadFromBackup - 2 variants > > > 6. setStoreKeepBinary - 2 variants > > > 7. setRebalanceMode - SYNC and ASYNC (2 variants) > > > 8. setSwapEnabled - 2 variants > > > 9. setCopyOnRead - 2 variants > > > 10. NearConfiguration disabled / default NearConfiguration / custom > > > NearConfiguration - 3 variants > > > 11. With and without a complex parameter. The complex parameter > > defines > > > not-default Eviction policy and filter, cache store configuration > > > (storeFactory and storeSessionListenerFactory), rebalancing > > > configuration, > > > affinity function, offHeapMaxMemory, interceptor, topology validator > > and > > > CacheEntryListener. > > > > > > I've run 123 Cache Full Api test cases for all permutations of > parameters > > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test > > > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so > > MAY > > > BE when all tests will pass it will take less time (3,5 hours for > > example). > > > > > > As we can see the tests take a lot of time. > > > > > > The following permutation should be supported too: > > > > > > 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1 > > > backups, 4 nodes and 2 backups - 3 variants > > > 2. Client and Server nodes - 2 variants > > > 3. Indexing enabled and disabled for cache - 2 variants > > > 4. IgniteConfiguration permutations - how many variants? I see at > > least > > > 4 (2 Marshallers, P2P). > > > > > > Plus we need add new test cases to test different key and value types > and > > > etc. > > > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split > > all > > > tests on 250 suites and run on all 30 TC agents it will take about > 30-40 > > > hours. Ok, we can do it during weekends. > > > > > > I think it will take too much time. > > > > > > As an option we can start a cache for each configuration and run tests > > > concurrently. But we need to implement this opportunity in our test > > > framework. > > > > > > Any other thoughts how we can decrease time for tests? > > > > > > Thanks, > > > -- Artem -- > > > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]> > > > wrote: > > > > > > > Artem, > > > > > > > > One more thing for new tests: I think test should start both server > and > > > > client nodes and use Ignite API from all nodes. > > > > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]> > > > wrote: > > > > > > > > > Dmitriy, > > > > > > > > > > Actually, I don't have a list with all the permutations. > > > > > > > > > > At first, we need to split in our discussion test cases and Ignite > > > > > configuration which should be covered. > > > > > > > > > > For example, new Full Api test cases for cache are based on old > Full > > > Api > > > > > test cases. So, it need to think what the test cases was not > covered > > > > > before. > > > > > > > > > > About Ignite configurations, I'm going to add permutation for each > > > > > IgniteConfiguration and CacheConfiguration property. > > > > > > > > > > By the way, the jira contains the following list of permutation > (feel > > > > free > > > > > to add something): > > > > > > > > > > The following tests should be added (for functional blocks): > > > > > > > > > > 1. Interceptor > > > > > 2. Queries: continuous, scan, SQL, fields and text queries. > > > > > 3. cache events > > > > > 4. We should also test with Serializable, Externalizable, and > > plain > > > > > Pojos for keys and values. > > > > > 5. The Pojo in the above test should contain an enum value > > > > > 6. We should also test Enums as keys and Enums as values > > > > > 7. All operations should have single-key and multi-key > operations > > > > > > > > > > New tests should cover all combinations for following properties: > > > > > > > > > > 1. cache modes > > > > > 2. operation from client nodes and server nodes > > > > > 3. store enabled/disabled > > > > > 4. evicts sycn/non-sync > > > > > 5. eviction policies > > > > > 6. near on/off > > > > > 7. marshallers (+ Binary marshaller with different mappers) > > > > > 8. keys and values - externalizable, serializable, binaryzable, > > > "none > > > > of > > > > > previous" > > > > > 9. classes available on servers: true/false > > > > > 10. Peer loading on/off > > > > > 11. Affinity functions > > > > > 12. expiry policies > > > > > > > > > > > > > > > Thanks, > > > > > -- Artem -- > > > > > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan < > > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > Artem, I think it is best to specify all the permutations here, > so > > > > others > > > > > > can make additional suggestions. Otherwise, we cannot get a full > > > > picture. > > > > > > > > > > > > Thanks, > > > > > > D. > > > > > > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > I thought a little bit more and think we need to add a support > > for > > > > the > > > > > > > following permutations too (I've added these to the jira > > > > description): > > > > > > > - We should also test with Serializable, Externalizable, and > > plain > > > > > Pojos > > > > > > > for keys and values. > > > > > > > - The Pojo in the above test should contain an enum value > > > > > > > - We should also test Enums as keys and Enums as values > > > > > > > - All operations should have single-key and multi-key > operations > > > > > > > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak < > > > [hidden email]> > > > > > > > wrote: > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > There is a branch at my fork and a pull request at Ignite. > See > > > > > comment > > > > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > > > > > > > But I have to notice that the branch under hard development > and > > > you > > > > > it > > > > > > > can > > > > > > > > not work (have compilation or test errors) at some moments. > > > > > > > > > > > > > > > > Good luck! > > > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> Artem, > > > > > > > >> > > > > > > > >> This is great. I have noticed from the ticket that you have > > > > created > > > > > > some > > > > > > > >> initial suite already. Is there a branch I can look at it? > > > > > > > >> > > > > > > > >> D. > > > > > > > >> > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > > > > [hidden email] > > > > > > > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > Igniters, > > > > > > > >> > > > > > > > > >> > I'm working on an enhancement of Full API coverage [1] > [2]. > > > > > > > >> > > > > > > > > >> > Ignite already has Full API test, but currently it's hard > to > > > > test > > > > > > all > > > > > > > >> > configuration combinations. > > > > > > > >> > > > > > > > > >> > Feel free to add comments in the jira if you have any > > thought. > > > > > > > >> > > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > > > > >> > [2] > > > > > > > >> > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > > > > >> > > > > > > > > >> > Thanks, > > > > > > > >> > -- Artem -- > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Sergey Kozlov > > GridGain Systems > > www.gridgain.com > > > -- Sergey Kozlov GridGain Systems www.gridgain.com |
How about such approach:
Generate all permutation descriptors and save them to database. On each agent start some process that will grab from DB some descriptors (lets say 100) mark them as "in progress" and execute them. Other agents will grab remaining "not in progress" descriptors and do the same. So we could run in parallel all permutations even if we have 1M of them. What do you think? On Mon, Feb 8, 2016 at 9:52 PM, Sergey Kozlov <[hidden email]> wrote: > 1000 caches x 50MB = 50GB heap. Do we really have >50GB RAM on each agents? > > On Mon, Feb 8, 2016 at 5:45 PM, Artem Shutak <[hidden email]> wrote: > > > Sergey, > > > > I think we should start more caches, like 1000 in one time. But we have > to > > have enough memory on our TC agents. As I know, empty cache is require > > about 50 mb (without indexing), am I right? > > > > You are right, I keep in mind that *backups* and *REPLICATED* mode make > no > > sense together, but we still have to test it in one node and multi node > > casesю > > > > Any other *no sense* combinations? > > > > I forgot about custom BinaryConfiguration at IgniteConfiguration for > > BinaryMarshaller. So, at least 6 IgniteConfigurations. > > > > -- Artem -- > > > > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]> > > wrote: > > > > > Hi Artem > > > > > > It's good idea to create 20-30 cache configurations at once and then to > > > iterate tests over those caches in parallel (but make sure that cache > > names > > > are unique). > > > Another point that some combinations make no sense like *backups *and > > > *REPLICATED > > > *cache > > > > > > > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> > > wrote: > > > > > > > Hi all, > > > > > > > > I have an update. > > > > > > > > I've started from *CacheConfiguration* permutations. I wrote out a > list > > > > with all CacheConfiguration setters and filtered it with Alexey G. > > > > > > > > Finally we have: > > > > > > > > 1. CacheMode - 3 variants > > > > 2. CacheAtomicityMode - 2 variants > > > > 3. CacheMemoryMode - 3 variants > > > > 4. setLoadPreviousValue - 2 variants > > > > 5. setReadFromBackup - 2 variants > > > > 6. setStoreKeepBinary - 2 variants > > > > 7. setRebalanceMode - SYNC and ASYNC (2 variants) > > > > 8. setSwapEnabled - 2 variants > > > > 9. setCopyOnRead - 2 variants > > > > 10. NearConfiguration disabled / default NearConfiguration / > custom > > > > NearConfiguration - 3 variants > > > > 11. With and without a complex parameter. The complex parameter > > > defines > > > > not-default Eviction policy and filter, cache store configuration > > > > (storeFactory and storeSessionListenerFactory), rebalancing > > > > configuration, > > > > affinity function, offHeapMaxMemory, interceptor, topology > validator > > > and > > > > CacheEntryListener. > > > > > > > > I've run 123 Cache Full Api test cases for all permutations of > > parameters > > > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test > > > > cases). All these tests take 4 hours 40 minutes. Not all tests pass, > so > > > MAY > > > > BE when all tests will pass it will take less time (3,5 hours for > > > example). > > > > > > > > As we can see the tests take a lot of time. > > > > > > > > The following permutation should be supported too: > > > > > > > > 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes > and 1 > > > > backups, 4 nodes and 2 backups - 3 variants > > > > 2. Client and Server nodes - 2 variants > > > > 3. Indexing enabled and disabled for cache - 2 variants > > > > 4. IgniteConfiguration permutations - how many variants? I see at > > > least > > > > 4 (2 Marshallers, P2P). > > > > > > > > Plus we need add new test cases to test different key and value types > > and > > > > etc. > > > > > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will > split > > > all > > > > tests on 250 suites and run on all 30 TC agents it will take about > > 30-40 > > > > hours. Ok, we can do it during weekends. > > > > > > > > I think it will take too much time. > > > > > > > > As an option we can start a cache for each configuration and run > tests > > > > concurrently. But we need to implement this opportunity in our test > > > > framework. > > > > > > > > Any other thoughts how we can decrease time for tests? > > > > > > > > Thanks, > > > > -- Artem -- > > > > > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]> > > > > wrote: > > > > > > > > > Artem, > > > > > > > > > > One more thing for new tests: I think test should start both server > > and > > > > > client nodes and use Ignite API from all nodes. > > > > > > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email] > > > > > > wrote: > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > Actually, I don't have a list with all the permutations. > > > > > > > > > > > > At first, we need to split in our discussion test cases and > Ignite > > > > > > configuration which should be covered. > > > > > > > > > > > > For example, new Full Api test cases for cache are based on old > > Full > > > > Api > > > > > > test cases. So, it need to think what the test cases was not > > covered > > > > > > before. > > > > > > > > > > > > About Ignite configurations, I'm going to add permutation for > each > > > > > > IgniteConfiguration and CacheConfiguration property. > > > > > > > > > > > > By the way, the jira contains the following list of permutation > > (feel > > > > > free > > > > > > to add something): > > > > > > > > > > > > The following tests should be added (for functional blocks): > > > > > > > > > > > > 1. Interceptor > > > > > > 2. Queries: continuous, scan, SQL, fields and text queries. > > > > > > 3. cache events > > > > > > 4. We should also test with Serializable, Externalizable, and > > > plain > > > > > > Pojos for keys and values. > > > > > > 5. The Pojo in the above test should contain an enum value > > > > > > 6. We should also test Enums as keys and Enums as values > > > > > > 7. All operations should have single-key and multi-key > > operations > > > > > > > > > > > > New tests should cover all combinations for following properties: > > > > > > > > > > > > 1. cache modes > > > > > > 2. operation from client nodes and server nodes > > > > > > 3. store enabled/disabled > > > > > > 4. evicts sycn/non-sync > > > > > > 5. eviction policies > > > > > > 6. near on/off > > > > > > 7. marshallers (+ Binary marshaller with different mappers) > > > > > > 8. keys and values - externalizable, serializable, > binaryzable, > > > > "none > > > > > of > > > > > > previous" > > > > > > 9. classes available on servers: true/false > > > > > > 10. Peer loading on/off > > > > > > 11. Affinity functions > > > > > > 12. expiry policies > > > > > > > > > > > > > > > > > > Thanks, > > > > > > -- Artem -- > > > > > > > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan < > > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Artem, I think it is best to specify all the permutations here, > > so > > > > > others > > > > > > > can make additional suggestions. Otherwise, we cannot get a > full > > > > > picture. > > > > > > > > > > > > > > Thanks, > > > > > > > D. > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > I thought a little bit more and think we need to add a > support > > > for > > > > > the > > > > > > > > following permutations too (I've added these to the jira > > > > > description): > > > > > > > > - We should also test with Serializable, Externalizable, and > > > plain > > > > > > Pojos > > > > > > > > for keys and values. > > > > > > > > - The Pojo in the above test should contain an enum value > > > > > > > > - We should also test Enums as keys and Enums as values > > > > > > > > - All operations should have single-key and multi-key > > operations > > > > > > > > > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak < > > > > [hidden email]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > There is a branch at my fork and a pull request at Ignite. > > See > > > > > > comment > > > > > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > > > > > > > > > But I have to notice that the branch under hard development > > and > > > > you > > > > > > it > > > > > > > > can > > > > > > > > > not work (have compilation or test errors) at some moments. > > > > > > > > > > > > > > > > > > Good luck! > > > > > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Artem, > > > > > > > > >> > > > > > > > > >> This is great. I have noticed from the ticket that you > have > > > > > created > > > > > > > some > > > > > > > > >> initial suite already. Is there a branch I can look at it? > > > > > > > > >> > > > > > > > > >> D. > > > > > > > > >> > > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > > > > > [hidden email] > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >> > Igniters, > > > > > > > > >> > > > > > > > > > >> > I'm working on an enhancement of Full API coverage [1] > > [2]. > > > > > > > > >> > > > > > > > > > >> > Ignite already has Full API test, but currently it's > hard > > to > > > > > test > > > > > > > all > > > > > > > > >> > configuration combinations. > > > > > > > > >> > > > > > > > > > >> > Feel free to add comments in the jira if you have any > > > thought. > > > > > > > > >> > > > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > > > > > >> > [2] > > > > > > > > >> > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > > > > > >> > > > > > > > > > >> > Thanks, > > > > > > > > >> > -- Artem -- > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Sergey Kozlov > > > GridGain Systems > > > www.gridgain.com > > > > > > > > > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com > -- Alexey Kuznetsov GridGain Systems www.gridgain.com |
Alexey,
First of all, it's strange to have disk-dependent tests for in-memory data fabric ;) I'm sure we can solve this issue without bottleneck. I see many technical issues with that approach. Another bad thing about the approach is we don't know how to find test result by configuration. I think, if we will split tests it should be done manually: by marshaller, enable/disable indexing, enabled/disabled near configuration and etc. It will be really simple when I will finish with the test framework. In my view, it's not really hard to run tests concurrently. My question is in approach. May be we can another options or we cannot do it at all by any reason. -- Artem -- On Mon, Feb 8, 2016 at 6:38 PM, Alexey Kuznetsov <[hidden email]> wrote: > How about such approach: > > Generate all permutation descriptors and save them to database. > On each agent start some process that will grab from DB some descriptors > (lets say 100) mark them as "in progress" and execute them. > Other agents will grab remaining "not in progress" descriptors and do the > same. > So we could run in parallel all permutations even if we have 1M of them. > > What do you think? > > > On Mon, Feb 8, 2016 at 9:52 PM, Sergey Kozlov <[hidden email]> > wrote: > > > 1000 caches x 50MB = 50GB heap. Do we really have >50GB RAM on each > agents? > > > > On Mon, Feb 8, 2016 at 5:45 PM, Artem Shutak <[hidden email]> > wrote: > > > > > Sergey, > > > > > > I think we should start more caches, like 1000 in one time. But we have > > to > > > have enough memory on our TC agents. As I know, empty cache is require > > > about 50 mb (without indexing), am I right? > > > > > > You are right, I keep in mind that *backups* and *REPLICATED* mode make > > no > > > sense together, but we still have to test it in one node and multi node > > > casesю > > > > > > Any other *no sense* combinations? > > > > > > I forgot about custom BinaryConfiguration at IgniteConfiguration for > > > BinaryMarshaller. So, at least 6 IgniteConfigurations. > > > > > > -- Artem -- > > > > > > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]> > > > wrote: > > > > > > > Hi Artem > > > > > > > > It's good idea to create 20-30 cache configurations at once and then > to > > > > iterate tests over those caches in parallel (but make sure that cache > > > names > > > > are unique). > > > > Another point that some combinations make no sense like *backups *and > > > > *REPLICATED > > > > *cache > > > > > > > > > > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I have an update. > > > > > > > > > > I've started from *CacheConfiguration* permutations. I wrote out a > > list > > > > > with all CacheConfiguration setters and filtered it with Alexey G. > > > > > > > > > > Finally we have: > > > > > > > > > > 1. CacheMode - 3 variants > > > > > 2. CacheAtomicityMode - 2 variants > > > > > 3. CacheMemoryMode - 3 variants > > > > > 4. setLoadPreviousValue - 2 variants > > > > > 5. setReadFromBackup - 2 variants > > > > > 6. setStoreKeepBinary - 2 variants > > > > > 7. setRebalanceMode - SYNC and ASYNC (2 variants) > > > > > 8. setSwapEnabled - 2 variants > > > > > 9. setCopyOnRead - 2 variants > > > > > 10. NearConfiguration disabled / default NearConfiguration / > > custom > > > > > NearConfiguration - 3 variants > > > > > 11. With and without a complex parameter. The complex parameter > > > > defines > > > > > not-default Eviction policy and filter, cache store > configuration > > > > > (storeFactory and storeSessionListenerFactory), rebalancing > > > > > configuration, > > > > > affinity function, offHeapMaxMemory, interceptor, topology > > validator > > > > and > > > > > CacheEntryListener. > > > > > > > > > > I've run 123 Cache Full Api test cases for all permutations of > > > parameters > > > > > 1-9 and got 256896 test cases (1152 configuration variants * 123 > test > > > > > cases). All these tests take 4 hours 40 minutes. Not all tests > pass, > > so > > > > MAY > > > > > BE when all tests will pass it will take less time (3,5 hours for > > > > example). > > > > > > > > > > As we can see the tests take a lot of time. > > > > > > > > > > The following permutation should be supported too: > > > > > > > > > > 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes > > and 1 > > > > > backups, 4 nodes and 2 backups - 3 variants > > > > > 2. Client and Server nodes - 2 variants > > > > > 3. Indexing enabled and disabled for cache - 2 variants > > > > > 4. IgniteConfiguration permutations - how many variants? I see > at > > > > least > > > > > 4 (2 Marshallers, P2P). > > > > > > > > > > Plus we need add new test cases to test different key and value > types > > > and > > > > > etc. > > > > > > > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will > > split > > > > all > > > > > tests on 250 suites and run on all 30 TC agents it will take about > > > 30-40 > > > > > hours. Ok, we can do it during weekends. > > > > > > > > > > I think it will take too much time. > > > > > > > > > > As an option we can start a cache for each configuration and run > > tests > > > > > concurrently. But we need to implement this opportunity in our test > > > > > framework. > > > > > > > > > > Any other thoughts how we can decrease time for tests? > > > > > > > > > > Thanks, > > > > > -- Artem -- > > > > > > > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov < > [hidden email]> > > > > > wrote: > > > > > > > > > > > Artem, > > > > > > > > > > > > One more thing for new tests: I think test should start both > server > > > and > > > > > > client nodes and use Ignite API from all nodes. > > > > > > > > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > Actually, I don't have a list with all the permutations. > > > > > > > > > > > > > > At first, we need to split in our discussion test cases and > > Ignite > > > > > > > configuration which should be covered. > > > > > > > > > > > > > > For example, new Full Api test cases for cache are based on old > > > Full > > > > > Api > > > > > > > test cases. So, it need to think what the test cases was not > > > covered > > > > > > > before. > > > > > > > > > > > > > > About Ignite configurations, I'm going to add permutation for > > each > > > > > > > IgniteConfiguration and CacheConfiguration property. > > > > > > > > > > > > > > By the way, the jira contains the following list of permutation > > > (feel > > > > > > free > > > > > > > to add something): > > > > > > > > > > > > > > The following tests should be added (for functional blocks): > > > > > > > > > > > > > > 1. Interceptor > > > > > > > 2. Queries: continuous, scan, SQL, fields and text queries. > > > > > > > 3. cache events > > > > > > > 4. We should also test with Serializable, Externalizable, > and > > > > plain > > > > > > > Pojos for keys and values. > > > > > > > 5. The Pojo in the above test should contain an enum value > > > > > > > 6. We should also test Enums as keys and Enums as values > > > > > > > 7. All operations should have single-key and multi-key > > > operations > > > > > > > > > > > > > > New tests should cover all combinations for following > properties: > > > > > > > > > > > > > > 1. cache modes > > > > > > > 2. operation from client nodes and server nodes > > > > > > > 3. store enabled/disabled > > > > > > > 4. evicts sycn/non-sync > > > > > > > 5. eviction policies > > > > > > > 6. near on/off > > > > > > > 7. marshallers (+ Binary marshaller with different mappers) > > > > > > > 8. keys and values - externalizable, serializable, > > binaryzable, > > > > > "none > > > > > > of > > > > > > > previous" > > > > > > > 9. classes available on servers: true/false > > > > > > > 10. Peer loading on/off > > > > > > > 11. Affinity functions > > > > > > > 12. expiry policies > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > -- Artem -- > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan < > > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Artem, I think it is best to specify all the permutations > here, > > > so > > > > > > others > > > > > > > > can make additional suggestions. Otherwise, we cannot get a > > full > > > > > > picture. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > D. > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > I thought a little bit more and think we need to add a > > support > > > > for > > > > > > the > > > > > > > > > following permutations too (I've added these to the jira > > > > > > description): > > > > > > > > > - We should also test with Serializable, Externalizable, > and > > > > plain > > > > > > > Pojos > > > > > > > > > for keys and values. > > > > > > > > > - The Pojo in the above test should contain an enum value > > > > > > > > > - We should also test Enums as keys and Enums as values > > > > > > > > > - All operations should have single-key and multi-key > > > operations > > > > > > > > > > > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak < > > > > > [hidden email]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > There is a branch at my fork and a pull request at > Ignite. > > > See > > > > > > > comment > > > > > > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > > > > > > > > > > > But I have to notice that the branch under hard > development > > > and > > > > > you > > > > > > > it > > > > > > > > > can > > > > > > > > > > not work (have compilation or test errors) at some > moments. > > > > > > > > > > > > > > > > > > > > Good luck! > > > > > > > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Artem, > > > > > > > > > >> > > > > > > > > > >> This is great. I have noticed from the ticket that you > > have > > > > > > created > > > > > > > > some > > > > > > > > > >> initial suite already. Is there a branch I can look at > it? > > > > > > > > > >> > > > > > > > > > >> D. > > > > > > > > > >> > > > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > >> > Igniters, > > > > > > > > > >> > > > > > > > > > > >> > I'm working on an enhancement of Full API coverage [1] > > > [2]. > > > > > > > > > >> > > > > > > > > > > >> > Ignite already has Full API test, but currently it's > > hard > > > to > > > > > > test > > > > > > > > all > > > > > > > > > >> > configuration combinations. > > > > > > > > > >> > > > > > > > > > > >> > Feel free to add comments in the jira if you have any > > > > thought. > > > > > > > > > >> > > > > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > > > > > > >> > [2] > > > > > > > > > >> > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > > > > > > >> > > > > > > > > > > >> > Thanks, > > > > > > > > > >> > -- Artem -- > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Sergey Kozlov > > > > GridGain Systems > > > > www.gridgain.com > > > > > > > > > > > > > > > -- > > Sergey Kozlov > > GridGain Systems > > www.gridgain.com > > > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com > |
In reply to this post by Artem Shutak
50MB for an empty cache! That's a lot of emptiness!
The Ignite team's recommendation to have one key/value type per cache now seems a little more dubious and difficult to follow in practice. Just curious, what is the reason for such waste? Is it because by default the initial cache size is set to 1.5M entries? Thanks Andrey > Date: Mon, 8 Feb 2016 17:45:33 +0300 > Subject: Re: Full API coverage enhancement > From: [hidden email] > To: [hidden email] > > Sergey, > > I think we should start more caches, like 1000 in one time. But we have to > have enough memory on our TC agents. As I know, empty cache is require > about 50 mb (without indexing), am I right? > > You are right, I keep in mind that *backups* and *REPLICATED* mode make no > sense together, but we still have to test it in one node and multi node > casesю > > Any other *no sense* combinations? > > I forgot about custom BinaryConfiguration at IgniteConfiguration for > BinaryMarshaller. So, at least 6 IgniteConfigurations. > > -- Artem -- > > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]> wrote: > > > Hi Artem > > > > It's good idea to create 20-30 cache configurations at once and then to > > iterate tests over those caches in parallel (but make sure that cache names > > are unique). > > Another point that some combinations make no sense like *backups *and > > *REPLICATED > > *cache > > > > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> wrote: > > > > > Hi all, > > > > > > I have an update. > > > > > > I've started from *CacheConfiguration* permutations. I wrote out a list > > > with all CacheConfiguration setters and filtered it with Alexey G. > > > > > > Finally we have: > > > > > > 1. CacheMode - 3 variants > > > 2. CacheAtomicityMode - 2 variants > > > 3. CacheMemoryMode - 3 variants > > > 4. setLoadPreviousValue - 2 variants > > > 5. setReadFromBackup - 2 variants > > > 6. setStoreKeepBinary - 2 variants > > > 7. setRebalanceMode - SYNC and ASYNC (2 variants) > > > 8. setSwapEnabled - 2 variants > > > 9. setCopyOnRead - 2 variants > > > 10. NearConfiguration disabled / default NearConfiguration / custom > > > NearConfiguration - 3 variants > > > 11. With and without a complex parameter. The complex parameter > > defines > > > not-default Eviction policy and filter, cache store configuration > > > (storeFactory and storeSessionListenerFactory), rebalancing > > > configuration, > > > affinity function, offHeapMaxMemory, interceptor, topology validator > > and > > > CacheEntryListener. > > > > > > I've run 123 Cache Full Api test cases for all permutations of parameters > > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test > > > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so > > MAY > > > BE when all tests will pass it will take less time (3,5 hours for > > example). > > > > > > As we can see the tests take a lot of time. > > > > > > The following permutation should be supported too: > > > > > > 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1 > > > backups, 4 nodes and 2 backups - 3 variants > > > 2. Client and Server nodes - 2 variants > > > 3. Indexing enabled and disabled for cache - 2 variants > > > 4. IgniteConfiguration permutations - how many variants? I see at > > least > > > 4 (2 Marshallers, P2P). > > > > > > Plus we need add new test cases to test different key and value types and > > > etc. > > > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split > > all > > > tests on 250 suites and run on all 30 TC agents it will take about 30-40 > > > hours. Ok, we can do it during weekends. > > > > > > I think it will take too much time. > > > > > > As an option we can start a cache for each configuration and run tests > > > concurrently. But we need to implement this opportunity in our test > > > framework. > > > > > > Any other thoughts how we can decrease time for tests? > > > > > > Thanks, > > > -- Artem -- > > > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]> > > > wrote: > > > > > > > Artem, > > > > > > > > One more thing for new tests: I think test should start both server and > > > > client nodes and use Ignite API from all nodes. > > > > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email]> > > > wrote: > > > > > > > > > Dmitriy, > > > > > > > > > > Actually, I don't have a list with all the permutations. > > > > > > > > > > At first, we need to split in our discussion test cases and Ignite > > > > > configuration which should be covered. > > > > > > > > > > For example, new Full Api test cases for cache are based on old Full > > > Api > > > > > test cases. So, it need to think what the test cases was not covered > > > > > before. > > > > > > > > > > About Ignite configurations, I'm going to add permutation for each > > > > > IgniteConfiguration and CacheConfiguration property. > > > > > > > > > > By the way, the jira contains the following list of permutation (feel > > > > free > > > > > to add something): > > > > > > > > > > The following tests should be added (for functional blocks): > > > > > > > > > > 1. Interceptor > > > > > 2. Queries: continuous, scan, SQL, fields and text queries. > > > > > 3. cache events > > > > > 4. We should also test with Serializable, Externalizable, and > > plain > > > > > Pojos for keys and values. > > > > > 5. The Pojo in the above test should contain an enum value > > > > > 6. We should also test Enums as keys and Enums as values > > > > > 7. All operations should have single-key and multi-key operations > > > > > > > > > > New tests should cover all combinations for following properties: > > > > > > > > > > 1. cache modes > > > > > 2. operation from client nodes and server nodes > > > > > 3. store enabled/disabled > > > > > 4. evicts sycn/non-sync > > > > > 5. eviction policies > > > > > 6. near on/off > > > > > 7. marshallers (+ Binary marshaller with different mappers) > > > > > 8. keys and values - externalizable, serializable, binaryzable, > > > "none > > > > of > > > > > previous" > > > > > 9. classes available on servers: true/false > > > > > 10. Peer loading on/off > > > > > 11. Affinity functions > > > > > 12. expiry policies > > > > > > > > > > > > > > > Thanks, > > > > > -- Artem -- > > > > > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan < > > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > Artem, I think it is best to specify all the permutations here, so > > > > others > > > > > > can make additional suggestions. Otherwise, we cannot get a full > > > > picture. > > > > > > > > > > > > Thanks, > > > > > > D. > > > > > > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <[hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > I thought a little bit more and think we need to add a support > > for > > > > the > > > > > > > following permutations too (I've added these to the jira > > > > description): > > > > > > > - We should also test with Serializable, Externalizable, and > > plain > > > > > Pojos > > > > > > > for keys and values. > > > > > > > - The Pojo in the above test should contain an enum value > > > > > > > - We should also test Enums as keys and Enums as values > > > > > > > - All operations should have single-key and multi-key operations > > > > > > > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak < > > > [hidden email]> > > > > > > > wrote: > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > There is a branch at my fork and a pull request at Ignite. See > > > > > comment > > > > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > > > > > > > But I have to notice that the branch under hard development and > > > you > > > > > it > > > > > > > can > > > > > > > > not work (have compilation or test errors) at some moments. > > > > > > > > > > > > > > > > Good luck! > > > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> Artem, > > > > > > > >> > > > > > > > >> This is great. I have noticed from the ticket that you have > > > > created > > > > > > some > > > > > > > >> initial suite already. Is there a branch I can look at it? > > > > > > > >> > > > > > > > >> D. > > > > > > > >> > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > > > > [hidden email] > > > > > > > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > Igniters, > > > > > > > >> > > > > > > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > > > > > > >> > > > > > > > > >> > Ignite already has Full API test, but currently it's hard to > > > > test > > > > > > all > > > > > > > >> > configuration combinations. > > > > > > > >> > > > > > > > > >> > Feel free to add comments in the jira if you have any > > thought. > > > > > > > >> > > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > > > > >> > [2] > > > > > > > >> > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > > > > >> > > > > > > > > >> > Thanks, > > > > > > > >> > -- Artem -- > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Sergey Kozlov > > GridGain Systems > > www.gridgain.com > > |
Andrey, I think the reason for the waste is some sort of internal
ring-buffer allocation which can be turned off. Alexey G, can you please comment? D. On Mon, Feb 8, 2016 at 10:21 AM, Andrey Kornev <[hidden email]> wrote: > 50MB for an empty cache! That's a lot of emptiness! > > The Ignite team's recommendation to have one key/value type per cache now > seems a little more dubious and difficult to follow in practice. > > Just curious, what is the reason for such waste? Is it because by default > the initial cache size is set to 1.5M entries? > > Thanks > Andrey > > > Date: Mon, 8 Feb 2016 17:45:33 +0300 > > Subject: Re: Full API coverage enhancement > > From: [hidden email] > > To: [hidden email] > > > > Sergey, > > > > I think we should start more caches, like 1000 in one time. But we have > to > > have enough memory on our TC agents. As I know, empty cache is require > > about 50 mb (without indexing), am I right? > > > > You are right, I keep in mind that *backups* and *REPLICATED* mode make > no > > sense together, but we still have to test it in one node and multi node > > casesю > > > > Any other *no sense* combinations? > > > > I forgot about custom BinaryConfiguration at IgniteConfiguration for > > BinaryMarshaller. So, at least 6 IgniteConfigurations. > > > > -- Artem -- > > > > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <[hidden email]> > wrote: > > > > > Hi Artem > > > > > > It's good idea to create 20-30 cache configurations at once and then to > > > iterate tests over those caches in parallel (but make sure that cache > names > > > are unique). > > > Another point that some combinations make no sense like *backups *and > > > *REPLICATED > > > *cache > > > > > > > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <[hidden email]> > wrote: > > > > > > > Hi all, > > > > > > > > I have an update. > > > > > > > > I've started from *CacheConfiguration* permutations. I wrote out a > list > > > > with all CacheConfiguration setters and filtered it with Alexey G. > > > > > > > > Finally we have: > > > > > > > > 1. CacheMode - 3 variants > > > > 2. CacheAtomicityMode - 2 variants > > > > 3. CacheMemoryMode - 3 variants > > > > 4. setLoadPreviousValue - 2 variants > > > > 5. setReadFromBackup - 2 variants > > > > 6. setStoreKeepBinary - 2 variants > > > > 7. setRebalanceMode - SYNC and ASYNC (2 variants) > > > > 8. setSwapEnabled - 2 variants > > > > 9. setCopyOnRead - 2 variants > > > > 10. NearConfiguration disabled / default NearConfiguration / > custom > > > > NearConfiguration - 3 variants > > > > 11. With and without a complex parameter. The complex parameter > > > defines > > > > not-default Eviction policy and filter, cache store configuration > > > > (storeFactory and storeSessionListenerFactory), rebalancing > > > > configuration, > > > > affinity function, offHeapMaxMemory, interceptor, topology > validator > > > and > > > > CacheEntryListener. > > > > > > > > I've run 123 Cache Full Api test cases for all permutations of > parameters > > > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test > > > > cases). All these tests take 4 hours 40 minutes. Not all tests pass, > so > > > MAY > > > > BE when all tests will pass it will take less time (3,5 hours for > > > example). > > > > > > > > As we can see the tests take a lot of time. > > > > > > > > The following permutation should be supported too: > > > > > > > > 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes > and 1 > > > > backups, 4 nodes and 2 backups - 3 variants > > > > 2. Client and Server nodes - 2 variants > > > > 3. Indexing enabled and disabled for cache - 2 variants > > > > 4. IgniteConfiguration permutations - how many variants? I see at > > > least > > > > 4 (2 Marshallers, P2P). > > > > > > > > Plus we need add new test cases to test different key and value > types and > > > > etc. > > > > > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will > split > > > all > > > > tests on 250 suites and run on all 30 TC agents it will take about > 30-40 > > > > hours. Ok, we can do it during weekends. > > > > > > > > I think it will take too much time. > > > > > > > > As an option we can start a cache for each configuration and run > tests > > > > concurrently. But we need to implement this opportunity in our test > > > > framework. > > > > > > > > Any other thoughts how we can decrease time for tests? > > > > > > > > Thanks, > > > > -- Artem -- > > > > > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <[hidden email]> > > > > wrote: > > > > > > > > > Artem, > > > > > > > > > > One more thing for new tests: I think test should start both > server and > > > > > client nodes and use Ignite API from all nodes. > > > > > > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <[hidden email] > > > > > > wrote: > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > Actually, I don't have a list with all the permutations. > > > > > > > > > > > > At first, we need to split in our discussion test cases and > Ignite > > > > > > configuration which should be covered. > > > > > > > > > > > > For example, new Full Api test cases for cache are based on old > Full > > > > Api > > > > > > test cases. So, it need to think what the test cases was not > covered > > > > > > before. > > > > > > > > > > > > About Ignite configurations, I'm going to add permutation for > each > > > > > > IgniteConfiguration and CacheConfiguration property. > > > > > > > > > > > > By the way, the jira contains the following list of permutation > (feel > > > > > free > > > > > > to add something): > > > > > > > > > > > > The following tests should be added (for functional blocks): > > > > > > > > > > > > 1. Interceptor > > > > > > 2. Queries: continuous, scan, SQL, fields and text queries. > > > > > > 3. cache events > > > > > > 4. We should also test with Serializable, Externalizable, and > > > plain > > > > > > Pojos for keys and values. > > > > > > 5. The Pojo in the above test should contain an enum value > > > > > > 6. We should also test Enums as keys and Enums as values > > > > > > 7. All operations should have single-key and multi-key > operations > > > > > > > > > > > > New tests should cover all combinations for following properties: > > > > > > > > > > > > 1. cache modes > > > > > > 2. operation from client nodes and server nodes > > > > > > 3. store enabled/disabled > > > > > > 4. evicts sycn/non-sync > > > > > > 5. eviction policies > > > > > > 6. near on/off > > > > > > 7. marshallers (+ Binary marshaller with different mappers) > > > > > > 8. keys and values - externalizable, serializable, > binaryzable, > > > > "none > > > > > of > > > > > > previous" > > > > > > 9. classes available on servers: true/false > > > > > > 10. Peer loading on/off > > > > > > 11. Affinity functions > > > > > > 12. expiry policies > > > > > > > > > > > > > > > > > > Thanks, > > > > > > -- Artem -- > > > > > > > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan < > > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Artem, I think it is best to specify all the permutations > here, so > > > > > others > > > > > > > can make additional suggestions. Otherwise, we cannot get a > full > > > > > picture. > > > > > > > > > > > > > > Thanks, > > > > > > > D. > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak < > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > I thought a little bit more and think we need to add a > support > > > for > > > > > the > > > > > > > > following permutations too (I've added these to the jira > > > > > description): > > > > > > > > - We should also test with Serializable, Externalizable, and > > > plain > > > > > > Pojos > > > > > > > > for keys and values. > > > > > > > > - The Pojo in the above test should contain an enum value > > > > > > > > - We should also test Enums as keys and Enums as values > > > > > > > > - All operations should have single-key and multi-key > operations > > > > > > > > > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak < > > > > [hidden email]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > There is a branch at my fork and a pull request at Ignite. > See > > > > > > comment > > > > > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > > > > > > > > > But I have to notice that the branch under hard > development and > > > > you > > > > > > it > > > > > > > > can > > > > > > > > > not work (have compilation or test errors) at some moments. > > > > > > > > > > > > > > > > > > Good luck! > > > > > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Artem, > > > > > > > > >> > > > > > > > > >> This is great. I have noticed from the ticket that you > have > > > > > created > > > > > > > some > > > > > > > > >> initial suite already. Is there a branch I can look at it? > > > > > > > > >> > > > > > > > > >> D. > > > > > > > > >> > > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > > > > > [hidden email] > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >> > Igniters, > > > > > > > > >> > > > > > > > > > >> > I'm working on an enhancement of Full API coverage [1] > [2]. > > > > > > > > >> > > > > > > > > > >> > Ignite already has Full API test, but currently it's > hard to > > > > > test > > > > > > > all > > > > > > > > >> > configuration combinations. > > > > > > > > >> > > > > > > > > > >> > Feel free to add comments in the jira if you have any > > > thought. > > > > > > > > >> > > > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > > > > > >> > [2] > > > > > > > > >> > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > > > > > >> > > > > > > > > > >> > Thanks, > > > > > > > > >> > -- Artem -- > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Sergey Kozlov > > > GridGain Systems > > > www.gridgain.com > > > > > |
Dmitriy, the size of the circular buffer can be decreased by setting
IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE system property. I was wondering what our next steps will be after implementing the framework. From the initial discussion I thought we want to convert some older tests to this new framework and run all new tests using this framework. However, from what Artem already has written, this sounds unrealistic because adding even a simple test case will induce hours of run time. I think we still need to separate more important and less important configurations, run important ones on a daily basis, and use all others for after-code-freeze runs, for example. Thoughts? |
Agree about separation. I think we need to define some internal
permutations that keep coming up with bugs. I can start the list here: 1. Serializable, Externalizable, neither. 1. We should run the whole suite 3 times, once for each serialization mode. Having 2 or 3 nodes in the cluster here should be enough. No need to test it on the changing cluster. 2. On-heap and off-heap with entry processor and peer-deployment 1. on-heap, entry-processor, peer-deployment=true/false 2. off-heap, entry-processor, peer-deployment=true/false 3. On-heap and off-heap with eviction policies 1. eviction policy, memory-mode=on-heap 2. eviction policy, memory-mode=off-heap 3. eviction policy, memory-mode=off-heap-values 4. On-heap and off-heap with continuous queries and peer-deployment=true/false 5. … I think we can come up with about 20 most important permutations. Thoughts? D. On Wed, Feb 10, 2016 at 9:24 AM, Alexey Goncharuk < [hidden email]> wrote: > Dmitriy, the size of the circular buffer can be decreased by setting > IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE system property. > > I was wondering what our next steps will be after implementing the > framework. From the initial discussion I thought we want to convert some > older tests to this new framework and run all new tests using this > framework. However, from what Artem already has written, this sounds > unrealistic because adding even a simple test case will induce hours of run > time. I think we still need to separate more important and less important > configurations, run important ones on a daily basis, and use all others for > after-code-freeze runs, for example. > > Thoughts? > |
Igniters,
I'm glad to announce that Configuration Variations test framework has been merged to master and is ready to use. The framework provides an opportunity to write a test-case once and to run it in many Ignite and Cache configuration variations, for different Ignite topologies (with and without clients) and execute the same test scenario from different Ignite nodes (server and client) automatically. Another useful feature is "runInAllDataModes" functionality that allows to run test-case and run it in all supported by framework data modes (like Serializable, Externalizable, plane, mixed and etc. objects). I've added the framework description on "Implementing Tests" [1] page. Please, feel free to comment. As proof-of-concept I've added IgniteCacheBasicConfigVariationsFullApiTestSuite and add it on TC. It does not mean that the work on IGNITE-2521 <https://issues.apache.org/jira/browse/IGNITE-2521> is completed and we have coverage for full Ignite API, but I think it is a good improvement for Ignite's test framework and Igniters can test functionality better and simpler. [1] https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests (see "Configuration variations test framework" section). Thanks, -- Artem -- On Thu, Feb 11, 2016 at 12:29 AM, Dmitriy Setrakyan <[hidden email]> wrote: > Agree about separation. I think we need to define some internal > permutations that keep coming up with bugs. I can start the list here: > > 1. Serializable, Externalizable, neither. > 1. We should run the whole suite 3 times, once for each serialization > mode. Having 2 or 3 nodes in the cluster here should be enough. > No need to > test it on the changing cluster. > 2. On-heap and off-heap with entry processor and peer-deployment > 1. on-heap, entry-processor, peer-deployment=true/false > 2. off-heap, entry-processor, peer-deployment=true/false > 3. On-heap and off-heap with eviction policies > 1. eviction policy, memory-mode=on-heap > 2. eviction policy, memory-mode=off-heap > 3. eviction policy, memory-mode=off-heap-values > 4. On-heap and off-heap with continuous queries and > peer-deployment=true/false > 5. … > > I think we can come up with about 20 most important permutations. Thoughts? > > D. > > > On Wed, Feb 10, 2016 at 9:24 AM, Alexey Goncharuk < > [hidden email]> wrote: > > > Dmitriy, the size of the circular buffer can be decreased by setting > > IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE system property. > > > > I was wondering what our next steps will be after implementing the > > framework. From the initial discussion I thought we want to convert some > > older tests to this new framework and run all new tests using this > > framework. However, from what Artem already has written, this sounds > > unrealistic because adding even a simple test case will induce hours of > run > > time. I think we still need to separate more important and less important > > configurations, run important ones on a daily basis, and use all others > for > > after-code-freeze runs, for example. > > > > Thoughts? > > > |
Free forum by Nabble | Edit this page |