Ignite-1.5 Release

[Vladimir Ozerov]

When I look into profiler, I see that actual work with portables takes only
relatively small fraction of time. The only significant hotspot I saw was
query parsing, but we already discussed this in another topic and Sergi
created a ticket.

To improve performance even further, we need to start working on
microoptimizations, because I see that query execution produces loooots of
garbage due to dozens of wrappers, primitives boxing, etc.. Something comes
form portables, something comes from indexing. I do not think that working
solely on portables can give us a breakthrough in performance.

On Thu, Nov 5, 2015 at 12:50 AM, Dmitriy Setrakyan <[hidden email]>
wrote:

> On Wed, Nov 4, 2015 at 10:25 AM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> >
> > Also I measured query performance on some local benchmarks and got
> > acceptable resutls - queries are about 5-7% slower with poratbles than
> with
> > OptimizedMarshaller. Looks very promising to me provided that we work
> with
> > deserialized objects now.
>
>
> Vladimir, I don’t think we can treat these results as acceptable. So far,
> Ignite has been doing very well on all competitive benchmarks, and we
> cannot afford to start loosing any of them.
>
> Now, I remember seeing emails about many more performance optimizations we
> can add, like aligning String representation with binary representation,
> etc. Do you think after adding all the optimizations we will still be
> slower or faster?
>
> D.
>

[Vladimir Ozerov]

Igniters,

One more thing - I found a ticket *IGNITE-1377 *where portable metadata
update could cause hangs in cache. As we are planning to have
PortableMarshaller as default one, looks like this ticket is better to be
fixed before the release.

On Thu, Nov 5, 2015 at 3:30 PM, Vladimir Ozerov <[hidden email]>
wrote:

> When I look into profiler, I see that actual work with portables takes
> only relatively small fraction of time. The only significant hotspot I saw
> was query parsing, but we already discussed this in another topic and Sergi
> created a ticket.
>
> To improve performance even further, we need to start working on
> microoptimizations, because I see that query execution produces loooots of
> garbage due to dozens of wrappers, primitives boxing, etc.. Something comes
> form portables, something comes from indexing. I do not think that working
> solely on portables can give us a breakthrough in performance.
>
> On Thu, Nov 5, 2015 at 12:50 AM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
>> On Wed, Nov 4, 2015 at 10:25 AM, Vladimir Ozerov <[hidden email]>
>> wrote:
>>
>> >
>> > Also I measured query performance on some local benchmarks and got
>> > acceptable resutls - queries are about 5-7% slower with poratbles than
>> with
>> > OptimizedMarshaller. Looks very promising to me provided that we work
>> with
>> > deserialized objects now.
>>
>>
>> Vladimir, I don’t think we can treat these results as acceptable. So far,
>> Ignite has been doing very well on all competitive benchmarks, and we
>> cannot afford to start loosing any of them.
>>
>> Now, I remember seeing emails about many more performance optimizations we
>> can add, like aligning String representation with binary representation,
>> etc. Do you think after adding all the optimizations we will still be
>> slower or faster?
>>
>> D.
>>
>
>

[Alexey Goncharuk]

I think it's a critical issue; good thing is that the ticket already
contains a proper description, so at least it is clear what to fix. I'll
try to take a look at this over the weekend.

2015-11-05 15:45 GMT+03:00 Vladimir Ozerov <[hidden email]>:

> Igniters,
>
> One more thing - I found a ticket *IGNITE-1377 *where portable metadata
> update could cause hangs in cache. As we are planning to have
> PortableMarshaller as default one, looks like this ticket is better to be
> fixed before the release.
>
> On Thu, Nov 5, 2015 at 3:30 PM, Vladimir Ozerov <[hidden email]>
> wrote:
>
> > When I look into profiler, I see that actual work with portables takes
> > only relatively small fraction of time. The only significant hotspot I
> saw
> > was query parsing, but we already discussed this in another topic and
> Sergi
> > created a ticket.
> >
> > To improve performance even further, we need to start working on
> > microoptimizations, because I see that query execution produces loooots
> of
> > garbage due to dozens of wrappers, primitives boxing, etc.. Something
> comes
> > form portables, something comes from indexing. I do not think that
> working
> > solely on portables can give us a breakthrough in performance.
> >
> > On Thu, Nov 5, 2015 at 12:50 AM, Dmitriy Setrakyan <
> [hidden email]>
> > wrote:
> >
> >> On Wed, Nov 4, 2015 at 10:25 AM, Vladimir Ozerov <[hidden email]>
> >> wrote:
> >>
> >> >
> >> > Also I measured query performance on some local benchmarks and got
> >> > acceptable resutls - queries are about 5-7% slower with poratbles than
> >> with
> >> > OptimizedMarshaller. Looks very promising to me provided that we work
> >> with
> >> > deserialized objects now.
> >>
> >>
> >> Vladimir, I don’t think we can treat these results as acceptable. So
> far,
> >> Ignite has been doing very well on all competitive benchmarks, and we
> >> cannot afford to start loosing any of them.
> >>
> >> Now, I remember seeing emails about many more performance optimizations
> we
> >> can add, like aligning String representation with binary representation,
> >> etc. Do you think after adding all the optimizations we will still be
> >> slower or faster?
> >>
> >> D.
> >>
> >
> >
>

[Alexey Goncharuk]

Igniters,

IGNITE-950 branch with the latest API changes was merged to the ignite-1282
branch. There is still a couple of pending changes related to configuration
(these changes must be made after the merge) and performance, but I think
now it is a good moment for the community to review and leave feedback to
the proposed changes.

Looking forward for the comments :)

2015-11-05 16:11 GMT+03:00 Alexey Goncharuk <[hidden email]>:

> I think it's a critical issue; good thing is that the ticket already
> contains a proper description, so at least it is clear what to fix. I'll
> try to take a look at this over the weekend.
>
> 2015-11-05 15:45 GMT+03:00 Vladimir Ozerov <[hidden email]>:
>
>> Igniters,
>>
>> One more thing - I found a ticket *IGNITE-1377 *where portable metadata
>> update could cause hangs in cache. As we are planning to have
>> PortableMarshaller as default one, looks like this ticket is better to be
>> fixed before the release.
>>
>> On Thu, Nov 5, 2015 at 3:30 PM, Vladimir Ozerov <[hidden email]>
>> wrote:
>>
>> > When I look into profiler, I see that actual work with portables takes
>> > only relatively small fraction of time. The only significant hotspot I
>> saw
>> > was query parsing, but we already discussed this in another topic and
>> Sergi
>> > created a ticket.
>> >
>> > To improve performance even further, we need to start working on
>> > microoptimizations, because I see that query execution produces loooots
>> of
>> > garbage due to dozens of wrappers, primitives boxing, etc.. Something
>> comes
>> > form portables, something comes from indexing. I do not think that
>> working
>> > solely on portables can give us a breakthrough in performance.
>> >
>> > On Thu, Nov 5, 2015 at 12:50 AM, Dmitriy Setrakyan <
>> [hidden email]>
>> > wrote:
>> >
>> >> On Wed, Nov 4, 2015 at 10:25 AM, Vladimir Ozerov <[hidden email]
>> >
>> >> wrote:
>> >>
>> >> >
>> >> > Also I measured query performance on some local benchmarks and got
>> >> > acceptable resutls - queries are about 5-7% slower with poratbles
>> than
>> >> with
>> >> > OptimizedMarshaller. Looks very promising to me provided that we work
>> >> with
>> >> > deserialized objects now.
>> >>
>> >>
>> >> Vladimir, I don’t think we can treat these results as acceptable. So
>> far,
>> >> Ignite has been doing very well on all competitive benchmarks, and we
>> >> cannot afford to start loosing any of them.
>> >>
>> >> Now, I remember seeing emails about many more performance
>> optimizations we
>> >> can add, like aligning String representation with binary
>> representation,
>> >> etc. Do you think after adding all the optimizations we will still be
>> >> slower or faster?
>> >>
>> >> D.
>> >>
>> >
>> >
>>
>
>

[Semyon Boikov]

I merged into 1.5 performance optimizations implemented by Yakov. With
these optimizations we got up to 20% throughput increase in transactional
cache benchmarks.

On Mon, Nov 2, 2015 at 4:35 PM, Yakov Zhdanov <[hidden email]> wrote:

> Guys,
>
> I think we can start preparation to Ignite-1.5 release which will include
> many interesting features:
>
> 1. Portable object API
> https://issues.apache.org/jira/browse/IGNITE-1486
>
> 2. Ignite.NET and Ignite C++
> https://issues.apache.org/jira/browse/IGNITE-1282
>
> 3. Optimistic serializable transactions
> https://issues.apache.org/jira/browse/IGNITE-1607
>
> 4. Distributed SQL joins - we will be able to query non-collocated data as
> well
> https://issues.apache.org/jira/browse/IGNITE-1232
>
> 5. Enhanced Oracle and IBM JDK interoperability
> https://issues.apache.org/jira/browse/IGNITE-1526
>
> 6. MQTT streamer
> https://issues.apache.org/jira/browse/IGNITE-535
>
> 7. Continuous query failover
> https://issues.apache.org/jira/browse/IGNITE-426
>
> 8. Significant transactional cache performance optimizations - I will merge
> these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
>
> 9. Many stability and fault-tolerance fixes.
>
> 10. I would also like to include distributed Semaphore. Vladislav, any
> chance you can finish with it this week?
> https://issues.apache.org/jira/browse/IGNITE-
> <https://issues.apache.org/jira/browse/IGNITE-426>638
>
> Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
> please respond to this email and let us know when can we expect your
> changes being merged to master and release branch?
>
> Can someone create ignite-1.5 release branch?
>
> --Yakov
>

[dsetrakyan]

Semyon,

Do you know what kind of Garbage Collector you have used for the
benchmarks? Was it G1?

D.

On Thu, Nov 5, 2015 at 5:55 AM, Semyon Boikov <[hidden email]> wrote:

> I merged into 1.5 performance optimizations implemented by Yakov. With
> these optimizations we got up to 20% throughput increase in transactional
> cache benchmarks.
>
> On Mon, Nov 2, 2015 at 4:35 PM, Yakov Zhdanov <[hidden email]> wrote:
>
> > Guys,
> >
> > I think we can start preparation to Ignite-1.5 release which will include
> > many interesting features:
> >
> > 1. Portable object API
> > https://issues.apache.org/jira/browse/IGNITE-1486
> >
> > 2. Ignite.NET and Ignite C++
> > https://issues.apache.org/jira/browse/IGNITE-1282
> >
> > 3. Optimistic serializable transactions
> > https://issues.apache.org/jira/browse/IGNITE-1607
> >
> > 4. Distributed SQL joins - we will be able to query non-collocated data
> as
> > well
> > https://issues.apache.org/jira/browse/IGNITE-1232
> >
> > 5. Enhanced Oracle and IBM JDK interoperability
> > https://issues.apache.org/jira/browse/IGNITE-1526
> >
> > 6. MQTT streamer
> > https://issues.apache.org/jira/browse/IGNITE-535
> >
> > 7. Continuous query failover
> > https://issues.apache.org/jira/browse/IGNITE-426
> >
> > 8. Significant transactional cache performance optimizations - I will
> merge
> > these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
> >
> > 9. Many stability and fault-tolerance fixes.
> >
> > 10. I would also like to include distributed Semaphore. Vladislav, any
> > chance you can finish with it this week?
> > https://issues.apache.org/jira/browse/IGNITE-
> > <https://issues.apache.org/jira/browse/IGNITE-426>638
> >
> > Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
> > please respond to this email and let us know when can we expect your
> > changes being merged to master and release branch?
> >
> > Can someone create ignite-1.5 release branch?
> >
> > --Yakov
> >
>

[Semyon Boikov]

We run with -XX:+UseConcMarkSweepGC -XX:+UseParNewGC.

On Thu, Nov 5, 2015 at 8:42 PM, Dmitriy Setrakyan <[hidden email]>
wrote:

> Semyon,
>
> Do you know what kind of Garbage Collector you have used for the
> benchmarks? Was it G1?
>
> D.
>
> On Thu, Nov 5, 2015 at 5:55 AM, Semyon Boikov <[hidden email]>
> wrote:
>
> > I merged into 1.5 performance optimizations implemented by Yakov. With
> > these optimizations we got up to 20% throughput increase in transactional
> > cache benchmarks.
> >
> > On Mon, Nov 2, 2015 at 4:35 PM, Yakov Zhdanov <[hidden email]>
> wrote:
> >
> > > Guys,
> > >
> > > I think we can start preparation to Ignite-1.5 release which will
> include
> > > many interesting features:
> > >
> > > 1. Portable object API
> > > https://issues.apache.org/jira/browse/IGNITE-1486
> > >
> > > 2. Ignite.NET and Ignite C++
> > > https://issues.apache.org/jira/browse/IGNITE-1282
> > >
> > > 3. Optimistic serializable transactions
> > > https://issues.apache.org/jira/browse/IGNITE-1607
> > >
> > > 4. Distributed SQL joins - we will be able to query non-collocated data
> > as
> > > well
> > > https://issues.apache.org/jira/browse/IGNITE-1232
> > >
> > > 5. Enhanced Oracle and IBM JDK interoperability
> > > https://issues.apache.org/jira/browse/IGNITE-1526
> > >
> > > 6. MQTT streamer
> > > https://issues.apache.org/jira/browse/IGNITE-535
> > >
> > > 7. Continuous query failover
> > > https://issues.apache.org/jira/browse/IGNITE-426
> > >
> > > 8. Significant transactional cache performance optimizations - I will
> > merge
> > > these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
> > >
> > > 9. Many stability and fault-tolerance fixes.
> > >
> > > 10. I would also like to include distributed Semaphore. Vladislav, any
> > > chance you can finish with it this week?
> > > https://issues.apache.org/jira/browse/IGNITE-
> > > <https://issues.apache.org/jira/browse/IGNITE-426>638
> > >
> > > Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
> > > please respond to this email and let us know when can we expect your
> > > changes being merged to master and release branch?
> > >
> > > Can someone create ignite-1.5 release branch?
> > >
> > > --Yakov
> > >
> >
>

[dsetrakyan]

I would also try G1 collector. It will be the default collector in Java 9.

On Sun, Nov 8, 2015 at 10:14 PM, Semyon Boikov <[hidden email]> wrote:

> We run with -XX:+UseConcMarkSweepGC -XX:+UseParNewGC.
>
> On Thu, Nov 5, 2015 at 8:42 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > Semyon,
> >
> > Do you know what kind of Garbage Collector you have used for the
> > benchmarks? Was it G1?
> >
> > D.
> >
> > On Thu, Nov 5, 2015 at 5:55 AM, Semyon Boikov <[hidden email]>
> > wrote:
> >
> > > I merged into 1.5 performance optimizations implemented by Yakov. With
> > > these optimizations we got up to 20% throughput increase in
> transactional
> > > cache benchmarks.
> > >
> > > On Mon, Nov 2, 2015 at 4:35 PM, Yakov Zhdanov <[hidden email]>
> > wrote:
> > >
> > > > Guys,
> > > >
> > > > I think we can start preparation to Ignite-1.5 release which will
> > include
> > > > many interesting features:
> > > >
> > > > 1. Portable object API
> > > > https://issues.apache.org/jira/browse/IGNITE-1486
> > > >
> > > > 2. Ignite.NET and Ignite C++
> > > > https://issues.apache.org/jira/browse/IGNITE-1282
> > > >
> > > > 3. Optimistic serializable transactions
> > > > https://issues.apache.org/jira/browse/IGNITE-1607
> > > >
> > > > 4. Distributed SQL joins - we will be able to query non-collocated
> data
> > > as
> > > > well
> > > > https://issues.apache.org/jira/browse/IGNITE-1232
> > > >
> > > > 5. Enhanced Oracle and IBM JDK interoperability
> > > > https://issues.apache.org/jira/browse/IGNITE-1526
> > > >
> > > > 6. MQTT streamer
> > > > https://issues.apache.org/jira/browse/IGNITE-535
> > > >
> > > > 7. Continuous query failover
> > > > https://issues.apache.org/jira/browse/IGNITE-426
> > > >
> > > > 8. Significant transactional cache performance optimizations - I will
> > > merge
> > > > these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
> > > >
> > > > 9. Many stability and fault-tolerance fixes.
> > > >
> > > > 10. I would also like to include distributed Semaphore. Vladislav,
> any
> > > > chance you can finish with it this week?
> > > > https://issues.apache.org/jira/browse/IGNITE-
> > > > <https://issues.apache.org/jira/browse/IGNITE-426>638
> > > >
> > > > Thanks to everyone involved! Guys, esp. assignees of mentioned
> issues,
> > > > please respond to this email and let us know when can we expect your
> > > > changes being merged to master and release branch?
> > > >
> > > > Can someone create ignite-1.5 release branch?
> > > >
> > > > --Yakov
> > > >
> > >
> >
>

[yzhdanov]

Vladislav, do you have any updates for
https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?

--Yakov

2015-11-02 22:23 GMT+03:00 Vladisav Jelisavcic <[hidden email]>:

> >10. I would also like to include distributed Semaphore. Vladislav, any
> >chance you can finish with it this week?
> >https://issues.apache.org/jira/browse/IGNITE-
> ><https://issues.apache.org/jira/browse/IGNITE-426>638
>
> I will have it done by thursday.
>
> Best regards,
> Vladisav
>
> On Mon, Nov 2, 2015 at 6:40 PM, Dmitriy Setrakyan <[hidden email]>
> wrote:
>
> > On Mon, Nov 2, 2015 at 6:58 AM, M G <[hidden email]> wrote:
> >
> > > Can/will this include
> https://issues.apache.org/jira/browse/IGNITE-1681
> > ?
> > >
> >
> > I don’t see why not. Would be nice if one of the committers could pick up
> > the review for this patch.
> >
> >
> > >
> > > On Mon, Nov 2, 2015 at 1:51 PM, Anton Vinogradov <
> > [hidden email]
> > > >
> > > wrote:
> > >
> > > > Branch ignite-1.5 created.
> > > >
> > > > On Mon, Nov 2, 2015 at 4:39 PM, Anton Vinogradov <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > I assume that correct link at 10th feature is
> > > > > https://issues.apache.org/jira/browse/IGNITE-638
> > > > >
> > > > > On Mon, Nov 2, 2015 at 4:35 PM, Yakov Zhdanov <[hidden email]
> >
> > > > wrote:
> > > > >
> > > > >> Guys,
> > > > >>
> > > > >> I think we can start preparation to Ignite-1.5 release which will
> > > > include
> > > > >> many interesting features:
> > > > >>
> > > > >> 1. Portable object API
> > > > >> https://issues.apache.org/jira/browse/IGNITE-1486
> > > > >>
> > > > >> 2. Ignite.NET and Ignite C++
> > > > >> https://issues.apache.org/jira/browse/IGNITE-1282
> > > > >>
> > > > >> 3. Optimistic serializable transactions
> > > > >> https://issues.apache.org/jira/browse/IGNITE-1607
> > > > >>
> > > > >> 4. Distributed SQL joins - we will be able to query non-collocated
> > > data
> > > > as
> > > > >> well
> > > > >> https://issues.apache.org/jira/browse/IGNITE-1232
> > > > >>
> > > > >> 5. Enhanced Oracle and IBM JDK interoperability
> > > > >> https://issues.apache.org/jira/browse/IGNITE-1526
> > > > >>
> > > > >> 6. MQTT streamer
> > > > >> https://issues.apache.org/jira/browse/IGNITE-535
> > > > >>
> > > > >> 7. Continuous query failover
> > > > >> https://issues.apache.org/jira/browse/IGNITE-426
> > > > >>
> > > > >> 8. Significant transactional cache performance optimizations - I
> > will
> > > > >> merge
> > > > >> these changes from 'ignite-1.4-slow-server-debug' today or
> tomorrow.
> > > > >>
> > > > >> 9. Many stability and fault-tolerance fixes.
> > > > >>
> > > > >> 10. I would also like to include distributed Semaphore. Vladislav,
> > any
> > > > >> chance you can finish with it this week?
> > > > >> https://issues.apache.org/jira/browse/IGNITE-
> > > > >> <https://issues.apache.org/jira/browse/IGNITE-426>638
> > > > >>
> > > > >> Thanks to everyone involved! Guys, esp. assignees of mentioned
> > issues,
> > > > >> please respond to this email and let us know when can we expect
> your
> > > > >> changes being merged to master and release branch?
> > > > >>
> > > > >> Can someone create ignite-1.5 release branch?
> > > > >>
> > > > >> --Yakov
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

[Alexey Goncharuk]

Guys,

A quick update on the release status.

I have spotted several potential slowdowns that I would like to add to 1.5
release. I am currently working in a separate branch and preliminary
testing showed that we can get about 4-5% performance improvement in tx-put
benchmark. There are other potential improvements that I did not benchmark
yet, but I believe it may add another 4-5% for the same benchmark.

Also, as Vladimir noticed, there is IGNITE-1377 issue that is critical for
the release. I can take a closer look at it once I finish with the
performance improvements, however if there is anyone who can handle it now
- it would be great!

2015-11-10 16:36 GMT+03:00 Yakov Zhdanov <[hidden email]>:

> Vladislav, do you have any updates for
> https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?
>
> --Yakov
>
> 2015-11-02 22:23 GMT+03:00 Vladisav Jelisavcic <[hidden email]>:
>
> > >10. I would also like to include distributed Semaphore. Vladislav, any
> > >chance you can finish with it this week?
> > >https://issues.apache.org/jira/browse/IGNITE-
> > ><https://issues.apache.org/jira/browse/IGNITE-426>638
> >
> > I will have it done by thursday.
> >
> > Best regards,
> > Vladisav
> >
> > On Mon, Nov 2, 2015 at 6:40 PM, Dmitriy Setrakyan <[hidden email]
> >
> > wrote:
> >
> > > On Mon, Nov 2, 2015 at 6:58 AM, M G <[hidden email]> wrote:
> > >
> > > > Can/will this include
> > https://issues.apache.org/jira/browse/IGNITE-1681
> > > ?
> > > >
> > >
> > > I don’t see why not. Would be nice if one of the committers could pick
> up
> > > the review for this patch.
> > >
> > >
> > > >
> > > > On Mon, Nov 2, 2015 at 1:51 PM, Anton Vinogradov <
> > > [hidden email]
> > > > >
> > > > wrote:
> > > >
> > > > > Branch ignite-1.5 created.
> > > > >
> > > > > On Mon, Nov 2, 2015 at 4:39 PM, Anton Vinogradov <
> > > > [hidden email]
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I assume that correct link at 10th feature is
> > > > > > https://issues.apache.org/jira/browse/IGNITE-638
> > > > > >
> > > > > > On Mon, Nov 2, 2015 at 4:35 PM, Yakov Zhdanov <
> [hidden email]
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Guys,
> > > > > >>
> > > > > >> I think we can start preparation to Ignite-1.5 release which
> will
> > > > > include
> > > > > >> many interesting features:
> > > > > >>
> > > > > >> 1. Portable object API
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-1486
> > > > > >>
> > > > > >> 2. Ignite.NET and Ignite C++
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-1282
> > > > > >>
> > > > > >> 3. Optimistic serializable transactions
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-1607
> > > > > >>
> > > > > >> 4. Distributed SQL joins - we will be able to query
> non-collocated
> > > > data
> > > > > as
> > > > > >> well
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-1232
> > > > > >>
> > > > > >> 5. Enhanced Oracle and IBM JDK interoperability
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-1526
> > > > > >>
> > > > > >> 6. MQTT streamer
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-535
> > > > > >>
> > > > > >> 7. Continuous query failover
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-426
> > > > > >>
> > > > > >> 8. Significant transactional cache performance optimizations - I
> > > will
> > > > > >> merge
> > > > > >> these changes from 'ignite-1.4-slow-server-debug' today or
> > tomorrow.
> > > > > >>
> > > > > >> 9. Many stability and fault-tolerance fixes.
> > > > > >>
> > > > > >> 10. I would also like to include distributed Semaphore.
> Vladislav,
> > > any
> > > > > >> chance you can finish with it this week?
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-
> > > > > >> <https://issues.apache.org/jira/browse/IGNITE-426>638
> > > > > >>
> > > > > >> Thanks to everyone involved! Guys, esp. assignees of mentioned
> > > issues,
> > > > > >> please respond to this email and let us know when can we expect
> > your
> > > > > >> changes being merged to master and release branch?
> > > > > >>
> > > > > >> Can someone create ignite-1.5 release branch?
> > > > > >>
> > > > > >> --Yakov
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Raul Kripalani]

Sorry I haven't made an appearance in this thread yet.

> 6. MQTT streamer
> https://issues.apache.org/jira/browse/IGNITE-535

Yes, it was merged to master before the ignite-1.5 was created.

I'd like to add:

Camel Streamer => https://issues.apache.org/jira/browse/IGNITE-1790
-- I'll merge this as soon as I finished with the OSGi tickets with demand.

OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI (or
whatever agreement we arrive to in mailing lists and Wiki)
-- https://issues.apache.org/jira/browse/IGNITE-1527
-- https://issues.apache.org/jira/browse/IGNITE-1877
-- I'm working actively on these two features.

*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]> wrote:

> Guys,
>
> I think we can start preparation to Ignite-1.5 release which will include
> many interesting features:
>
> 1. Portable object API
> https://issues.apache.org/jira/browse/IGNITE-1486
>
> 2. Ignite.NET and Ignite C++
> https://issues.apache.org/jira/browse/IGNITE-1282
>
> 3. Optimistic serializable transactions
> https://issues.apache.org/jira/browse/IGNITE-1607
>
> 4. Distributed SQL joins - we will be able to query non-collocated data as
> well
> https://issues.apache.org/jira/browse/IGNITE-1232
>
> 5. Enhanced Oracle and IBM JDK interoperability
> https://issues.apache.org/jira/browse/IGNITE-1526
>
> 6. MQTT streamer
> https://issues.apache.org/jira/browse/IGNITE-535
>
> 7. Continuous query failover
> https://issues.apache.org/jira/browse/IGNITE-426
>
> 8. Significant transactional cache performance optimizations - I will merge
> these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
>
> 9. Many stability and fault-tolerance fixes.
>
> 10. I would also like to include distributed Semaphore. Vladislav, any
> chance you can finish with it this week?
> https://issues.apache.org/jira/browse/IGNITE-
> <https://issues.apache.org/jira/browse/IGNITE-426>638
>
> Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
> please respond to this email and let us know when can we expect your
> changes being merged to master and release branch?
>
> Can someone create ignite-1.5 release branch?
>
> --Yakov
>

[Vladisav Jelisavcic]

Yakov,

sorry  for running a bit late.

> Vladislav, do you have any updates for
> https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?
>
> --Yakov

I have problems with some fail-over scenarios;
It seems that if the two nodes are in the middle of acquiring or releasing
the semaphore,
and one of them fails, all nodes get:

[09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
<ignite-atomics-sys-cache> Failed to compare and set:
o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to acquire lock for keys (primary node left grid, retry transaction
if possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
[name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
...
Caused by: class
org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed
to acquire lock for keys (primary node left grid, retry transaction if
possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
[name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
... 10 more


I'm still trying to find out how to exactly reproduce this behavior,
I'll send you more details once I try few more things.

I am still using partitioned cache, does it make sense to use replicated
cache instead?


Other than that, I'm done with everything else.

Thanks,
Vladisav



On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]> wrote:

> Sorry I haven't made an appearance in this thread yet.
>
> > 6. MQTT streamer
> > https://issues.apache.org/jira/browse/IGNITE-535
>
> Yes, it was merged to master before the ignite-1.5 was created.
>
> I'd like to add:
>
> Camel Streamer => https://issues.apache.org/jira/browse/IGNITE-1790
> -- I'll merge this as soon as I finished with the OSGi tickets with demand.
>
> OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI (or
> whatever agreement we arrive to in mailing lists and Wiki)
> -- https://issues.apache.org/jira/browse/IGNITE-1527
> -- https://issues.apache.org/jira/browse/IGNITE-1877
> -- I'm working actively on these two features.
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]> wrote:
>
> > Guys,
> >
> > I think we can start preparation to Ignite-1.5 release which will include
> > many interesting features:
> >
> > 1. Portable object API
> > https://issues.apache.org/jira/browse/IGNITE-1486
> >
> > 2. Ignite.NET and Ignite C++
> > https://issues.apache.org/jira/browse/IGNITE-1282
> >
> > 3. Optimistic serializable transactions
> > https://issues.apache.org/jira/browse/IGNITE-1607
> >
> > 4. Distributed SQL joins - we will be able to query non-collocated data
> as
> > well
> > https://issues.apache.org/jira/browse/IGNITE-1232
> >
> > 5. Enhanced Oracle and IBM JDK interoperability
> > https://issues.apache.org/jira/browse/IGNITE-1526
> >
> > 6. MQTT streamer
> > https://issues.apache.org/jira/browse/IGNITE-535
> >
> > 7. Continuous query failover
> > https://issues.apache.org/jira/browse/IGNITE-426
> >
> > 8. Significant transactional cache performance optimizations - I will
> merge
> > these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
> >
> > 9. Many stability and fault-tolerance fixes.
> >
> > 10. I would also like to include distributed Semaphore. Vladislav, any
> > chance you can finish with it this week?
> > https://issues.apache.org/jira/browse/IGNITE-
> > <https://issues.apache.org/jira/browse/IGNITE-426>638
> >
> > Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
> > please respond to this email and let us know when can we expect your
> > changes being merged to master and release branch?
> >
> > Can someone create ignite-1.5 release branch?
> >
> > --Yakov
> >
>

[yzhdanov]

Vladislav, how can I take a look at the code and run your tests?

--Yakov

2015-11-11 12:33 GMT+03:00 Vladisav Jelisavcic <[hidden email]>:

> Yakov,
>
> sorry  for running a bit late.
>
> > Vladislav, do you have any updates for
> > https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?
> >
> > --Yakov
>
> I have problems with some fail-over scenarios;
> It seems that if the two nodes are in the middle of acquiring or releasing
> the semaphore,
> and one of them fails, all nodes get:
>
> [09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
> <ignite-atomics-sys-cache> Failed to compare and set:
> o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
> class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> Failed to acquire lock for keys (primary node left grid, retry transaction
> if possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> ...
> Caused by: class
> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed
> to acquire lock for keys (primary node left grid, retry transaction if
> possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
> ... 10 more
>
>
> I'm still trying to find out how to exactly reproduce this behavior,
> I'll send you more details once I try few more things.
>
> I am still using partitioned cache, does it make sense to use replicated
> cache instead?
>
>
> Other than that, I'm done with everything else.
>
> Thanks,
> Vladisav
>
>
>
> On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]> wrote:
>
> > Sorry I haven't made an appearance in this thread yet.
> >
> > > 6. MQTT streamer
> > > https://issues.apache.org/jira/browse/IGNITE-535
> >
> > Yes, it was merged to master before the ignite-1.5 was created.
> >
> > I'd like to add:
> >
> > Camel Streamer => https://issues.apache.org/jira/browse/IGNITE-1790
> > -- I'll merge this as soon as I finished with the OSGi tickets with
> demand.
> >
> > OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI (or
> > whatever agreement we arrive to in mailing lists and Wiki)
> > -- https://issues.apache.org/jira/browse/IGNITE-1527
> > -- https://issues.apache.org/jira/browse/IGNITE-1877
> > -- I'm working actively on these two features.
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]>
> wrote:
> >
> > > Guys,
> > >
> > > I think we can start preparation to Ignite-1.5 release which will
> include
> > > many interesting features:
> > >
> > > 1. Portable object API
> > > https://issues.apache.org/jira/browse/IGNITE-1486
> > >
> > > 2. Ignite.NET and Ignite C++
> > > https://issues.apache.org/jira/browse/IGNITE-1282
> > >
> > > 3. Optimistic serializable transactions
> > > https://issues.apache.org/jira/browse/IGNITE-1607
> > >
> > > 4. Distributed SQL joins - we will be able to query non-collocated data
> > as
> > > well
> > > https://issues.apache.org/jira/browse/IGNITE-1232
> > >
> > > 5. Enhanced Oracle and IBM JDK interoperability
> > > https://issues.apache.org/jira/browse/IGNITE-1526
> > >
> > > 6. MQTT streamer
> > > https://issues.apache.org/jira/browse/IGNITE-535
> > >
> > > 7. Continuous query failover
> > > https://issues.apache.org/jira/browse/IGNITE-426
> > >
> > > 8. Significant transactional cache performance optimizations - I will
> > merge
> > > these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
> > >
> > > 9. Many stability and fault-tolerance fixes.
> > >
> > > 10. I would also like to include distributed Semaphore. Vladislav, any
> > > chance you can finish with it this week?
> > > https://issues.apache.org/jira/browse/IGNITE-
> > > <https://issues.apache.org/jira/browse/IGNITE-426>638
> > >
> > > Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
> > > please respond to this email and let us know when can we expect your
> > > changes being merged to master and release branch?
> > >
> > > Can someone create ignite-1.5 release branch?
> > >
> > > --Yakov
> > >
> >
>

[Denis Magda]

Hi Vladislav,

Please see below..

On 11/11/2015 12:33 PM, Vladisav Jelisavcic wrote:

> Yakov,
>
> sorry  for running a bit late.
>
>> Vladislav, do you have any updates for
>> https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?
>>
>> --Yakov
> I have problems with some fail-over scenarios;
> It seems that if the two nodes are in the middle of acquiring or releasing
> the semaphore,
> and one of them fails, all nodes get:
>
> [09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
> <ignite-atomics-sys-cache> Failed to compare and set:
> o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
> class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> Failed to acquire lock for keys (primary node left grid, retry transaction
> if possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> ...
> Caused by: class
> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed
> to acquire lock for keys (primary node left grid, retry transaction if
> possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
> ... 10 more

You have to process this exception manually at your implementation layer
since your data structure uses a transactional cache.
Below is a kind of template I used when it was required to process this
and some other exeptions. You can use it as-is.

int retries = GridCacheAdapter.MAX_RETRIES;

IgniteCheckedException err =null;

for (int i =0; i < retries; i++) {
     try {
         //Your transactional code that may fail
     }
     catch (IgniteCheckedException e) {
         if (i == retries)
             throw e;

         if (X.hasCause(e, ClusterTopologyCheckedException.class)) {
             ClusterTopologyCheckedException topErr = e.getCause(ClusterTopologyCheckedException.class);

             topErr.retryReadyFuture().get();
         }
         else if (X.hasCause(e, IgniteTxRollbackCheckedException.class))
             U.sleep(1);
         else throw e;
     }
}


>
> I'm still trying to find out how to exactly reproduce this behavior,
> I'll send you more details once I try few more things.
There is the test suite called
GridCacheAbstractDataStructuresFailoverSelfTest that checks Ignite
atomics and data structures with fail-over scenario.
The suite will let you reproduce ClusterTopologyCheckedException easily.
Just add your tests there referring to the tests of other data structures.

Presently I'm improving this test suite under my work on IGNITE-801 and
IGNITE-803. If you finish your task earlier then I'll adopt your tests
to a new test approach.

>
> I am still using partitioned cache, does it make sense to use replicated
> cache instead?
>
Yeah, you should support this as well. Cache mode for the data
structures is changed using CollectionConfigurations while for atomics
using AtomicsConfiguration.

--
Denis

> Other than that, I'm done with everything else.
>
> Thanks,
> Vladisav
>
>
>
> On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]> wrote:
>
>> Sorry I haven't made an appearance in this thread yet.
>>
>>> 6. MQTT streamer
>>> https://issues.apache.org/jira/browse/IGNITE-535
>> Yes, it was merged to master before the ignite-1.5 was created.
>>
>> I'd like to add:
>>
>> Camel Streamer => https://issues.apache.org/jira/browse/IGNITE-1790
>> -- I'll merge this as soon as I finished with the OSGi tickets with demand.
>>
>> OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI (or
>> whatever agreement we arrive to in mailing lists and Wiki)
>> -- https://issues.apache.org/jira/browse/IGNITE-1527
>> -- https://issues.apache.org/jira/browse/IGNITE-1877
>> -- I'm working actively on these two features.
>>
>> *Raúl Kripalani*
>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
>> Messaging Engineer
>> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>> http://blog.raulkr.net | twitter: @raulvk
>>
>> On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]> wrote:
>>
>>> Guys,
>>>
>>> I think we can start preparation to Ignite-1.5 release which will include
>>> many interesting features:
>>>
>>> 1. Portable object API
>>> https://issues.apache.org/jira/browse/IGNITE-1486
>>>
>>> 2. Ignite.NET and Ignite C++
>>> https://issues.apache.org/jira/browse/IGNITE-1282
>>>
>>> 3. Optimistic serializable transactions
>>> https://issues.apache.org/jira/browse/IGNITE-1607
>>>
>>> 4. Distributed SQL joins - we will be able to query non-collocated data
>> as
>>> well
>>> https://issues.apache.org/jira/browse/IGNITE-1232
>>>
>>> 5. Enhanced Oracle and IBM JDK interoperability
>>> https://issues.apache.org/jira/browse/IGNITE-1526
>>>
>>> 6. MQTT streamer
>>> https://issues.apache.org/jira/browse/IGNITE-535
>>>
>>> 7. Continuous query failover
>>> https://issues.apache.org/jira/browse/IGNITE-426
>>>
>>> 8. Significant transactional cache performance optimizations - I will
>> merge
>>> these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
>>>
>>> 9. Many stability and fault-tolerance fixes.
>>>
>>> 10. I would also like to include distributed Semaphore. Vladislav, any
>>> chance you can finish with it this week?
>>> https://issues.apache.org/jira/browse/IGNITE-
>>> <https://issues.apache.org/jira/browse/IGNITE-426>638
>>>
>>> Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
>>> please respond to this email and let us know when can we expect your
>>> changes being merged to master and release branch?
>>>
>>> Can someone create ignite-1.5 release branch?
>>>
>>> --Yakov
>>>

[Vladimir Ozerov]

At the moment I am getting closer to finish with IGNITE-1816 - optional
compaction of portable object footers. In general feature works. In the
nearest time I am going to perform some microbenchmarks to understand
implications. Two things left:
- I have to rework lots of binary object unit tests because they were
designed to run with disabled metadata, and compact footers require
metadata.
- Need to add cross-node validation.

On Wed, Nov 11, 2015 at 3:48 PM, Denis Magda <[hidden email]> wrote:

> Hi Vladislav,
>
> Please see below..
>
>
> On 11/11/2015 12:33 PM, Vladisav Jelisavcic wrote:
>
>> Yakov,
>>
>> sorry  for running a bit late.
>>
>> Vladislav, do you have any updates for
>>> https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?
>>>
>>> --Yakov
>>>
>> I have problems with some fail-over scenarios;
>> It seems that if the two nodes are in the middle of acquiring or releasing
>> the semaphore,
>> and one of them fails, all nodes get:
>>
>> [09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
>> <ignite-atomics-sys-cache> Failed to compare and set:
>> o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
>> class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
>> Failed to acquire lock for keys (primary node left grid, retry transaction
>> if possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
>> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
>> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
>> ...
>> Caused by: class
>> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed
>> to acquire lock for keys (primary node left grid, retry transaction if
>> possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
>> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
>> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
>> at
>>
>> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
>> ... 10 more
>>
> You have to process this exception manually at your implementation layer
> since your data structure uses a transactional cache.
> Below is a kind of template I used when it was required to process this
> and some other exeptions. You can use it as-is.
>
> int retries = GridCacheAdapter.MAX_RETRIES;
>
> IgniteCheckedException err =null;
>
> for (int i =0; i < retries; i++) {
>     try {
>         //Your transactional code that may fail
>     }
>     catch (IgniteCheckedException e) {
>         if (i == retries)
>             throw e;
>
>         if (X.hasCause(e, ClusterTopologyCheckedException.class)) {
>             ClusterTopologyCheckedException topErr =
> e.getCause(ClusterTopologyCheckedException.class);
>
>             topErr.retryReadyFuture().get();
>         }
>         else if (X.hasCause(e, IgniteTxRollbackCheckedException.class))
>             U.sleep(1);
>         else throw e;
>     }
> }
>
>
>
>> I'm still trying to find out how to exactly reproduce this behavior,
>> I'll send you more details once I try few more things.
>>
> There is the test suite called
> GridCacheAbstractDataStructuresFailoverSelfTest that checks Ignite atomics
> and data structures with fail-over scenario.
> The suite will let you reproduce ClusterTopologyCheckedException easily.
> Just add your tests there referring to the tests of other data structures.
>
> Presently I'm improving this test suite under my work on IGNITE-801 and
> IGNITE-803. If you finish your task earlier then I'll adopt your tests to a
> new test approach.
>
>
>> I am still using partitioned cache, does it make sense to use replicated
>> cache instead?
>>
>> Yeah, you should support this as well. Cache mode for the data structures
> is changed using CollectionConfigurations while for atomics using
> AtomicsConfiguration.
>
> --
> Denis
>
>
> Other than that, I'm done with everything else.
>>
>> Thanks,
>> Vladisav
>>
>>
>>
>> On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]> wrote:
>>
>> Sorry I haven't made an appearance in this thread yet.
>>>
>>> 6. MQTT streamer
>>>> https://issues.apache.org/jira/browse/IGNITE-535
>>>>
>>> Yes, it was merged to master before the ignite-1.5 was created.
>>>
>>> I'd like to add:
>>>
>>> Camel Streamer => https://issues.apache.org/jira/browse/IGNITE-1790
>>> -- I'll merge this as soon as I finished with the OSGi tickets with
>>> demand.
>>>
>>> OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI (or
>>> whatever agreement we arrive to in mailing lists and Wiki)
>>> -- https://issues.apache.org/jira/browse/IGNITE-1527
>>> -- https://issues.apache.org/jira/browse/IGNITE-1877
>>> -- I'm working actively on these two features.
>>>
>>> *Raúl Kripalani*
>>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
>>> Messaging Engineer
>>> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>>> http://blog.raulkr.net | twitter: @raulvk
>>>
>>> On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]>
>>> wrote:
>>>
>>> Guys,
>>>>
>>>> I think we can start preparation to Ignite-1.5 release which will
>>>> include
>>>> many interesting features:
>>>>
>>>> 1. Portable object API
>>>> https://issues.apache.org/jira/browse/IGNITE-1486
>>>>
>>>> 2. Ignite.NET and Ignite C++
>>>> https://issues.apache.org/jira/browse/IGNITE-1282
>>>>
>>>> 3. Optimistic serializable transactions
>>>> https://issues.apache.org/jira/browse/IGNITE-1607
>>>>
>>>> 4. Distributed SQL joins - we will be able to query non-collocated data
>>>>
>>> as
>>>
>>>> well
>>>> https://issues.apache.org/jira/browse/IGNITE-1232
>>>>
>>>> 5. Enhanced Oracle and IBM JDK interoperability
>>>> https://issues.apache.org/jira/browse/IGNITE-1526
>>>>
>>>> 6. MQTT streamer
>>>> https://issues.apache.org/jira/browse/IGNITE-535
>>>>
>>>> 7. Continuous query failover
>>>> https://issues.apache.org/jira/browse/IGNITE-426
>>>>
>>>> 8. Significant transactional cache performance optimizations - I will
>>>>
>>> merge
>>>
>>>> these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
>>>>
>>>> 9. Many stability and fault-tolerance fixes.
>>>>
>>>> 10. I would also like to include distributed Semaphore. Vladislav, any
>>>> chance you can finish with it this week?
>>>> https://issues.apache.org/jira/browse/IGNITE-
>>>> <https://issues.apache.org/jira/browse/IGNITE-426>638
>>>>
>>>> Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
>>>> please respond to this email and let us know when can we expect your
>>>> changes being merged to master and release branch?
>>>>
>>>> Can someone create ignite-1.5 release branch?
>>>>
>>>> --Yakov
>>>>
>>>>
>

[Vladisav Jelisavcic]

Hi Denis,

Thanks a lot, it looks like my test setup was wrong,
I added semaphore tests to GridCacheAbstractDataStructuresFailoverSelfTest
suite.
Now I have following problem:
when I run tests with TOP_CHANGE_THREAD_CNT = 3
tests fail when they reach stop() with the following exception:

class org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is
stopping: 09c5e8b8-8998-468e-960d-223220354fd3
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:113)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:946)
at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1823)
at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1769)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2133)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2096)
at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:314)
at org.apache.ignite.Ignition.stop(Ignition.java:223)
at
org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:802)
at
org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:784)
at
org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest.access$500(GridCacheAbstractDataStructuresFailoverSelfTest.java:54)
at
org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest$5.apply(GridCacheAbstractDataStructuresFailoverSelfTest.java:459)

When I run tests with TOP_CHANGE_THEAD_CNT = 1
everything is running ok;


@Yakov
I made a new commit to my IGNITE-638 branch,
can you please take a look?



Best regards,
Vladisav

> On Wed, Nov 11, 2015 at 3:48 PM, Denis Magda <[hidden email]> wrote:
>
> > Hi Vladislav,
> >
> > Please see below..
> >
> >
> > On 11/11/2015 12:33 PM, Vladisav Jelisavcic wrote:
> >
> >> Yakov,
> >>
> >> sorry  for running a bit late.
> >>
> >> Vladislav, do you have any updates for
> >>> https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?
> >>>
> >>> --Yakov
> >>>
> >> I have problems with some fail-over scenarios;
> >> It seems that if the two nodes are in the middle of acquiring or
> releasing
> >> the semaphore,
> >> and one of them fails, all nodes get:
> >>
> >> [09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
> >> <ignite-atomics-sys-cache> Failed to compare and set:
> >> o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
> >> class
> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> >> Failed to acquire lock for keys (primary node left grid, retry
> transaction
> >> if possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
> >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> >> ...
> >> Caused by: class
> >> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> Failed
> >> to acquire lock for keys (primary node left grid, retry transaction if
> >> possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
> >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> >> at
> >>
> >>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
> >> ... 10 more
> >>
> > You have to process this exception manually at your implementation layer
> > since your data structure uses a transactional cache.
> > Below is a kind of template I used when it was required to process this
> > and some other exeptions. You can use it as-is.
> >
> > int retries = GridCacheAdapter.MAX_RETRIES;
> >
> > IgniteCheckedException err =null;
> >
> > for (int i =0; i < retries; i++) {
> >     try {
> >         //Your transactional code that may fail
> >     }
> >     catch (IgniteCheckedException e) {
> >         if (i == retries)
> >             throw e;
> >
> >         if (X.hasCause(e, ClusterTopologyCheckedException.class)) {
> >             ClusterTopologyCheckedException topErr =
> > e.getCause(ClusterTopologyCheckedException.class);
> >
> >             topErr.retryReadyFuture().get();
> >         }
> >         else if (X.hasCause(e, IgniteTxRollbackCheckedException.class))
> >             U.sleep(1);
> >         else throw e;
> >     }
> > }
> >
> >
> >
> >> I'm still trying to find out how to exactly reproduce this behavior,
> >> I'll send you more details once I try few more things.
> >>
> > There is the test suite called
> > GridCacheAbstractDataStructuresFailoverSelfTest that checks Ignite
> atomics
> > and data structures with fail-over scenario.
> > The suite will let you reproduce ClusterTopologyCheckedException easily.
> > Just add your tests there referring to the tests of other data
> structures.
> >
> > Presently I'm improving this test suite under my work on IGNITE-801 and
> > IGNITE-803. If you finish your task earlier then I'll adopt your tests
> to a
> > new test approach.
> >
> >
> >> I am still using partitioned cache, does it make sense to use replicated
> >> cache instead?
> >>
> >> Yeah, you should support this as well. Cache mode for the data
> structures
> > is changed using CollectionConfigurations while for atomics using
> > AtomicsConfiguration.
> >
> > --
> > Denis
> >
> >
> > Other than that, I'm done with everything else.
> >>
> >> Thanks,
> >> Vladisav
> >>
> >>
> >>
> >> On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]>
> wrote:
> >>
> >> Sorry I haven't made an appearance in this thread yet.
> >>>
> >>> 6. MQTT streamer
> >>>> https://issues.apache.org/jira/browse/IGNITE-535
> >>>>
> >>> Yes, it was merged to master before the ignite-1.5 was created.
> >>>
> >>> I'd like to add:
> >>>
> >>> Camel Streamer => https://issues.apache.org/jira/browse/IGNITE-1790
> >>> -- I'll merge this as soon as I finished with the OSGi tickets with
> >>> demand.
> >>>
> >>> OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI (or
> >>> whatever agreement we arrive to in mailing lists and Wiki)
> >>> -- https://issues.apache.org/jira/browse/IGNITE-1527
> >>> -- https://issues.apache.org/jira/browse/IGNITE-1877
> >>> -- I'm working actively on these two features.
> >>>
> >>> *Raúl Kripalani*
> >>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> and
> >>> Messaging Engineer
> >>> http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> >>> http://blog.raulkr.net | twitter: @raulvk
> >>>
> >>> On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]>
> >>> wrote:
> >>>
> >>> Guys,
> >>>>
> >>>> I think we can start preparation to Ignite-1.5 release which will
> >>>> include
> >>>> many interesting features:
> >>>>
> >>>> 1. Portable object API
> >>>> https://issues.apache.org/jira/browse/IGNITE-1486
> >>>>
> >>>> 2. Ignite.NET and Ignite C++
> >>>> https://issues.apache.org/jira/browse/IGNITE-1282
> >>>>
> >>>> 3. Optimistic serializable transactions
> >>>> https://issues.apache.org/jira/browse/IGNITE-1607
> >>>>
> >>>> 4. Distributed SQL joins - we will be able to query non-collocated
> data
> >>>>
> >>> as
> >>>
> >>>> well
> >>>> https://issues.apache.org/jira/browse/IGNITE-1232
> >>>>
> >>>> 5. Enhanced Oracle and IBM JDK interoperability
> >>>> https://issues.apache.org/jira/browse/IGNITE-1526
> >>>>
> >>>> 6. MQTT streamer
> >>>> https://issues.apache.org/jira/browse/IGNITE-535
> >>>>
> >>>> 7. Continuous query failover
> >>>> https://issues.apache.org/jira/browse/IGNITE-426
> >>>>
> >>>> 8. Significant transactional cache performance optimizations - I will
> >>>>
> >>> merge
> >>>
> >>>> these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
> >>>>
> >>>> 9. Many stability and fault-tolerance fixes.
> >>>>
> >>>> 10. I would also like to include distributed Semaphore. Vladislav, any
> >>>> chance you can finish with it this week?
> >>>> https://issues.apache.org/jira/browse/IGNITE-
> >>>> <https://issues.apache.org/jira/browse/IGNITE-426>638
> >>>>
> >>>> Thanks to everyone involved! Guys, esp. assignees of mentioned issues,
> >>>> please respond to this email and let us know when can we expect your
> >>>> changes being merged to master and release branch?
> >>>>
> >>>> Can someone create ignite-1.5 release branch?
> >>>>
> >>>> --Yakov
> >>>>
> >>>>
> >
>

[Vladimir Ozerov]

Folks,

I am working on portable marshalling performance at the moment. I took a
scenario which is good for OptimziedMarshaller and is very unfortunate for
PortableMarshaller: object with 40 int fields.
My idea here is that this scenario is so bad for Portable, so that in the
vast majority of sensible real-life scenarios it will show better relative
numbers.

*Marshalling*
Initially I got the following numbers:
Optimized: 1.9KK/sec
Portable:  *0.7KK/sec*

Applied several optimizations:
- Removed reflection and boxing for primitive fields
- Removed some garbage (e.g. arrays instead of Collections)
- Decreased amount of thread-local lookups

Results after this:
Optimized: 1.9KK/sec
Portable:  *1.2KK/sec*

Further profiling shows that now the most amount of time is spend in footer
preparation and write. I have some ideas how to improve it further, will
check them soon, what will hopefully bring us to ~1.5KK/sec. We will not be
faster than "optimized" here, though.

*Unmarshalling*
Initial measurements:
Optimized: 3KK/sec
Portable:  *0.6KK/sec, 5 times slower*

Will start working on this on soon. There is a lot of optos here, some of
them already battle-tested in .Net, so I expect we will be able to get
~2KK/sec once they are applied.

On Fri, Nov 13, 2015 at 4:43 PM, Vladisav Jelisavcic <[hidden email]>
wrote:

> Hi Denis,
>
> Thanks a lot, it looks like my test setup was wrong,
> I added semaphore tests to GridCacheAbstractDataStructuresFailoverSelfTest
> suite.
> Now I have following problem:
> when I run tests with TOP_CHANGE_THREAD_CNT = 3
> tests fail when they reach stop() with the following exception:
>
> class org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is
> stopping: 09c5e8b8-8998-468e-960d-223220354fd3
> at
>
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:382)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:113)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:946)
> at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1823)
> at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1769)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2133)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2096)
> at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:314)
> at org.apache.ignite.Ignition.stop(Ignition.java:223)
> at
>
> org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:802)
> at
>
> org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:784)
> at
>
> org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest.access$500(GridCacheAbstractDataStructuresFailoverSelfTest.java:54)
> at
>
> org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest$5.apply(GridCacheAbstractDataStructuresFailoverSelfTest.java:459)
>
> When I run tests with TOP_CHANGE_THEAD_CNT = 1
> everything is running ok;
>
>
> @Yakov
> I made a new commit to my IGNITE-638 branch,
> can you please take a look?
>
>
>
> Best regards,
> Vladisav
>
>
>
>
> > On Wed, Nov 11, 2015 at 3:48 PM, Denis Magda <[hidden email]>
> wrote:
> >
> > > Hi Vladislav,
> > >
> > > Please see below..
> > >
> > >
> > > On 11/11/2015 12:33 PM, Vladisav Jelisavcic wrote:
> > >
> > >> Yakov,
> > >>
> > >> sorry  for running a bit late.
> > >>
> > >> Vladislav, do you have any updates for
> > >>> https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?
> > >>>
> > >>> --Yakov
> > >>>
> > >> I have problems with some fail-over scenarios;
> > >> It seems that if the two nodes are in the middle of acquiring or
> > releasing
> > >> the semaphore,
> > >> and one of them fails, all nodes get:
> > >>
> > >> [09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
> > >> <ignite-atomics-sys-cache> Failed to compare and set:
> > >>
> o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
> > >> class
> > org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> > >> Failed to acquire lock for keys (primary node left grid, retry
> > transaction
> > >> if possible) [keys=[UserKeyCacheObjectImpl
> [val=GridCacheInternalKeyImpl
> > >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> > >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> > >> ...
> > >> Caused by: class
> > >> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> > Failed
> > >> to acquire lock for keys (primary node left grid, retry transaction if
> > >> possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
> > >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> > >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> > >> at
> > >>
> > >>
> >
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
> > >> ... 10 more
> > >>
> > > You have to process this exception manually at your implementation
> layer
> > > since your data structure uses a transactional cache.
> > > Below is a kind of template I used when it was required to process this
> > > and some other exeptions. You can use it as-is.
> > >
> > > int retries = GridCacheAdapter.MAX_RETRIES;
> > >
> > > IgniteCheckedException err =null;
> > >
> > > for (int i =0; i < retries; i++) {
> > >     try {
> > >         //Your transactional code that may fail
> > >     }
> > >     catch (IgniteCheckedException e) {
> > >         if (i == retries)
> > >             throw e;
> > >
> > >         if (X.hasCause(e, ClusterTopologyCheckedException.class)) {
> > >             ClusterTopologyCheckedException topErr =
> > > e.getCause(ClusterTopologyCheckedException.class);
> > >
> > >             topErr.retryReadyFuture().get();
> > >         }
> > >         else if (X.hasCause(e, IgniteTxRollbackCheckedException.class))
> > >             U.sleep(1);
> > >         else throw e;
> > >     }
> > > }
> > >
> > >
> > >
> > >> I'm still trying to find out how to exactly reproduce this behavior,
> > >> I'll send you more details once I try few more things.
> > >>
> > > There is the test suite called
> > > GridCacheAbstractDataStructuresFailoverSelfTest that checks Ignite
> > atomics
> > > and data structures with fail-over scenario.
> > > The suite will let you reproduce ClusterTopologyCheckedException
> easily.
> > > Just add your tests there referring to the tests of other data
> > structures.
> > >
> > > Presently I'm improving this test suite under my work on IGNITE-801 and
> > > IGNITE-803. If you finish your task earlier then I'll adopt your tests
> > to a
> > > new test approach.
> > >
> > >
> > >> I am still using partitioned cache, does it make sense to use
> replicated
> > >> cache instead?
> > >>
> > >> Yeah, you should support this as well. Cache mode for the data
> > structures
> > > is changed using CollectionConfigurations while for atomics using
> > > AtomicsConfiguration.
> > >
> > > --
> > > Denis
> > >
> > >
> > > Other than that, I'm done with everything else.
> > >>
> > >> Thanks,
> > >> Vladisav
> > >>
> > >>
> > >>
> > >> On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]>
> > wrote:
> > >>
> > >> Sorry I haven't made an appearance in this thread yet.
> > >>>
> > >>> 6. MQTT streamer
> > >>>> https://issues.apache.org/jira/browse/IGNITE-535
> > >>>>
> > >>> Yes, it was merged to master before the ignite-1.5 was created.
> > >>>
> > >>> I'd like to add:
> > >>>
> > >>> Camel Streamer => https://issues.apache.org/jira/browse/IGNITE-1790
> > >>> -- I'll merge this as soon as I finished with the OSGi tickets with
> > >>> demand.
> > >>>
> > >>> OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI (or
> > >>> whatever agreement we arrive to in mailing lists and Wiki)
> > >>> -- https://issues.apache.org/jira/browse/IGNITE-1527
> > >>> -- https://issues.apache.org/jira/browse/IGNITE-1877
> > >>> -- I'm working actively on these two features.
> > >>>
> > >>> *Raúl Kripalani*
> > >>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> > and
> > >>> Messaging Engineer
> > >>> http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > >>> http://blog.raulkr.net | twitter: @raulvk
> > >>>
> > >>> On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]>
> > >>> wrote:
> > >>>
> > >>> Guys,
> > >>>>
> > >>>> I think we can start preparation to Ignite-1.5 release which will
> > >>>> include
> > >>>> many interesting features:
> > >>>>
> > >>>> 1. Portable object API
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1486
> > >>>>
> > >>>> 2. Ignite.NET and Ignite C++
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1282
> > >>>>
> > >>>> 3. Optimistic serializable transactions
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1607
> > >>>>
> > >>>> 4. Distributed SQL joins - we will be able to query non-collocated
> > data
> > >>>>
> > >>> as
> > >>>
> > >>>> well
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1232
> > >>>>
> > >>>> 5. Enhanced Oracle and IBM JDK interoperability
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1526
> > >>>>
> > >>>> 6. MQTT streamer
> > >>>> https://issues.apache.org/jira/browse/IGNITE-535
> > >>>>
> > >>>> 7. Continuous query failover
> > >>>> https://issues.apache.org/jira/browse/IGNITE-426
> > >>>>
> > >>>> 8. Significant transactional cache performance optimizations - I
> will
> > >>>>
> > >>> merge
> > >>>
> > >>>> these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
> > >>>>
> > >>>> 9. Many stability and fault-tolerance fixes.
> > >>>>
> > >>>> 10. I would also like to include distributed Semaphore. Vladislav,
> any
> > >>>> chance you can finish with it this week?
> > >>>> https://issues.apache.org/jira/browse/IGNITE-
> > >>>> <https://issues.apache.org/jira/browse/IGNITE-426>638
> > >>>>
> > >>>> Thanks to everyone involved! Guys, esp. assignees of mentioned
> issues,
> > >>>> please respond to this email and let us know when can we expect your
> > >>>> changes being merged to master and release branch?
> > >>>>
> > >>>> Can someone create ignite-1.5 release branch?
> > >>>>
> > >>>> --Yakov
> > >>>>
> > >>>>
> > >
> >
>

[Denis Magda]

Hi Vladislav,

Most likely there is a minor issue in your test.

I think Yakov would be able to run and check the test as a part of the
ongoing review.

Thanks,

Denis

On Friday, November 13, 2015, Vladisav Jelisavcic <[hidden email]>
wrote:

> Hi Denis,
>
> Thanks a lot, it looks like my test setup was wrong,
> I added semaphore tests to GridCacheAbstractDataStructuresFailoverSelfTest
> suite.
> Now I have following problem:
> when I run tests with TOP_CHANGE_THREAD_CNT = 3
> tests fail when they reach stop() with the following exception:
>
> class org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is
> stopping: 09c5e8b8-8998-468e-960d-223220354fd3
> at
>
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:382)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:113)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:946)
> at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1823)
> at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1769)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2133)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2096)
> at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:314)
> at org.apache.ignite.Ignition.stop(Ignition.java:223)
> at
>
> org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:802)
> at
>
> org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:784)
> at
>
> org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest.access$500(GridCacheAbstractDataStructuresFailoverSelfTest.java:54)
> at
>
> org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest$5.apply(GridCacheAbstractDataStructuresFailoverSelfTest.java:459)
>
> When I run tests with TOP_CHANGE_THEAD_CNT = 1
> everything is running ok;
>
>
> @Yakov
> I made a new commit to my IGNITE-638 branch,
> can you please take a look?
>
>
>
> Best regards,
> Vladisav
>
>
>
>
> > On Wed, Nov 11, 2015 at 3:48 PM, Denis Magda <[hidden email]
> <javascript:;>> wrote:
> >
> > > Hi Vladislav,
> > >
> > > Please see below..
> > >
> > >
> > > On 11/11/2015 12:33 PM, Vladisav Jelisavcic wrote:
> > >
> > >> Yakov,
> > >>
> > >> sorry  for running a bit late.
> > >>
> > >> Vladislav, do you have any updates for
> > >>> https://issues.apache.org/jira/browse/IGNITE-638? Or any questions?
> > >>>
> > >>> --Yakov
> > >>>
> > >> I have problems with some fail-over scenarios;
> > >> It seems that if the two nodes are in the middle of acquiring or
> > releasing
> > >> the semaphore,
> > >> and one of them fails, all nodes get:
> > >>
> > >> [09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
> > >> <ignite-atomics-sys-cache> Failed to compare and set:
> > >>
> o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
> > >> class
> > org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> > >> Failed to acquire lock for keys (primary node left grid, retry
> > transaction
> > >> if possible) [keys=[UserKeyCacheObjectImpl
> [val=GridCacheInternalKeyImpl
> > >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> > >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> > >> ...
> > >> Caused by: class
> > >> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> > Failed
> > >> to acquire lock for keys (primary node left grid, retry transaction if
> > >> possible) [keys=[UserKeyCacheObjectImpl [val=GridCacheInternalKeyImpl
> > >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> > >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> > >> at
> > >>
> > >>
> >
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
> > >> ... 10 more
> > >>
> > > You have to process this exception manually at your implementation
> layer
> > > since your data structure uses a transactional cache.
> > > Below is a kind of template I used when it was required to process this
> > > and some other exeptions. You can use it as-is.
> > >
> > > int retries = GridCacheAdapter.MAX_RETRIES;
> > >
> > > IgniteCheckedException err =null;
> > >
> > > for (int i =0; i < retries; i++) {
> > >     try {
> > >         //Your transactional code that may fail
> > >     }
> > >     catch (IgniteCheckedException e) {
> > >         if (i == retries)
> > >             throw e;
> > >
> > >         if (X.hasCause(e, ClusterTopologyCheckedException.class)) {
> > >             ClusterTopologyCheckedException topErr =
> > > e.getCause(ClusterTopologyCheckedException.class);
> > >
> > >             topErr.retryReadyFuture().get();
> > >         }
> > >         else if (X.hasCause(e, IgniteTxRollbackCheckedException.class))
> > >             U.sleep(1);
> > >         else throw e;
> > >     }
> > > }
> > >
> > >
> > >
> > >> I'm still trying to find out how to exactly reproduce this behavior,
> > >> I'll send you more details once I try few more things.
> > >>
> > > There is the test suite called
> > > GridCacheAbstractDataStructuresFailoverSelfTest that checks Ignite
> > atomics
> > > and data structures with fail-over scenario.
> > > The suite will let you reproduce ClusterTopologyCheckedException
> easily.
> > > Just add your tests there referring to the tests of other data
> > structures.
> > >
> > > Presently I'm improving this test suite under my work on IGNITE-801 and
> > > IGNITE-803. If you finish your task earlier then I'll adopt your tests
> > to a
> > > new test approach.
> > >
> > >
> > >> I am still using partitioned cache, does it make sense to use
> replicated
> > >> cache instead?
> > >>
> > >> Yeah, you should support this as well. Cache mode for the data
> > structures
> > > is changed using CollectionConfigurations while for atomics using
> > > AtomicsConfiguration.
> > >
> > > --
> > > Denis
> > >
> > >
> > > Other than that, I'm done with everything else.
> > >>
> > >> Thanks,
> > >> Vladisav
> > >>
> > >>
> > >>
> > >> On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]
> <javascript:;>>
> > wrote:
> > >>
> > >> Sorry I haven't made an appearance in this thread yet.
> > >>>
> > >>> 6. MQTT streamer
> > >>>> https://issues.apache.org/jira/browse/IGNITE-535
> > >>>>
> > >>> Yes, it was merged to master before the ignite-1.5 was created.
> > >>>
> > >>> I'd like to add:
> > >>>
> > >>> Camel Streamer => https://issues.apache.org/jira/browse/IGNITE-1790
> > >>> -- I'll merge this as soon as I finished with the OSGi tickets with
> > >>> demand.
> > >>>
> > >>> OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI (or
> > >>> whatever agreement we arrive to in mailing lists and Wiki)
> > >>> -- https://issues.apache.org/jira/browse/IGNITE-1527
> > >>> -- https://issues.apache.org/jira/browse/IGNITE-1877
> > >>> -- I'm working actively on these two features.
> > >>>
> > >>> *Raúl Kripalani*
> > >>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> > and
> > >>> Messaging Engineer
> > >>> http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > >>> http://blog.raulkr.net | twitter: @raulvk
> > >>>
> > >>> On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]
> <javascript:;>>
> > >>> wrote:
> > >>>
> > >>> Guys,
> > >>>>
> > >>>> I think we can start preparation to Ignite-1.5 release which will
> > >>>> include
> > >>>> many interesting features:
> > >>>>
> > >>>> 1. Portable object API
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1486
> > >>>>
> > >>>> 2. Ignite.NET and Ignite C++
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1282
> > >>>>
> > >>>> 3. Optimistic serializable transactions
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1607
> > >>>>
> > >>>> 4. Distributed SQL joins - we will be able to query non-collocated
> > data
> > >>>>
> > >>> as
> > >>>
> > >>>> well
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1232
> > >>>>
> > >>>> 5. Enhanced Oracle and IBM JDK interoperability
> > >>>> https://issues.apache.org/jira/browse/IGNITE-1526
> > >>>>
> > >>>> 6. MQTT streamer
> > >>>> https://issues.apache.org/jira/browse/IGNITE-535
> > >>>>
> > >>>> 7. Continuous query failover
> > >>>> https://issues.apache.org/jira/browse/IGNITE-426
> > >>>>
> > >>>> 8. Significant transactional cache performance optimizations - I
> will
> > >>>>
> > >>> merge
> > >>>
> > >>>> these changes from 'ignite-1.4-slow-server-debug' today or tomorrow.
> > >>>>
> > >>>> 9. Many stability and fault-tolerance fixes.
> > >>>>
> > >>>> 10. I would also like to include distributed Semaphore. Vladislav,
> any
> > >>>> chance you can finish with it this week?
> > >>>> https://issues.apache.org/jira/browse/IGNITE-
> > >>>> <https://issues.apache.org/jira/browse/IGNITE-426>638
> > >>>>
> > >>>> Thanks to everyone involved! Guys, esp. assignees of mentioned
> issues,
> > >>>> please respond to this email and let us know when can we expect your
> > >>>> changes being merged to master and release branch?
> > >>>>
> > >>>> Can someone create ignite-1.5 release branch?
> > >>>>
> > >>>> --Yakov
> > >>>>
> > >>>>
> > >
> >
>

[yzhdanov]

Guys,

Can everyone provide updates on ongoing issues status?

Vlad, I will review Semaphore tomorrow (Russian time).

Thanks!

--Yakov

2015-11-14 15:38 GMT+03:00 Denis Magda <[hidden email]>:

> Hi Vladislav,
>
> Most likely there is a minor issue in your test.
>
> I think Yakov would be able to run and check the test as a part of the
> ongoing review.
>
> Thanks,
>
> Denis
>
> On Friday, November 13, 2015, Vladisav Jelisavcic <[hidden email]>
> wrote:
>
> > Hi Denis,
> >
> > Thanks a lot, it looks like my test setup was wrong,
> > I added semaphore tests to
> GridCacheAbstractDataStructuresFailoverSelfTest
> > suite.
> > Now I have following problem:
> > when I run tests with TOP_CHANGE_THREAD_CNT = 3
> > tests fail when they reach stop() with the following exception:
> >
> > class org.apache.ignite.internal.IgniteInterruptedCheckedException: Node
> is
> > stopping: 09c5e8b8-8998-468e-960d-223220354fd3
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:382)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:113)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:946)
> > at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1823)
> > at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1769)
> > at
> >
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2133)
> > at
> >
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2096)
> > at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:314)
> > at org.apache.ignite.Ignition.stop(Ignition.java:223)
> > at
> >
> >
> org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:802)
> > at
> >
> >
> org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:784)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest.access$500(GridCacheAbstractDataStructuresFailoverSelfTest.java:54)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest$5.apply(GridCacheAbstractDataStructuresFailoverSelfTest.java:459)
> >
> > When I run tests with TOP_CHANGE_THEAD_CNT = 1
> > everything is running ok;
> >
> >
> > @Yakov
> > I made a new commit to my IGNITE-638 branch,
> > can you please take a look?
> >
> >
> >
> > Best regards,
> > Vladisav
> >
> >
> >
> >
> > > On Wed, Nov 11, 2015 at 3:48 PM, Denis Magda <[hidden email]
> > <javascript:;>> wrote:
> > >
> > > > Hi Vladislav,
> > > >
> > > > Please see below..
> > > >
> > > >
> > > > On 11/11/2015 12:33 PM, Vladisav Jelisavcic wrote:
> > > >
> > > >> Yakov,
> > > >>
> > > >> sorry  for running a bit late.
> > > >>
> > > >> Vladislav, do you have any updates for
> > > >>> https://issues.apache.org/jira/browse/IGNITE-638? Or any
> questions?
> > > >>>
> > > >>> --Yakov
> > > >>>
> > > >> I have problems with some fail-over scenarios;
> > > >> It seems that if the two nodes are in the middle of acquiring or
> > > releasing
> > > >> the semaphore,
> > > >> and one of them fails, all nodes get:
> > > >>
> > > >> [09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
> > > >> <ignite-atomics-sys-cache> Failed to compare and set:
> > > >>
> > o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
> > > >> class
> > > org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> > > >> Failed to acquire lock for keys (primary node left grid, retry
> > > transaction
> > > >> if possible) [keys=[UserKeyCacheObjectImpl
> > [val=GridCacheInternalKeyImpl
> > > >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> > > >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> > > >> ...
> > > >> Caused by: class
> > > >> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> > > Failed
> > > >> to acquire lock for keys (primary node left grid, retry transaction
> if
> > > >> possible) [keys=[UserKeyCacheObjectImpl
> [val=GridCacheInternalKeyImpl
> > > >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> > > >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> > > >> at
> > > >>
> > > >>
> > >
> >
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
> > > >> ... 10 more
> > > >>
> > > > You have to process this exception manually at your implementation
> > layer
> > > > since your data structure uses a transactional cache.
> > > > Below is a kind of template I used when it was required to process
> this
> > > > and some other exeptions. You can use it as-is.
> > > >
> > > > int retries = GridCacheAdapter.MAX_RETRIES;
> > > >
> > > > IgniteCheckedException err =null;
> > > >
> > > > for (int i =0; i < retries; i++) {
> > > >     try {
> > > >         //Your transactional code that may fail
> > > >     }
> > > >     catch (IgniteCheckedException e) {
> > > >         if (i == retries)
> > > >             throw e;
> > > >
> > > >         if (X.hasCause(e, ClusterTopologyCheckedException.class)) {
> > > >             ClusterTopologyCheckedException topErr =
> > > > e.getCause(ClusterTopologyCheckedException.class);
> > > >
> > > >             topErr.retryReadyFuture().get();
> > > >         }
> > > >         else if (X.hasCause(e,
> IgniteTxRollbackCheckedException.class))
> > > >             U.sleep(1);
> > > >         else throw e;
> > > >     }
> > > > }
> > > >
> > > >
> > > >
> > > >> I'm still trying to find out how to exactly reproduce this behavior,
> > > >> I'll send you more details once I try few more things.
> > > >>
> > > > There is the test suite called
> > > > GridCacheAbstractDataStructuresFailoverSelfTest that checks Ignite
> > > atomics
> > > > and data structures with fail-over scenario.
> > > > The suite will let you reproduce ClusterTopologyCheckedException
> > easily.
> > > > Just add your tests there referring to the tests of other data
> > > structures.
> > > >
> > > > Presently I'm improving this test suite under my work on IGNITE-801
> and
> > > > IGNITE-803. If you finish your task earlier then I'll adopt your
> tests
> > > to a
> > > > new test approach.
> > > >
> > > >
> > > >> I am still using partitioned cache, does it make sense to use
> > replicated
> > > >> cache instead?
> > > >>
> > > >> Yeah, you should support this as well. Cache mode for the data
> > > structures
> > > > is changed using CollectionConfigurations while for atomics using
> > > > AtomicsConfiguration.
> > > >
> > > > --
> > > > Denis
> > > >
> > > >
> > > > Other than that, I'm done with everything else.
> > > >>
> > > >> Thanks,
> > > >> Vladisav
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]
> > <javascript:;>>
> > > wrote:
> > > >>
> > > >> Sorry I haven't made an appearance in this thread yet.
> > > >>>
> > > >>> 6. MQTT streamer
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-535
> > > >>>>
> > > >>> Yes, it was merged to master before the ignite-1.5 was created.
> > > >>>
> > > >>> I'd like to add:
> > > >>>
> > > >>> Camel Streamer =>
> https://issues.apache.org/jira/browse/IGNITE-1790
> > > >>> -- I'll merge this as soon as I finished with the OSGi tickets with
> > > >>> demand.
> > > >>>
> > > >>> OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI
> (or
> > > >>> whatever agreement we arrive to in mailing lists and Wiki)
> > > >>> -- https://issues.apache.org/jira/browse/IGNITE-1527
> > > >>> -- https://issues.apache.org/jira/browse/IGNITE-1877
> > > >>> -- I'm working actively on these two features.
> > > >>>
> > > >>> *Raúl Kripalani*
> > > >>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data
> > > and
> > > >>> Messaging Engineer
> > > >>> http://about.me/raulkripalani |
> > > http://www.linkedin.com/in/raulkripalani
> > > >>> http://blog.raulkr.net | twitter: @raulvk
> > > >>>
> > > >>> On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <[hidden email]
> > <javascript:;>>
> > > >>> wrote:
> > > >>>
> > > >>> Guys,
> > > >>>>
> > > >>>> I think we can start preparation to Ignite-1.5 release which will
> > > >>>> include
> > > >>>> many interesting features:
> > > >>>>
> > > >>>> 1. Portable object API
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-1486
> > > >>>>
> > > >>>> 2. Ignite.NET and Ignite C++
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-1282
> > > >>>>
> > > >>>> 3. Optimistic serializable transactions
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-1607
> > > >>>>
> > > >>>> 4. Distributed SQL joins - we will be able to query non-collocated
> > > data
> > > >>>>
> > > >>> as
> > > >>>
> > > >>>> well
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-1232
> > > >>>>
> > > >>>> 5. Enhanced Oracle and IBM JDK interoperability
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-1526
> > > >>>>
> > > >>>> 6. MQTT streamer
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-535
> > > >>>>
> > > >>>> 7. Continuous query failover
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-426
> > > >>>>
> > > >>>> 8. Significant transactional cache performance optimizations - I
> > will
> > > >>>>
> > > >>> merge
> > > >>>
> > > >>>> these changes from 'ignite-1.4-slow-server-debug' today or
> tomorrow.
> > > >>>>
> > > >>>> 9. Many stability and fault-tolerance fixes.
> > > >>>>
> > > >>>> 10. I would also like to include distributed Semaphore. Vladislav,
> > any
> > > >>>> chance you can finish with it this week?
> > > >>>> https://issues.apache.org/jira/browse/IGNITE-
> > > >>>> <https://issues.apache.org/jira/browse/IGNITE-426>638
> > > >>>>
> > > >>>> Thanks to everyone involved! Guys, esp. assignees of mentioned
> > issues,
> > > >>>> please respond to this email and let us know when can we expect
> your
> > > >>>> changes being merged to master and release branch?
> > > >>>>
> > > >>>> Can someone create ignite-1.5 release branch?
> > > >>>>
> > > >>>> --Yakov
> > > >>>>
> > > >>>>
> > > >
> > >
> >
>

[Alexey Goncharuk]

Igniters,

I am finalizing my optimizations in tx cache (currently there are a couple
of failing tests on CI), I am expecting these changes to be merged in
ignite-1.5 branch tomorrow.

2015-11-16 19:23 GMT+03:00 Yakov Zhdanov <[hidden email]>:

> Guys,
>
> Can everyone provide updates on ongoing issues status?
>
> Vlad, I will review Semaphore tomorrow (Russian time).
>
> Thanks!
>
> --Yakov
>
> 2015-11-14 15:38 GMT+03:00 Denis Magda <[hidden email]>:
>
> > Hi Vladislav,
> >
> > Most likely there is a minor issue in your test.
> >
> > I think Yakov would be able to run and check the test as a part of the
> > ongoing review.
> >
> > Thanks,
> >
> > Denis
> >
> > On Friday, November 13, 2015, Vladisav Jelisavcic <[hidden email]>
> > wrote:
> >
> > > Hi Denis,
> > >
> > > Thanks a lot, it looks like my test setup was wrong,
> > > I added semaphore tests to
> > GridCacheAbstractDataStructuresFailoverSelfTest
> > > suite.
> > > Now I have following problem:
> > > when I run tests with TOP_CHANGE_THREAD_CNT = 3
> > > tests fail when they reach stop() with the following exception:
> > >
> > > class org.apache.ignite.internal.IgniteInterruptedCheckedException:
> Node
> > is
> > > stopping: 09c5e8b8-8998-468e-960d-223220354fd3
> > > at
> > >
> > >
> >
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:382)
> > > at
> > >
> > >
> >
> org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:113)
> > > at
> > >
> > >
> >
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:946)
> > > at
> org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1823)
> > > at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1769)
> > > at
> > >
> > >
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2133)
> > > at
> > >
> > >
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2096)
> > > at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:314)
> > > at org.apache.ignite.Ignition.stop(Ignition.java:223)
> > > at
> > >
> > >
> >
> org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:802)
> > > at
> > >
> > >
> >
> org.apache.ignite.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:784)
> > > at
> > >
> > >
> >
> org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest.access$500(GridCacheAbstractDataStructuresFailoverSelfTest.java:54)
> > > at
> > >
> > >
> >
> org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest$5.apply(GridCacheAbstractDataStructuresFailoverSelfTest.java:459)
> > >
> > > When I run tests with TOP_CHANGE_THEAD_CNT = 1
> > > everything is running ok;
> > >
> > >
> > > @Yakov
> > > I made a new commit to my IGNITE-638 branch,
> > > can you please take a look?
> > >
> > >
> > >
> > > Best regards,
> > > Vladisav
> > >
> > >
> > >
> > >
> > > > On Wed, Nov 11, 2015 at 3:48 PM, Denis Magda <[hidden email]
> > > <javascript:;>> wrote:
> > > >
> > > > > Hi Vladislav,
> > > > >
> > > > > Please see below..
> > > > >
> > > > >
> > > > > On 11/11/2015 12:33 PM, Vladisav Jelisavcic wrote:
> > > > >
> > > > >> Yakov,
> > > > >>
> > > > >> sorry  for running a bit late.
> > > > >>
> > > > >> Vladislav, do you have any updates for
> > > > >>> https://issues.apache.org/jira/browse/IGNITE-638? Or any
> > questions?
> > > > >>>
> > > > >>> --Yakov
> > > > >>>
> > > > >> I have problems with some fail-over scenarios;
> > > > >> It seems that if the two nodes are in the middle of acquiring or
> > > > releasing
> > > > >> the semaphore,
> > > > >> and one of them fails, all nodes get:
> > > > >>
> > > > >>
> [09:36:38,509][ERROR][ignite-#13%pub-null%][GridCacheSemaphoreImpl]
> > > > >> <ignite-atomics-sys-cache> Failed to compare and set:
> > > > >>
> > >
> o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$1@5528b728
> > > > >> class
> > > > org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> > > > >> Failed to acquire lock for keys (primary node left grid, retry
> > > > transaction
> > > > >> if possible) [keys=[UserKeyCacheObjectImpl
> > > [val=GridCacheInternalKeyImpl
> > > > >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> > > > >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> > > > >> ...
> > > > >> Caused by: class
> > > > >>
> org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
> > > > Failed
> > > > >> to acquire lock for keys (primary node left grid, retry
> transaction
> > if
> > > > >> possible) [keys=[UserKeyCacheObjectImpl
> > [val=GridCacheInternalKeyImpl
> > > > >> [name=ac83b8cb-3052-49a6-9301-81b20b0ecf3a], hasValBytes=true]],
> > > > >> node=c321fcc4-5db5-4b03-9811-6a5587f2c253]
> > > > >> at
> > > > >>
> > > > >>
> > > >
> > >
> >
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.newTopologyException(GridDhtColocatedLockFuture.java:1199)
> > > > >> ... 10 more
> > > > >>
> > > > > You have to process this exception manually at your implementation
> > > layer
> > > > > since your data structure uses a transactional cache.
> > > > > Below is a kind of template I used when it was required to process
> > this
> > > > > and some other exeptions. You can use it as-is.
> > > > >
> > > > > int retries = GridCacheAdapter.MAX_RETRIES;
> > > > >
> > > > > IgniteCheckedException err =null;
> > > > >
> > > > > for (int i =0; i < retries; i++) {
> > > > >     try {
> > > > >         //Your transactional code that may fail
> > > > >     }
> > > > >     catch (IgniteCheckedException e) {
> > > > >         if (i == retries)
> > > > >             throw e;
> > > > >
> > > > >         if (X.hasCause(e, ClusterTopologyCheckedException.class)) {
> > > > >             ClusterTopologyCheckedException topErr =
> > > > > e.getCause(ClusterTopologyCheckedException.class);
> > > > >
> > > > >             topErr.retryReadyFuture().get();
> > > > >         }
> > > > >         else if (X.hasCause(e,
> > IgniteTxRollbackCheckedException.class))
> > > > >             U.sleep(1);
> > > > >         else throw e;
> > > > >     }
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > >> I'm still trying to find out how to exactly reproduce this
> behavior,
> > > > >> I'll send you more details once I try few more things.
> > > > >>
> > > > > There is the test suite called
> > > > > GridCacheAbstractDataStructuresFailoverSelfTest that checks Ignite
> > > > atomics
> > > > > and data structures with fail-over scenario.
> > > > > The suite will let you reproduce ClusterTopologyCheckedException
> > > easily.
> > > > > Just add your tests there referring to the tests of other data
> > > > structures.
> > > > >
> > > > > Presently I'm improving this test suite under my work on IGNITE-801
> > and
> > > > > IGNITE-803. If you finish your task earlier then I'll adopt your
> > tests
> > > > to a
> > > > > new test approach.
> > > > >
> > > > >
> > > > >> I am still using partitioned cache, does it make sense to use
> > > replicated
> > > > >> cache instead?
> > > > >>
> > > > >> Yeah, you should support this as well. Cache mode for the data
> > > > structures
> > > > > is changed using CollectionConfigurations while for atomics using
> > > > > AtomicsConfiguration.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > >
> > > > > Other than that, I'm done with everything else.
> > > > >>
> > > > >> Thanks,
> > > > >> Vladisav
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Nov 10, 2015 at 7:19 PM, Raul Kripalani <[hidden email]
> > > <javascript:;>>
> > > > wrote:
> > > > >>
> > > > >> Sorry I haven't made an appearance in this thread yet.
> > > > >>>
> > > > >>> 6. MQTT streamer
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-535
> > > > >>>>
> > > > >>> Yes, it was merged to master before the ignite-1.5 was created.
> > > > >>>
> > > > >>> I'd like to add:
> > > > >>>
> > > > >>> Camel Streamer =>
> > https://issues.apache.org/jira/browse/IGNITE-1790
> > > > >>> -- I'll merge this as soon as I finished with the OSGi tickets
> with
> > > > >>> demand.
> > > > >>>
> > > > >>> OSGi Manifests, Karaf features and possible ClassLoaderCodec SPI
> > (or
> > > > >>> whatever agreement we arrive to in mailing lists and Wiki)
> > > > >>> -- https://issues.apache.org/jira/browse/IGNITE-1527
> > > > >>> -- https://issues.apache.org/jira/browse/IGNITE-1877
> > > > >>> -- I'm working actively on these two features.
> > > > >>>
> > > > >>> *Raúl Kripalani*
> > > > >>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> > Data
> > > > and
> > > > >>> Messaging Engineer
> > > > >>> http://about.me/raulkripalani |
> > > > http://www.linkedin.com/in/raulkripalani
> > > > >>> http://blog.raulkr.net | twitter: @raulvk
> > > > >>>
> > > > >>> On Mon, Nov 2, 2015 at 1:35 PM, Yakov Zhdanov <
> [hidden email]
> > > <javascript:;>>
> > > > >>> wrote:
> > > > >>>
> > > > >>> Guys,
> > > > >>>>
> > > > >>>> I think we can start preparation to Ignite-1.5 release which
> will
> > > > >>>> include
> > > > >>>> many interesting features:
> > > > >>>>
> > > > >>>> 1. Portable object API
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-1486
> > > > >>>>
> > > > >>>> 2. Ignite.NET and Ignite C++
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-1282
> > > > >>>>
> > > > >>>> 3. Optimistic serializable transactions
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-1607
> > > > >>>>
> > > > >>>> 4. Distributed SQL joins - we will be able to query
> non-collocated
> > > > data
> > > > >>>>
> > > > >>> as
> > > > >>>
> > > > >>>> well
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-1232
> > > > >>>>
> > > > >>>> 5. Enhanced Oracle and IBM JDK interoperability
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-1526
> > > > >>>>
> > > > >>>> 6. MQTT streamer
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-535
> > > > >>>>
> > > > >>>> 7. Continuous query failover
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-426
> > > > >>>>
> > > > >>>> 8. Significant transactional cache performance optimizations - I
> > > will
> > > > >>>>
> > > > >>> merge
> > > > >>>
> > > > >>>> these changes from 'ignite-1.4-slow-server-debug' today or
> > tomorrow.
> > > > >>>>
> > > > >>>> 9. Many stability and fault-tolerance fixes.
> > > > >>>>
> > > > >>>> 10. I would also like to include distributed Semaphore.
> Vladislav,
> > > any
> > > > >>>> chance you can finish with it this week?
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-
> > > > >>>> <https://issues.apache.org/jira/browse/IGNITE-426>638
> > > > >>>>
> > > > >>>> Thanks to everyone involved! Guys, esp. assignees of mentioned
> > > issues,
> > > > >>>> please respond to this email and let us know when can we expect
> > your
> > > > >>>> changes being merged to master and release branch?
> > > > >>>>
> > > > >>>> Can someone create ignite-1.5 release branch?
> > > > >>>>
> > > > >>>> --Yakov
> > > > >>>>
> > > > >>>>
> > > > >
> > > >
> > >
> >
>