Cache 6 suite is broken

[Ivan Pavlukhin]

Hi,

I noticied that Cache 6 suite finishes badly (exit code 137) almost
everytime [1] on master. It started recently. A problematic test
inside is IgniteCache150ClientsTest. I made some attempts to fix the
test (including reverting recent commits), but had no luck. And a hard
part here is that I do not know whether the issue was caused by code
changes or it is infrastructural one. My best attempt was increasing
NetworkTimeout in test IgniteConfiguration [2]. After that the suite
was able to finish, but IgniteCache150ClientsTest failed. Also, the
problematic test seems so flaky, there are many failures in history.

Do not have any really good solution in my mind. I see following options:
1. Use workaround in PR [2].
2. Ignore test and dig deeper in scope of a separate ticket.

What do you think?

[1] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
[2] https://github.com/apache/ignite/pull/7142/files
--
Best regards,
Ivan Pavlukhin

[Ivan Pavlukhin]

An update. After increasing NetworkTimeout [1] the test passes rarely [2].

[1] https://github.com/apache/ignite/pull/7142/files
[2] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview

пн, 16 дек. 2019 г. в 09:07, Ivan Pavlukhin <[hidden email]>:

>
> Hi,
>
> I noticied that Cache 6 suite finishes badly (exit code 137) almost
> everytime [1] on master. It started recently. A problematic test
> inside is IgniteCache150ClientsTest. I made some attempts to fix the
> test (including reverting recent commits), but had no luck. And a hard
> part here is that I do not know whether the issue was caused by code
> changes or it is infrastructural one. My best attempt was increasing
> NetworkTimeout in test IgniteConfiguration [2]. After that the suite
> was able to finish, but IgniteCache150ClientsTest failed. Also, the
> problematic test seems so flaky, there are many failures in history.
>
> Do not have any really good solution in my mind. I see following options:
> 1. Use workaround in PR [2].
> 2. Ignore test and dig deeper in scope of a separate ticket.
>
> What do you think?
>
> [1] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> [2] https://github.com/apache/ignite/pull/7142/files
> --
> Best regards,
> Ivan Pavlukhin

--
Best regards,
Ivan Pavlukhin

[Ivan Pavlukhin]

Igniters,

I run tests several times for my PR [1] and results [2] looks somehow
better than in master [3]. But the problematic test seems too flaky.
So, I cannot decide between 2 options:
1. Apply PR [1].
2. Ignore the problematic test.
(Unfortunately do not have a chance to investigate and fix)

Someone's opinion would be helpful here.

[1] https://github.com/apache/ignite/pull/7142
[2] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
[3] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview

пн, 16 дек. 2019 г. в 13:38, Ivan Pavlukhin <[hidden email]>:

>
> An update. After increasing NetworkTimeout [1] the test passes rarely [2].
>
> [1] https://github.com/apache/ignite/pull/7142/files
> [2] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
>
> пн, 16 дек. 2019 г. в 09:07, Ivan Pavlukhin <[hidden email]>:
> >
> > Hi,
> >
> > I noticied that Cache 6 suite finishes badly (exit code 137) almost
> > everytime [1] on master. It started recently. A problematic test
> > inside is IgniteCache150ClientsTest. I made some attempts to fix the
> > test (including reverting recent commits), but had no luck. And a hard
> > part here is that I do not know whether the issue was caused by code
> > changes or it is infrastructural one. My best attempt was increasing
> > NetworkTimeout in test IgniteConfiguration [2]. After that the suite
> > was able to finish, but IgniteCache150ClientsTest failed. Also, the
> > problematic test seems so flaky, there are many failures in history.
> >
> > Do not have any really good solution in my mind. I see following options:
> > 1. Use workaround in PR [2].
> > 2. Ignore test and dig deeper in scope of a separate ticket.
> >
> > What do you think?
> >
> > [1] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> > [2] https://github.com/apache/ignite/pull/7142/files
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
> --
> Best regards,
> Ivan Pavlukhin

--
Best regards,
Ivan Pavlukhin

[Dmitry Pavlov]

Hi Ivan, Igniters,

My proposal is to apply PR and take a look to wider statistics in the
master branch. WDYT?

Sincerely,
Dmitriy Pavlov

ср, 18 дек. 2019 г. в 09:36, Ivan Pavlukhin <[hidden email]>:

> Igniters,
>
> I run tests several times for my PR [1] and results [2] looks somehow
> better than in master [3]. But the problematic test seems too flaky.
> So, I cannot decide between 2 options:
> 1. Apply PR [1].
> 2. Ignore the problematic test.
> (Unfortunately do not have a chance to investigate and fix)
>
> Someone's opinion would be helpful here.
>
> [1] https://github.com/apache/ignite/pull/7142
> [2]
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
> [3]
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
>
> пн, 16 дек. 2019 г. в 13:38, Ivan Pavlukhin <[hidden email]>:
> >
> > An update. After increasing NetworkTimeout [1] the test passes rarely
> [2].
> >
> > [1] https://github.com/apache/ignite/pull/7142/files
> > [2]
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
> >
> > пн, 16 дек. 2019 г. в 09:07, Ivan Pavlukhin <[hidden email]>:
> > >
> > > Hi,
> > >
> > > I noticied that Cache 6 suite finishes badly (exit code 137) almost
> > > everytime [1] on master. It started recently. A problematic test
> > > inside is IgniteCache150ClientsTest. I made some attempts to fix the
> > > test (including reverting recent commits), but had no luck. And a hard
> > > part here is that I do not know whether the issue was caused by code
> > > changes or it is infrastructural one. My best attempt was increasing
> > > NetworkTimeout in test IgniteConfiguration [2]. After that the suite
> > > was able to finish, but IgniteCache150ClientsTest failed. Also, the
> > > problematic test seems so flaky, there are many failures in history.
> > >
> > > Do not have any really good solution in my mind. I see following
> options:
> > > 1. Use workaround in PR [2].
> > > 2. Ignore test and dig deeper in scope of a separate ticket.
> > >
> > > What do you think?
> > >
> > > [1]
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> > > [2] https://github.com/apache/ignite/pull/7142/files
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[Ivan Pavlukhin]

So, if nobody objects, I will merge PR [1] on Friday.

[1] https://github.com/apache/ignite/pull/7142

ср, 18 дек. 2019 г. в 13:13, Dmitriy Pavlov <[hidden email]>:

>
> Hi Ivan, Igniters,
>
> My proposal is to apply PR and take a look to wider statistics in the
> master branch. WDYT?
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 18 дек. 2019 г. в 09:36, Ivan Pavlukhin <[hidden email]>:
>
> > Igniters,
> >
> > I run tests several times for my PR [1] and results [2] looks somehow
> > better than in master [3]. But the problematic test seems too flaky.
> > So, I cannot decide between 2 options:
> > 1. Apply PR [1].
> > 2. Ignore the problematic test.
> > (Unfortunately do not have a chance to investigate and fix)
> >
> > Someone's opinion would be helpful here.
> >
> > [1] https://github.com/apache/ignite/pull/7142
> > [2]
> > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
> > [3]
> > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> >
> > пн, 16 дек. 2019 г. в 13:38, Ivan Pavlukhin <[hidden email]>:
> > >
> > > An update. After increasing NetworkTimeout [1] the test passes rarely
> > [2].
> > >
> > > [1] https://github.com/apache/ignite/pull/7142/files
> > > [2]
> > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
> > >
> > > пн, 16 дек. 2019 г. в 09:07, Ivan Pavlukhin <[hidden email]>:
> > > >
> > > > Hi,
> > > >
> > > > I noticied that Cache 6 suite finishes badly (exit code 137) almost
> > > > everytime [1] on master. It started recently. A problematic test
> > > > inside is IgniteCache150ClientsTest. I made some attempts to fix the
> > > > test (including reverting recent commits), but had no luck. And a hard
> > > > part here is that I do not know whether the issue was caused by code
> > > > changes or it is infrastructural one. My best attempt was increasing
> > > > NetworkTimeout in test IgniteConfiguration [2]. After that the suite
> > > > was able to finish, but IgniteCache150ClientsTest failed. Also, the
> > > > problematic test seems so flaky, there are many failures in history.
> > > >
> > > > Do not have any really good solution in my mind. I see following
> > options:
> > > > 1. Use workaround in PR [2].
> > > > 2. Ignore test and dig deeper in scope of a separate ticket.
> > > >
> > > > What do you think?
> > > >
> > > > [1]
> > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> > > > [2] https://github.com/apache/ignite/pull/7142/files
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >

--
Best regards,
Ivan Pavlukhin

[Ivan Pavlukhin]

FYI

I applied the patch.

ср, 18 дек. 2019 г. в 14:46, Ivan Pavlukhin <[hidden email]>:

>
> So, if nobody objects, I will merge PR [1] on Friday.
>
> [1] https://github.com/apache/ignite/pull/7142
>
> ср, 18 дек. 2019 г. в 13:13, Dmitriy Pavlov <[hidden email]>:
> >
> > Hi Ivan, Igniters,
> >
> > My proposal is to apply PR and take a look to wider statistics in the
> > master branch. WDYT?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > ср, 18 дек. 2019 г. в 09:36, Ivan Pavlukhin <[hidden email]>:
> >
> > > Igniters,
> > >
> > > I run tests several times for my PR [1] and results [2] looks somehow
> > > better than in master [3]. But the problematic test seems too flaky.
> > > So, I cannot decide between 2 options:
> > > 1. Apply PR [1].
> > > 2. Ignore the problematic test.
> > > (Unfortunately do not have a chance to investigate and fix)
> > >
> > > Someone's opinion would be helpful here.
> > >
> > > [1] https://github.com/apache/ignite/pull/7142
> > > [2]
> > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
> > > [3]
> > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> > >
> > > пн, 16 дек. 2019 г. в 13:38, Ivan Pavlukhin <[hidden email]>:
> > > >
> > > > An update. After increasing NetworkTimeout [1] the test passes rarely
> > > [2].
> > > >
> > > > [1] https://github.com/apache/ignite/pull/7142/files
> > > > [2]
> > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
> > > >
> > > > пн, 16 дек. 2019 г. в 09:07, Ivan Pavlukhin <[hidden email]>:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I noticied that Cache 6 suite finishes badly (exit code 137) almost
> > > > > everytime [1] on master. It started recently. A problematic test
> > > > > inside is IgniteCache150ClientsTest. I made some attempts to fix the
> > > > > test (including reverting recent commits), but had no luck. And a hard
> > > > > part here is that I do not know whether the issue was caused by code
> > > > > changes or it is infrastructural one. My best attempt was increasing
> > > > > NetworkTimeout in test IgniteConfiguration [2]. After that the suite
> > > > > was able to finish, but IgniteCache150ClientsTest failed. Also, the
> > > > > problematic test seems so flaky, there are many failures in history.
> > > > >
> > > > > Do not have any really good solution in my mind. I see following
> > > options:
> > > > > 1. Use workaround in PR [2].
> > > > > 2. Ignore test and dig deeper in scope of a separate ticket.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > [1]
> > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> > > > > [2] https://github.com/apache/ignite/pull/7142/files
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin

--
Best regards,
Ivan Pavlukhin

[Zhenya Stanilovsky]

Looks like i found problem root cause, first run completed on TC, hope it would be ok with other builds.
 

>
>
>------- Forwarded message -------
>From: "Ivan Pavlukhin" < [hidden email] >
>To: dev < [hidden email] >
>Cc:
>Subject: Re: Cache 6 suite is broken
>Date: Fri, 20 Dec 2019 10:04:07 +0300
>
>FYI
>
>I applied the patch.
>
>ср, 18 дек. 2019 г. в 14:46, Ivan Pavlukhin < [hidden email] >:
>>
>> So, if nobody objects, I will merge PR [1] on Friday.
>>
>> [1]  https://github.com/apache/ignite/pull/7142
>>
>> ср, 18 дек. 2019 г. в 13:13, Dmitriy Pavlov < [hidden email] >:
>> >
>> > Hi Ivan, Igniters,
>> >
>> > My proposal is to apply PR and take a look to wider statistics in the
>> > master branch. WDYT?
>> >
>> > Sincerely,
>> > Dmitriy Pavlov
>> >
>> > ср, 18 дек. 2019 г. в 09:36, Ivan Pavlukhin < [hidden email] >:
>> >
>> > > Igniters,
>> > >
>> > > I run tests several times for my PR [1] and results [2] looks somehow
>> > > better than in master [3]. But the problematic test seems too flaky.
>> > > So, I cannot decide between 2 options:
>> > > 1. Apply PR [1].
>> > > 2. Ignore the problematic test.
>> > > (Unfortunately do not have a chance to investigate and fix)
>> > >
>> > > Someone's opinion would be helpful here.
>> > >
>> > > [1]  https://github.com/apache/ignite/pull/7142
>> > > [2]
>> > >
>>  https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
>> > > [3]
>> > >
>>  https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
>> > >
>> > > пн, 16 дек. 2019 г. в 13:38, Ivan Pavlukhin < [hidden email] >:
>> > > >
>> > > > An update. After increasing NetworkTimeout [1] the test passes
>> rarely
>> > > [2].
>> > > >
>> > > > [1]  https://github.com/apache/ignite/pull/7142/files
>> > > > [2]
>> > >
>>  https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
>> > > >
>> > > > пн, 16 дек. 2019 г. в 09:07, Ivan Pavlukhin < [hidden email] >:
>> > > > >
>> > > > > Hi,
>> > > > >
>> > > > > I noticied that Cache 6 suite finishes badly (exit code 137)
>> almost
>> > > > > everytime [1] on master. It started recently. A problematic test
>> > > > > inside is IgniteCache150ClientsTest. I made some attempts to fix
>> the
>> > > > > test (including reverting recent commits), but had no luck. And
>> a hard
>> > > > > part here is that I do not know whether the issue was caused by
>> code
>> > > > > changes or it is infrastructural one. My best attempt was
>> increasing
>> > > > > NetworkTimeout in test IgniteConfiguration [2]. After that the
>> suite
>> > > > > was able to finish, but IgniteCache150ClientsTest failed. Also,
>> the
>> > > > > problematic test seems so flaky, there are many failures in
>> history.
>> > > > >
>> > > > > Do not have any really good solution in my mind. I see following
>> > > options:
>> > > > > 1. Use workaround in PR [2].
>> > > > > 2. Ignore test and dig deeper in scope of a separate ticket.
>> > > > >
>> > > > > What do you think?
>> > > > >
>> > > > > [1]
>> > >
>>  https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
>> > > > > [2]  https://github.com/apache/ignite/pull/7142/files
>> > > > > --
>> > > > > Best regards,
>> > > > > Ivan Pavlukhin
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best regards,
>> > > > Ivan Pavlukhin
>> > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > > Ivan Pavlukhin
>> > >
>>
>>
>>
>> --
>> Best regards,
>> Ivan Pavlukhin

 
 
 
 

[Ivan Pavlukhin]

Zhenya,

Good news! Could you please tell what is the root cause and how did you fix it?

вт, 28 янв. 2020 г. в 18:58, Zhenya Stanilovsky <[hidden email]>:

>
>
>
> Looks like i found problem root cause, first run completed on TC, hope it would be ok with other builds.
>
> >
> >
> >------- Forwarded message -------
> >From: "Ivan Pavlukhin" < [hidden email] >
> >To: dev < [hidden email] >
> >Cc:
> >Subject: Re: Cache 6 suite is broken
> >Date: Fri, 20 Dec 2019 10:04:07 +0300
> >
> >FYI
> >
> >I applied the patch.
> >
> >ср, 18 дек. 2019 г. в 14:46, Ivan Pavlukhin < [hidden email] >:
> >>
> >> So, if nobody objects, I will merge PR [1] on Friday.
> >>
> >> [1]  https://github.com/apache/ignite/pull/7142
> >>
> >> ср, 18 дек. 2019 г. в 13:13, Dmitriy Pavlov < [hidden email] >:
> >> >
> >> > Hi Ivan, Igniters,
> >> >
> >> > My proposal is to apply PR and take a look to wider statistics in the
> >> > master branch. WDYT?
> >> >
> >> > Sincerely,
> >> > Dmitriy Pavlov
> >> >
> >> > ср, 18 дек. 2019 г. в 09:36, Ivan Pavlukhin < [hidden email] >:
> >> >
> >> > > Igniters,
> >> > >
> >> > > I run tests several times for my PR [1] and results [2] looks somehow
> >> > > better than in master [3]. But the problematic test seems too flaky.
> >> > > So, I cannot decide between 2 options:
> >> > > 1. Apply PR [1].
> >> > > 2. Ignore the problematic test.
> >> > > (Unfortunately do not have a chance to investigate and fix)
> >> > >
> >> > > Someone's opinion would be helpful here.
> >> > >
> >> > > [1]  https://github.com/apache/ignite/pull/7142
> >> > > [2]
> >> > >
> >>  https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
> >> > > [3]
> >> > >
> >>  https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> >> > >
> >> > > пн, 16 дек. 2019 г. в 13:38, Ivan Pavlukhin < [hidden email] >:
> >> > > >
> >> > > > An update. After increasing NetworkTimeout [1] the test passes
> >> rarely
> >> > > [2].
> >> > > >
> >> > > > [1]  https://github.com/apache/ignite/pull/7142/files
> >> > > > [2]
> >> > >
> >>  https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=pull%2F7142%2Fhead&buildTypeTab=overview
> >> > > >
> >> > > > пн, 16 дек. 2019 г. в 09:07, Ivan Pavlukhin < [hidden email] >:
> >> > > > >
> >> > > > > Hi,
> >> > > > >
> >> > > > > I noticied that Cache 6 suite finishes badly (exit code 137)
> >> almost
> >> > > > > everytime [1] on master. It started recently. A problematic test
> >> > > > > inside is IgniteCache150ClientsTest. I made some attempts to fix
> >> the
> >> > > > > test (including reverting recent commits), but had no luck. And
> >> a hard
> >> > > > > part here is that I do not know whether the issue was caused by
> >> code
> >> > > > > changes or it is infrastructural one. My best attempt was
> >> increasing
> >> > > > > NetworkTimeout in test IgniteConfiguration [2]. After that the
> >> suite
> >> > > > > was able to finish, but IgniteCache150ClientsTest failed. Also,
> >> the
> >> > > > > problematic test seems so flaky, there are many failures in
> >> history.
> >> > > > >
> >> > > > > Do not have any really good solution in my mind. I see following
> >> > > options:
> >> > > > > 1. Use workaround in PR [2].
> >> > > > > 2. Ignore test and dig deeper in scope of a separate ticket.
> >> > > > >
> >> > > > > What do you think?
> >> > > > >
> >> > > > > [1]
> >> > >
> >>  https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache6?branch=%3Cdefault%3E&buildTypeTab=overview
> >> > > > > [2]  https://github.com/apache/ignite/pull/7142/files
> >> > > > > --
> >> > > > > Best regards,
> >> > > > > Ivan Pavlukhin
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Best regards,
> >> > > > Ivan Pavlukhin
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Best regards,
> >> > > Ivan Pavlukhin
> >> > >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >> Ivan Pavlukhin
>
>
>
>

--
Best regards,
Ivan Pavlukhin