Upsource lags and connection timeouts

[Anton Vinogradov-2]

Folks,

Who is responsible for Upsouce [1]?
I see a performance issues last week.
Can we check Upsource health?

[1] https://reviews.ignite.apache.org

[Anton Vinogradov-2]

Also, I see following on each resolve attempt.
UpsourceRequestExceptionImpl: 108: Internal error: An error occurred during
flushing data to database



вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:

> Folks,
>
> Who is responsible for Upsouce [1]?
> I see a performance issues last week.
> Can we check Upsource health?
>
> [1] https://reviews.ignite.apache.org
>

[Anton Vinogradov-2]

Still, see the same issues.

Denis,

Could you please prioritize the fix?

вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:

> Also, I see following on each resolve attempt.
> UpsourceRequestExceptionImpl: 108: Internal error: An error occurred
> during flushing data to database
>
>
>
> вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
>
>> Folks,
>>
>> Who is responsible for Upsouce [1]?
>> I see a performance issues last week.
>> Can we check Upsource health?
>>
>> [1] https://reviews.ignite.apache.org
>>
>

[Dmitriy Pavlov]

Hi Anton, do you know someone else is facing with this issue?

Could it be a local problem?

чт, 20 сент. 2018 г. в 22:18, Anton Vinogradov <[hidden email]>:

> Still, see the same issues.
>
> Denis,
>
> Could you please prioritize the fix?
>
> вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
>
> > Also, I see following on each resolve attempt.
> > UpsourceRequestExceptionImpl: 108: Internal error: An error occurred
> > during flushing data to database
> >
> >
> >
> > вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
> >
> >> Folks,
> >>
> >> Who is responsible for Upsouce [1]?
> >> I see a performance issues last week.
> >> Can we check Upsource health?
> >>
> >> [1] https://reviews.ignite.apache.org
> >>
> >
>

[Mmuzaf]

Dmitry,

I've had the same issues with using Upsource.
And we have another discussion [1] with almost the same.

[1]
http://apache-ignite-developers.2346864.n4.nabble.com/CRUD-issues-in-Ignite-Upsource-td35217.html

On Thu, 20 Sep 2018 at 23:18 Dmitriy Pavlov <[hidden email]> wrote:

> Hi Anton, do you know someone else is facing with this issue?
>
> Could it be a local problem?
>
> чт, 20 сент. 2018 г. в 22:18, Anton Vinogradov <[hidden email]>:
>
> > Still, see the same issues.
> >
> > Denis,
> >
> > Could you please prioritize the fix?
> >
> > вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
> >
> > > Also, I see following on each resolve attempt.
> > > UpsourceRequestExceptionImpl: 108: Internal error: An error occurred
> > > during flushing data to database
> > >
> > >
> > >
> > > вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
> > >
> > >> Folks,
> > >>
> > >> Who is responsible for Upsouce [1]?
> > >> I see a performance issues last week.
> > >> Can we check Upsource health?
> > >>
> > >> [1] https://reviews.ignite.apache.org
> > >>
> > >
> >
>

--
--
Maxim Muzafarov

[vveider]

Anton,


We shared this issue with Jetbrains and they found no current problems in logs, except possible problems with database connection (Apache Cassandra).
Currently there are no errors in logs.

Possible troublemakers though exist:
 - MTCGA utility “lives” on the same server, possible resource race condition, will try to separate services (at least on disk subsystem level)
 - connection to GitHub is an issue too — sometimes fetch speed drops significantly, that will be investigated also

> On 20 Sep 2018, at 22:18, Anton Vinogradov <[hidden email]> wrote:
>
> Still, see the same issues.
>
> Denis,
>
> Could you please prioritize the fix?
>
> вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
>
>> Also, I see following on each resolve attempt.
>> UpsourceRequestExceptionImpl: 108: Internal error: An error occurred
>> during flushing data to database
>>
>>
>>
>> вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
>>
>>> Folks,
>>>
>>> Who is responsible for Upsouce [1]?
>>> I see a performance issues last week.
>>> Can we check Upsource health?
>>>
>>> [1] https://reviews.ignite.apache.org
>>>
>>

[Dmitriy Pavlov]

Hi Anton, Petr,

I confirm that TC Bot writes a lot of stuff, so it may create problems for
Cassandra storage.

The latest version of Bot has optimization of data saving, and its reduces
amount of data to be saved to disk. But still, disc separation is a better
solution.

Sincerely,
Dmitriy Pavlov

пт, 21 сент. 2018 г. в 19:19, Petr Ivanov <[hidden email]>:

> Anton,
>
>
> We shared this issue with Jetbrains and they found no current problems in
> logs, except possible problems with database connection (Apache Cassandra).
> Currently there are no errors in logs.
>
> Possible troublemakers though exist:
>  - MTCGA utility “lives” on the same server, possible resource race
> condition, will try to separate services (at least on disk subsystem level)
>  - connection to GitHub is an issue too — sometimes fetch speed drops
> significantly, that will be investigated also
>
>
> > On 20 Sep 2018, at 22:18, Anton Vinogradov <[hidden email]> wrote:
> >
> > Still, see the same issues.
> >
> > Denis,
> >
> > Could you please prioritize the fix?
> >
> > вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
> >
> >> Also, I see following on each resolve attempt.
> >> UpsourceRequestExceptionImpl: 108: Internal error: An error occurred
> >> during flushing data to database
> >>
> >>
> >>
> >> вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
> >>
> >>> Folks,
> >>>
> >>> Who is responsible for Upsouce [1]?
> >>> I see a performance issues last week.
> >>> Can we check Upsource health?
> >>>
> >>> [1] https://reviews.ignite.apache.org
> >>>
> >>
>
>

[Anton Vinogradov-2]

Folks,

Thanks for the investigation!
Could you please let us know once the problem will be solved?

пт, 21 сент. 2018 г. в 19:28, Dmitriy Pavlov <[hidden email]>:

> Hi Anton, Petr,
>
> I confirm that TC Bot writes a lot of stuff, so it may create problems for
> Cassandra storage.
>
> The latest version of Bot has optimization of data saving, and its reduces
> amount of data to be saved to disk. But still, disc separation is a better
> solution.
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 21 сент. 2018 г. в 19:19, Petr Ivanov <[hidden email]>:
>
> > Anton,
> >
> >
> > We shared this issue with Jetbrains and they found no current problems in
> > logs, except possible problems with database connection (Apache
> Cassandra).
> > Currently there are no errors in logs.
> >
> > Possible troublemakers though exist:
> >  - MTCGA utility “lives” on the same server, possible resource race
> > condition, will try to separate services (at least on disk subsystem
> level)
> >  - connection to GitHub is an issue too — sometimes fetch speed drops
> > significantly, that will be investigated also
> >
> >
> > > On 20 Sep 2018, at 22:18, Anton Vinogradov <[hidden email]> wrote:
> > >
> > > Still, see the same issues.
> > >
> > > Denis,
> > >
> > > Could you please prioritize the fix?
> > >
> > > вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
> > >
> > >> Also, I see following on each resolve attempt.
> > >> UpsourceRequestExceptionImpl: 108: Internal error: An error occurred
> > >> during flushing data to database
> > >>
> > >>
> > >>
> > >> вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
> > >>
> > >>> Folks,
> > >>>
> > >>> Who is responsible for Upsouce [1]?
> > >>> I see a performance issues last week.
> > >>> Can we check Upsource health?
> > >>>
> > >>> [1] https://reviews.ignite.apache.org
> > >>>
> > >>
> >
> >
>

[vveider]

New dedicated disk subsystem is installed for MTCGA utility, hope that'll help to free resources for other services.

> On 21 Sep 2018, at 21:10, Anton Vinogradov <[hidden email]> wrote:
>
> Folks,
>
> Thanks for the investigation!
> Could you please let us know once the problem will be solved?
>
> пт, 21 сент. 2018 г. в 19:28, Dmitriy Pavlov <[hidden email]>:
>
>> Hi Anton, Petr,
>>
>> I confirm that TC Bot writes a lot of stuff, so it may create problems for
>> Cassandra storage.
>>
>> The latest version of Bot has optimization of data saving, and its reduces
>> amount of data to be saved to disk. But still, disc separation is a better
>> solution.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пт, 21 сент. 2018 г. в 19:19, Petr Ivanov <[hidden email]>:
>>
>>> Anton,
>>>
>>>
>>> We shared this issue with Jetbrains and they found no current problems in
>>> logs, except possible problems with database connection (Apache
>> Cassandra).
>>> Currently there are no errors in logs.
>>>
>>> Possible troublemakers though exist:
>>> - MTCGA utility “lives” on the same server, possible resource race
>>> condition, will try to separate services (at least on disk subsystem
>> level)
>>> - connection to GitHub is an issue too — sometimes fetch speed drops
>>> significantly, that will be investigated also
>>>
>>>
>>>> On 20 Sep 2018, at 22:18, Anton Vinogradov <[hidden email]> wrote:
>>>>
>>>> Still, see the same issues.
>>>>
>>>> Denis,
>>>>
>>>> Could you please prioritize the fix?
>>>>
>>>> вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
>>>>
>>>>> Also, I see following on each resolve attempt.
>>>>> UpsourceRequestExceptionImpl: 108: Internal error: An error occurred
>>>>> during flushing data to database
>>>>>
>>>>>
>>>>>
>>>>> вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
>>>>>
>>>>>> Folks,
>>>>>>
>>>>>> Who is responsible for Upsouce [1]?
>>>>>> I see a performance issues last week.
>>>>>> Can we check Upsource health?
>>>>>>
>>>>>> [1] https://reviews.ignite.apache.org
>>>>>>
>>>>>
>>>
>>>
>>

[Anton Vinogradov-2]

Thank you very much!

вт, 9 окт. 2018 г. в 17:43, Petr Ivanov <[hidden email]>:

> New dedicated disk subsystem is installed for MTCGA utility, hope that'll
> help to free resources for other services.
>
>
> > On 21 Sep 2018, at 21:10, Anton Vinogradov <[hidden email]> wrote:
> >
> > Folks,
> >
> > Thanks for the investigation!
> > Could you please let us know once the problem will be solved?
> >
> > пт, 21 сент. 2018 г. в 19:28, Dmitriy Pavlov <[hidden email]>:
> >
> >> Hi Anton, Petr,
> >>
> >> I confirm that TC Bot writes a lot of stuff, so it may create problems
> for
> >> Cassandra storage.
> >>
> >> The latest version of Bot has optimization of data saving, and its
> reduces
> >> amount of data to be saved to disk. But still, disc separation is a
> better
> >> solution.
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> пт, 21 сент. 2018 г. в 19:19, Petr Ivanov <[hidden email]>:
> >>
> >>> Anton,
> >>>
> >>>
> >>> We shared this issue with Jetbrains and they found no current problems
> in
> >>> logs, except possible problems with database connection (Apache
> >> Cassandra).
> >>> Currently there are no errors in logs.
> >>>
> >>> Possible troublemakers though exist:
> >>> - MTCGA utility “lives” on the same server, possible resource race
> >>> condition, will try to separate services (at least on disk subsystem
> >> level)
> >>> - connection to GitHub is an issue too — sometimes fetch speed drops
> >>> significantly, that will be investigated also
> >>>
> >>>
> >>>> On 20 Sep 2018, at 22:18, Anton Vinogradov <[hidden email]> wrote:
> >>>>
> >>>> Still, see the same issues.
> >>>>
> >>>> Denis,
> >>>>
> >>>> Could you please prioritize the fix?
> >>>>
> >>>> вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
> >>>>
> >>>>> Also, I see following on each resolve attempt.
> >>>>> UpsourceRequestExceptionImpl: 108: Internal error: An error occurred
> >>>>> during flushing data to database
> >>>>>
> >>>>>
> >>>>>
> >>>>> вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
> >>>>>
> >>>>>> Folks,
> >>>>>>
> >>>>>> Who is responsible for Upsouce [1]?
> >>>>>> I see a performance issues last week.
> >>>>>> Can we check Upsource health?
> >>>>>>
> >>>>>> [1] https://reviews.ignite.apache.org
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

[Dmitriy Pavlov]

Hi Petr,

I can't stress this enough, but this modification had a really positive
impact to TC bot performance.

Now it provides responses much faster. I hope it will help us to use this
tool and integrate into our processes.

Thank you for your efforts.

Sincerely,
Dmitriy Pavlov

вт, 9 окт. 2018 г. в 18:00, Anton Vinogradov <[hidden email]>:

> Thank you very much!
>
> вт, 9 окт. 2018 г. в 17:43, Petr Ivanov <[hidden email]>:
>
> > New dedicated disk subsystem is installed for MTCGA utility, hope that'll
> > help to free resources for other services.
> >
> >
> > > On 21 Sep 2018, at 21:10, Anton Vinogradov <[hidden email]> wrote:
> > >
> > > Folks,
> > >
> > > Thanks for the investigation!
> > > Could you please let us know once the problem will be solved?
> > >
> > > пт, 21 сент. 2018 г. в 19:28, Dmitriy Pavlov <[hidden email]>:
> > >
> > >> Hi Anton, Petr,
> > >>
> > >> I confirm that TC Bot writes a lot of stuff, so it may create problems
> > for
> > >> Cassandra storage.
> > >>
> > >> The latest version of Bot has optimization of data saving, and its
> > reduces
> > >> amount of data to be saved to disk. But still, disc separation is a
> > better
> > >> solution.
> > >>
> > >> Sincerely,
> > >> Dmitriy Pavlov
> > >>
> > >> пт, 21 сент. 2018 г. в 19:19, Petr Ivanov <[hidden email]>:
> > >>
> > >>> Anton,
> > >>>
> > >>>
> > >>> We shared this issue with Jetbrains and they found no current
> problems
> > in
> > >>> logs, except possible problems with database connection (Apache
> > >> Cassandra).
> > >>> Currently there are no errors in logs.
> > >>>
> > >>> Possible troublemakers though exist:
> > >>> - MTCGA utility “lives” on the same server, possible resource race
> > >>> condition, will try to separate services (at least on disk subsystem
> > >> level)
> > >>> - connection to GitHub is an issue too — sometimes fetch speed drops
> > >>> significantly, that will be investigated also
> > >>>
> > >>>
> > >>>> On 20 Sep 2018, at 22:18, Anton Vinogradov <[hidden email]> wrote:
> > >>>>
> > >>>> Still, see the same issues.
> > >>>>
> > >>>> Denis,
> > >>>>
> > >>>> Could you please prioritize the fix?
> > >>>>
> > >>>> вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
> > >>>>
> > >>>>> Also, I see following on each resolve attempt.
> > >>>>> UpsourceRequestExceptionImpl: 108: Internal error: An error
> occurred
> > >>>>> during flushing data to database
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
> > >>>>>
> > >>>>>> Folks,
> > >>>>>>
> > >>>>>> Who is responsible for Upsouce [1]?
> > >>>>>> I see a performance issues last week.
> > >>>>>> Can we check Upsource health?
> > >>>>>>
> > >>>>>> [1] https://reviews.ignite.apache.org
> > >>>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>

[vveider]

I am glad to help.

> On 9 Oct 2018, at 19:21, Dmitriy Pavlov <[hidden email]> wrote:
>
> Hi Petr,
>
> I can't stress this enough, but this modification had a really positive
> impact to TC bot performance.
>
> Now it provides responses much faster. I hope it will help us to use this
> tool and integrate into our processes.
>
> Thank you for your efforts.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 9 окт. 2018 г. в 18:00, Anton Vinogradov <[hidden email]>:
>
>> Thank you very much!
>>
>> вт, 9 окт. 2018 г. в 17:43, Petr Ivanov <[hidden email]>:
>>
>>> New dedicated disk subsystem is installed for MTCGA utility, hope that'll
>>> help to free resources for other services.
>>>
>>>
>>>> On 21 Sep 2018, at 21:10, Anton Vinogradov <[hidden email]> wrote:
>>>>
>>>> Folks,
>>>>
>>>> Thanks for the investigation!
>>>> Could you please let us know once the problem will be solved?
>>>>
>>>> пт, 21 сент. 2018 г. в 19:28, Dmitriy Pavlov <[hidden email]>:
>>>>
>>>>> Hi Anton, Petr,
>>>>>
>>>>> I confirm that TC Bot writes a lot of stuff, so it may create problems
>>> for
>>>>> Cassandra storage.
>>>>>
>>>>> The latest version of Bot has optimization of data saving, and its
>>> reduces
>>>>> amount of data to be saved to disk. But still, disc separation is a
>>> better
>>>>> solution.
>>>>>
>>>>> Sincerely,
>>>>> Dmitriy Pavlov
>>>>>
>>>>> пт, 21 сент. 2018 г. в 19:19, Petr Ivanov <[hidden email]>:
>>>>>
>>>>>> Anton,
>>>>>>
>>>>>>
>>>>>> We shared this issue with Jetbrains and they found no current
>> problems
>>> in
>>>>>> logs, except possible problems with database connection (Apache
>>>>> Cassandra).
>>>>>> Currently there are no errors in logs.
>>>>>>
>>>>>> Possible troublemakers though exist:
>>>>>> - MTCGA utility “lives” on the same server, possible resource race
>>>>>> condition, will try to separate services (at least on disk subsystem
>>>>> level)
>>>>>> - connection to GitHub is an issue too — sometimes fetch speed drops
>>>>>> significantly, that will be investigated also
>>>>>>
>>>>>>
>>>>>>> On 20 Sep 2018, at 22:18, Anton Vinogradov <[hidden email]> wrote:
>>>>>>>
>>>>>>> Still, see the same issues.
>>>>>>>
>>>>>>> Denis,
>>>>>>>
>>>>>>> Could you please prioritize the fix?
>>>>>>>
>>>>>>> вт, 18 сент. 2018 г. в 14:01, Anton Vinogradov <[hidden email]>:
>>>>>>>
>>>>>>>> Also, I see following on each resolve attempt.
>>>>>>>> UpsourceRequestExceptionImpl: 108: Internal error: An error
>> occurred
>>>>>>>> during flushing data to database
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov <[hidden email]>:
>>>>>>>>
>>>>>>>>> Folks,
>>>>>>>>>
>>>>>>>>> Who is responsible for Upsouce [1]?
>>>>>>>>> I see a performance issues last week.
>>>>>>>>> Can we check Upsource health?
>>>>>>>>>
>>>>>>>>> [1] https://reviews.ignite.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>